Fill the Server: 80% off your first 4 monthly invoices, monthly billing. Ends Nov 30, 2026. 25 spots in this batch.FILL25
Fill the Server, annual: about 29.63% off your first annual invoice. Ends Nov 30, 2026.FILL25Y
Fill the Server, 2 year term: about 15.69% off your first biennial invoice. Ends Nov 30, 2026.FILL25B
Fill the Server, 3 year term: about 11.11% off your first triennial invoice. Ends Nov 30, 2026.FILL25T
80% off your first 3 months, monthly billing. Ends Dec 25, 2027.3MONTHS80
50% off your first annual Shared term. Ends Dec 31, 2027.50SHARED
Shared Starter, 3 year term: C$105.84 total.MOOSE294
20% off your first 3 monthly VPS invoices. Ends Dec 31, 2027.20VPS3
25% off your first annual base VPS term.25VPS

← All guides

Object storage

Large file uploads to S3-compatible storage: design for interruption

Published August 2026 · 8 min read

Large objects expose weaknesses that tiny S3 tests never encounter. A successful 1 MB upload does not prove that a 200 GB archive can survive a WAN interruption, stay within quota or complete without exhausting client resources. Test interruption and restart behavior before trusting a large-file workflow with a production backup window or customer upload.

Use multipart for very large objects

Multipart upload lets the client retry a failed piece without retransmitting the entire object. It can also send parts concurrently to use available bandwidth more effectively.

Choose part size with provider limits and client memory in mind. Extremely small parts create excessive request overhead; extremely large parts reduce the benefit of granular retries.

Bound concurrency

Parallel uploads can saturate CPU, memory, source disk or network before they reach the storage service. Start conservatively and raise concurrency while monitoring end-to-end throughput and errors.

A backup server sharing bandwidth with production traffic may need a rate limit even when the object endpoint itself can accept more.

Leave quota headroom

In-progress parts can consume backend capacity. Starting a large upload with only a few megabytes below the hard quota is likely to fail or leave cleanup work.

Alert customers before capacity reaches the hard stop and make package upgrades idempotent so a quota change does not alter credentials or bucket identity.

Verify the completed bytes

ETag semantics can differ for multipart objects, so do not blindly treat every ETag as a simple MD5 checksum. Use application checksums or download verification appropriate to the client.

For backup archives, a restore or archive integrity test is stronger evidence than an upload-complete message.

Clean abandoned uploads

Interrupted clients can leave multipart uploads that were never completed. Inventory and abort old abandoned sessions based on conservative age rules.

Cleanup should identify exact upload IDs and bucket ownership so a maintenance job cannot abort another active customer transfer.

Next step

Make large S3 uploads resilient with multipart transfer, bounded concurrency, checksums, resumable workflows, quota headroom and explicit cleanup of failed uploads.

Explore DotMoose Object Storage

Keep reading

Related guides.

boto3 S3-compatible storage: custom endpoint, put_object and connection tests

Connect boto3 to S3-compatible storage with endpoint_url. Start with a read-only list, then optionally verify put_object with one isolated write/read/delete round trip.

Read the guide

S3 multipart upload explained: part size, retries, checksums and cleanup

Learn how S3 multipart upload works: numbered parts, AWS part-size limits, parallel retries, completion, checksums and cleanup of abandoned uploads.

Read the guide

S3 presigned URLs explained: temporary access without sharing keys

Learn how S3 presigned URLs authorize a specific operation for a limited time, where they help, what they expose and how to use them safely.

Read the guide

More object storage guides

We will handle the move. You check it first.

Your current site stays online until your new one is ready. You approve the copy first, and if anything is not right we switch you back right away. The price you pick is the price you renew at.

Start my free moveSee plans and prices
See pricesFree move