Fill the Server: 80% off your first 4 monthly invoices, monthly billing. Ends Nov 30, 2026. 25 spots in this batch.FILL25
Fill the Server, annual: about 29.63% off your first annual invoice. Ends Nov 30, 2026.FILL25Y
Fill the Server, 2 year term: about 15.69% off your first biennial invoice. Ends Nov 30, 2026.FILL25B
Fill the Server, 3 year term: about 11.11% off your first triennial invoice. Ends Nov 30, 2026.FILL25T
80% off your first 3 months, monthly billing. Ends Dec 25, 2027.3MONTHS80
50% off your first annual Shared term. Ends Dec 31, 2027.50SHARED
Shared Starter, 3 year term: C$105.84 total.MOOSE294
20% off your first 3 monthly VPS invoices. Ends Dec 31, 2027.20VPS3
25% off your first annual base VPS term.25VPS

← All guides

Object storage

Millions of small objects: what changes in object-storage design?

Published August 2026 · 8 min read

A terabyte stored as ten huge archives behaves differently from a terabyte stored as fifty million tiny objects. Capacity may be identical while request count, listing behavior, lifecycle work and application metadata become the dominant operational concerns.

Do not use bucket listing as the application database

Large listings are paginated and can require many requests. Applications should normally keep their own index of business objects so ordinary user queries do not require a full bucket scan.

Use object listing for reconciliation, maintenance and bounded prefixes. It is not a replacement for relational or search indexes.

Measure request behavior

Small objects produce more requests per stored gigabyte. Even when a provider does not charge per request, client concurrency, rate limits and latency can affect throughput.

Benchmark realistic object sizes and key distributions. A single 5 GB upload benchmark does not predict a thumbnail workload.

Plan lifecycle operations

Deleting millions of expired objects can take time and create a large control-plane workload. Lifecycle rules are useful when the provider can process them efficiently, but the application should not assume expiry is instantaneous.

Track backlog and storage after cleanup so an intended retention rule does not become silent capacity growth.

Use key prefixes deliberately

Use key prefixes to group related objects when listing them. A prefix is part of an object key, not a filesystem directory. Avoid putting unbounded timestamp or user-controlled text into keys without normalization.

Hash or shard prefixes can help some systems distribute load, but follow current provider guidance before applying tuning rules from another or older S3 implementation.

Monitor object count alongside bytes

Authoritative storage usage should include both used bytes and object count. A sudden object-count spike with modest byte growth can reveal a runaway job before the quota fills.

Capacity alerts alone can miss an application generating millions of useless zero- or tiny-byte objects.

Next step

Plan S3-compatible workloads with very high object counts by measuring listing, request concurrency, metadata overhead, lifecycle behavior and application indexing separately from capacity.

Explore DotMoose Object Storage

Keep reading

Related guides.

boto3 S3-compatible storage: custom endpoint, put_object and connection tests

Connect boto3 to S3-compatible storage with endpoint_url. Start with a read-only list, then optionally verify put_object with one isolated write/read/delete round trip.

Read the guide

S3 multipart upload explained: part size, retries, checksums and cleanup

Learn how S3 multipart upload works: numbered parts, AWS part-size limits, parallel retries, completion, checksums and cleanup of abandoned uploads.

Read the guide

S3 presigned URLs explained: temporary access without sharing keys

Learn how S3 presigned URLs authorize a specific operation for a limited time, where they help, what they expose and how to use them safely.

Read the guide

More object storage guides

We will handle the move. You check it first.

Your current site stays online until your new one is ready. You approve the copy first, and if anything is not right we switch you back right away. The price you pick is the price you renew at.

Start my free moveSee plans and prices
See pricesFree move