S3 multipart upload: part sizes, retries, and cleanup

S3 multipart upload sends one large file as several independently uploaded parts, then assembles them into a single object. It lets a client retry a failed part without sending the complete file again. It can also use parallel uploads, although the improvement depends on the network, client, and workload.

This guide covers Amazon S3 general-purpose buckets, a Python upload example, and the checks needed to avoid incomplete uploads accumulating unnoticed. For the broader choice between S3, disks, and shared files, start with our Amazon cloud storage guide.

When to use S3 multipart upload

The Amazon S3 User Guide recommends considering multipart uploads when an object reaches about 100 MB. Treat that as a starting point for testing. Small files may not benefit from the extra requests; large files on an unreliable connection often benefit from retrying smaller pieces.

For an ordinary server-side file transfer, begin with a managed AWS SDK method. Build a custom multipart workflow when you need control over persisted upload sessions, browser uploads, or coordination across workers. S3 multipart upload is a sequence of API operations, not an HTTP multipart/form-data form.

How the multipart upload process works

1. Initiate: request a new upload for a bucket and object key. S3 returns an upload ID.
2. Upload parts: send each part with that ID and its part number, keeping the returned ETag and applicable checksum.
3. Complete: submit the ordered part records so S3 can assemble the object.

The initiation request is also where a custom client supplies object metadata and encryption settings. Keep the bucket, key, and upload ID together. A filename alone cannot identify an unfinished upload.

UploadPart accepts parts independently. Reusing the same part number within that upload replaces the earlier part. A custom client must preserve the correct completion records instead of treating every successful response as a new part.

Current limits and practical part sizes

The AWS multipart limits, checked October 29, 2026, specify:

• A maximum object size of 48.8 TiB.
• Up to 10,000 parts, numbered from 1 through 10,000.
• Parts from 5 MiB to 5 GiB, with no minimum size for the final part.

MiB and GiB use powers of 1,024. Calculate the number of parts before starting: divide the file size by the chosen part size and round up. Keep that count within the limit and leave room for the largest files your application accepts.

For example, a 50 GiB file divided into 64 MiB parts produces 800 parts. The basic sequence then has 802 API operations: one initiation, 800 part uploads, and one completion, before retries or verification requests. This is arithmetic for planning, not a performance benchmark.

Upload a large file with Boto3

Use a private test bucket, a non-sensitive file, and an approved AWS role or credential profile. Install Boto3 in your Python environment. Do not place access keys in the script. Replace the bucket name and region below, and put your test file at sample.bin.

The upload_file method manages the transfer. This example chooses a unique destination key for each run and requests SHA-256 checksums:

from pathlib import Path
from uuid import uuid4

import boto3
from boto3.s3.transfer import TransferConfig

MiB = 1024 ** 2
source = Path("sample.bin")
bucket = "YOUR_PRIVATE_TEST_BUCKET"
key = f"multipart-demo/{uuid4().hex}/{source.name}"

s3 = boto3.client("s3", region_name="us-east-1")
transfer = TransferConfig(
    multipart_threshold=128 * MiB,
    multipart_chunksize=64 * MiB,
    max_concurrency=4,
    preferred_transfer_client="classic",
)

s3.upload_file(
    str(source),
    bucket,
    key,
    Config=transfer,
    ExtraArgs={"ChecksumAlgorithm": "SHA256"},
)
print(f"Upload completed: s3://{bucket}/{key}")

The transfer configuration separates the threshold for switching to multipart from the size of each part. Here, a 160 MiB file becomes two 64 MiB parts and a final 32 MiB part. The maximum concurrency is four, which does not guarantee four times the throughput.

The example explicitly selects the classic transfer client so its transfer settings have a clear scope. Tune one setting at a time with representative files. Record transfer duration, retries, CPU use, and memory before increasing concurrency.

Permissions for upload and cleanup

Scope the signing identity to the intended destination. Upload operations need the relevant s3:PutObject permission. Cleanup and inspection can additionally require s3:AbortMultipartUpload, s3:ListMultipartUploadParts, and s3:ListBucketMultipartUploads. Review the multipart permission mapping, including bucket policies and encryption-key access.

With SSE-KMS, the required KMS permissions also matter. A successful start does not prove that later parts or completion will be authorized. Test the actual application role rather than relying on an administrator's result.

Retries and durable resume are different

The managed SDK handles eligible retries during the transfer. This script does not save an upload ID or part records for use after the process exits. Running it again generates a new key and starts a new transfer; it does not resume a previous session.

If you need to resume object uploads across restarts, design persistent session records containing the source identity, destination, upload ID, part numbers, ETags, and checksum information. Confirm the source file has not changed before reusing uploaded parts. Also define which worker owns completion and when an abandoned session should be stopped.

The ListParts API supports inspection and is paginated. Keep your own upload records for completion and use listings to reconcile them. Missing pages or unfinished requests can otherwise produce an incomplete picture.

Verify completion and integrity

A multipart ETag is not a reliable whole-file MD5 digest. S3 supports both full-object and composite checksum types; the SHA-256 multipart checksum is composite. Do not compare it directly with an ordinary SHA-256 hash of the entire local file. Follow the checksum rules for the chosen algorithm and upload method.

For a bounded acceptance test, download the completed object through an authorized account and compare its bytes, or calculate the same whole-file digest locally on both copies. Matching size alone does not establish matching content. Include verification traffic and storage in the test budget.

A client calling CompleteMultipartUpload directly must inspect the response body: an initial HTTP 200 can contain an embedded error. AWS SDKs handle that condition. Mark a job successful only after completion succeeds, then record the destination key and any version information needed for later retrieval.

Stop incomplete uploads and control costs

Uploaded parts consume billable storage until the upload is completed or aborted. A failed application job is not evidence that every stored part has been removed. Review the current S3 pricing for requests, storage, and any applicable retrieval or transfer costs.

To inspect the test prefix with the AWS CLI, use the multipart upload listing:

aws s3api list-multipart-uploads --bucket YOUR_PRIVATE_TEST_BUCKET --prefix multipart-demo/

Review the key, upload ID, initiation time, and owner before removing anything. After stopping the test's upload workers, abort only the disposable session you identified:

aws s3api abort-multipart-upload --bucket YOUR_PRIVATE_TEST_BUCKET --key "EXACT_TEST_KEY" --upload-id "EXACT_TEST_UPLOAD_ID"

The abort API warns that requests already in flight may still finish. Verify that no parts remain; another abort may be needed if concurrent work survived the first request. Do not run broad cleanup against sessions that another application is still using.

Add a scoped lifecycle safety net

For general-purpose buckets, an AbortIncompleteMultipartUpload lifecycle rule can make unfinished uploads eligible for cleanup after a chosen number of days. Scope it to the appropriate prefix and allow enough time for legitimate long-running transfers. It does not delete completed objects.

Review existing lifecycle settings before adding the rule. Keep immediate failure cleanup in the application; the lifecycle rule is a fallback, not a promise of removal at an exact moment.

What the example has been tested to do

We ran this example locally with Python 3.12.14 and Boto3/Botocore 1.43.121, using dummy credentials, mocked S3 responses, and blocked network connections. The checks covered a 1 MiB single upload, a 160 MiB multipart upload with ordered completion records, and a simulated part failure that triggered an abort.

Those checks do not establish live AWS permissions, checksum acceptance, network retry behavior, throughput, or lifecycle timing. Before production:

• Run small and multipart transfers under the real application role.
• Retrieve the intended object and compare its content with the source.
• Test interrupted transfers, failed parts, and process restarts separately.
• Verify cleanup after failure and inspect outstanding upload sessions.
• Check costs and resource use with representative files and concurrency.

For the surrounding application decisions, see our cloud storage for applications guide. If you use an S3-compatible provider, verify its limits, checksums, lifecycle support, and SDK behavior separately. This Amazon S3 procedure does not establish support for another service or for directory buckets.