AWS S3 sync: preview transfers, control deletions, and verify files

aws s3 sync copies new and selected changed files between a local directory and an S3 prefix, or between two S3 locations. It runs in one direction each time you invoke it. Start with --dryrun, check the destination carefully, and keep deletion separate from the first transfer.

The useful question is not just whether the command finished. It is whether the intended files reached the intended destination without overwriting or deleting something you needed. This guide provides a small review workflow for AWS CLI v2 and Amazon S3 general purpose buckets. Bucket names, profile names, regions, and paths are examples to replace.

When to use S3 sync

Use sync for a repeatable export directory, a selected dataset download, or a one-time bucket-to-bucket transfer. It examines the source and destination to decide which items need copying. It does not keep watching for changes after the command exits.

aws s3 cp is useful for explicitly copying a file, or recursively copying a set of files. Sync adds comparison rules to avoid some unnecessary transfers. Neither command turns an object store into a mounted filesystem. Our Amazon cloud storage guide explains how S3 differs from file and block storage.

Confirm your identity and prepare a small source directory

Use an existing authorized AWS profile and confirm which identity it selects:

aws --version
aws sts get-caller-identity --profile storage-review

GetCallerIdentity reports the account and user or role. A successful identity check does not establish permission to read or write the bucket. Use your normal sign-in or role workflow; do not paste credentials into the commands or article.

Prepare ./exports/ with a few non-sensitive files that are no longer being written. Select an existing destination bucket and a dedicated test prefix, such as review/. Confirm that the prefix is unused before relying on it as a clean test area. Do not start with your entire home directory or the root of a production bucket.

For normal transfers, permissions commonly include listing the relevant bucket and reading source objects or writing destination objects. Deletion, KMS encryption, tags, and multipart operations can require additional permissions. Match the actual operations to AWS’s S3 permissions reference; avoid granting full S3 access just to get past an error.

Preview a local upload, then run the same transfer

In this command the local directory is the source and the S3 prefix is the destination:

aws s3 sync ./exports/ s3://amzn-s3-demo-bucket/review/ --profile storage-review --no-follow-symlinks --dryrun

Read every proposed operation in the small test. A file at ./exports/reports/week.csv belongs under the key review/reports/week.csv. The name exports is not automatically added to that destination prefix. For sync, adding or removing a trailing slash does not change directory-content semantics. AWS documents the path argument rules.

--dryrun displays planned operations without performing the transfer. It can still need network access to list S3 objects. It is not a full permission test or a locked transaction plan: data or permissions can change before the real command runs. --no-follow-symlinks avoids following symbolic links within the local upload tree.

After confirming the source, destination, and preview, remove only --dryrun:

aws s3 sync ./exports/ s3://amzn-s3-demo-bucket/review/ --profile storage-review --no-follow-symlinks

This command can write objects and overwrite destination keys selected by the comparison rules, even though it has no --delete. Keep the source stable during the run. If writers are active, produce a completed export or snapshot first rather than assuming sync gives you a consistent point-in-time dataset.

Download or copy between buckets

For a download, put the S3 location first and use a new local directory:

aws s3 sync s3://amzn-s3-demo-bucket/review/ ./restore-review/ --profile storage-review --no-follow-symlinks --dryrun

Review the plan before removing --dryrun. Local files can also be overwritten. On a case-insensitive filesystem, keys such as Report.csv and report.csv cannot necessarily be represented as separate files. Check naming collisions before downloading an unfamiliar dataset.

For two AWS buckets, use separate source and destination locations. Here the source is in Ireland and the destination in Frankfurt:

aws s3 sync s3://amzn-s3-demo-source/review/ s3://amzn-s3-demo-destination/review/ --profile storage-review --source-region eu-west-1 --region eu-central-1 --dryrun

Replace both region values with the buckets’ actual regions. --source-region identifies the source; --region identifies the destination. Do not point the source and destination at the same underlying objects. Review the sync command reference for options supported by your installed version.

Understand what “changed” means

The default comparison uses whether an item exists, its size, and modification times. The detailed timestamp behavior depends on transfer direction. It is not a full byte-by-byte comparison of every source file with its destination. Equal size alone does not prove equal content.

• --size-only removes timestamps from the comparison. Same-sized content changes can be missed.
• --exact-timestamps applies to S3-to-local sync and makes exact timestamp matching matter for same-sized files.
• The current CLI v2 reference includes --no-overwrite, which transfers only items missing from the destination. Check aws s3 sync help if your installation does not recognize it.

A quiet second run is useful evidence that the comparison found no work. It is not an integrity audit. Similarly, a metadata change alone does not mean sync will recopy an otherwise unchanged item. Do not use a routine sync as an assumed bulk metadata rewrite.

Filter files deliberately

To preview only CSV files, exclude everything first and then include the desired pattern:

aws s3 sync ./exports/ s3://amzn-s3-demo-bucket/review/ --profile storage-review --no-follow-symlinks --exclude "*" --include "*.csv" --dryrun

Files are included by default, so an include rule alone does not narrow the transfer. Later filter rules take precedence. Quote patterns so the shell does not expand them before AWS CLI receives them. Check the preview for nested paths, case differences, and files you did not intend to include. A filename extension is not a security check; a CSV export can still contain sensitive data.

Keep --delete out of the first transfer

--delete removes destination items missing from the source, within the operation’s scope and filters. Reversing the command reverses which side can lose files. An accidentally empty source directory can therefore be dangerous.

If you intend to maintain a mirror, review deletion separately. This example previews deletions without executing them:

aws s3 sync ./exports/ s3://amzn-s3-demo-bucket/review/ --profile storage-review --no-follow-symlinks --delete --dryrun

Inspect every proposed deletion, confirm the source is complete, and verify recovery before authorizing the real mirror operation. Do not remove --dryrun just because an earlier upload preview looked correct. Excluded items are also excluded from deletion, so filters affect the final mirror. AWS’s sync examples show the directional deletion behavior.

In a versioning-enabled bucket, ordinary deletion usually creates a delete marker, while overwrites create new versions. Recovery depends on retained versions and permissions; lifecycle rules can remove older versions. Review S3 Versioning before treating it as a recovery mechanism. Local files do not gain this protection.

Check completion and verify a downloaded sample

Read errors and capture the exit code immediately after the transfer. In a POSIX shell:

echo $?

In PowerShell:

$LASTEXITCODE

Zero indicates success as reported by the command. For S3 commands, 1 indicates a failed transfer and 2 can indicate skipped files; other nonzero codes identify additional failures. Use the AWS CLI return-code reference. Do not let a scheduled job label a nonzero result as successful merely because some objects arrived.

The AWS CLI provides checksum validation during transfers when supported. This differs from the decision about which items sync selects. For a separate spot check, download a sample into a fresh directory, keep the original unchanged, and compare its complete contents. This Python 3.11+ helper uses hashlib file hashing. Save it as compare_files.py:

import hashlib
import sys
from pathlib import Path

def digest(path):
    with path.open("rb") as source:
        return hashlib.file_digest(source, "sha256").hexdigest()

def main(args):
    if len(args) != 2:
        print("Usage: compare_files.py ORIGINAL DOWNLOADED", file=sys.stderr)
        return 2
    original, downloaded = map(Path, args)
    if original.resolve() == downloaded.resolve():
        print("Choose two different files.", file=sys.stderr)
        return 2
    try:
        match = digest(original) == digest(downloaded)
    except OSError as error:
        print(error, file=sys.stderr)
        return 2
    print("MATCH" if match else "MISMATCH")
    return 0 if match else 1

if __name__ == "__main__":
    raise SystemExit(main(sys.argv[1:]))

python compare_files.py ./exports/sample.csv ./restore-review/sample.csv

A match checks those two files only. It does not prove that every expected object is present, that metadata matches, or that a database export can be restored. For a migration, reconcile the expected key list as well as checking file contents, and test the destination application before retiring the source.

Budget for repeated scans and recovery

Sync can list a large prefix even when few objects need copying. Include requests, transfer, retrieval, destination storage, and encryption-related charges where applicable. Check the relevant regions and storage classes on the S3 pricing page. A dry run can still involve billable listing requests; it is not a zero-cost guarantee.

Sync is a useful transfer step in a cloud backup plan, but it does not supply independent retention or a tested restore by itself. For continuous AWS bucket replication, evaluate S3 Replication rather than assuming a scheduled sync has the same behavior.

For another S3-compatible provider, verify its endpoint, credentials, supported operations, and checksum behavior separately. These AWS examples do not establish feature parity for Hivenet or any other service.