Move big data fast: rclone + checksums (practical patterns)

Moving datasets is half the job. Do it once, do it right. Here’s a clean, repeatable way to get large data into and out of your GPU computing service with integrity checks and repeatable copy jobs.

What this covers

• Install and configure rclone on a CUDA‑ready template
• Copy to/from S3‑compatible storage and SSH/SFTP servers
• Make and verify SHA‑256 manifests
• Rerun copy jobs after disconnects and verify the result
• Pick chunk sizes, parallelism, and compression that matter

Use rclone for cloud/object storage and SFTP transfers. Use rsync for local or SSH copies when you need its file-update and POSIX metadata options.

Run your workload on Compute

Check the available instance configurations and current pricing in the Compute console before launching.

Try Compute now

1) Install rclone (once per template)

These examples use Bash in a Linux container. Install rclone using the official installation instructions for your image and architecture, then check the version. The archive example also needs GNU tar, zstd, and sha256sum. Replace the example paths and remote names with your own, confirm permissions and available disk space, and preview copy commands with --dry-run before using a destination that contains important data.

rclone version


Build rclone into the container image referenced by your custom template so new instances have it. A template references an image and launch settings; it does not capture packages installed later in a running container.

2) Configure a remote (S3 or SSH)

Start the interactive config:

rclone config


Add a remote:

• S3 (AWS, MinIO, Wasabi, etc.): choose s3, then use your provider’s region, endpoint, and authentication settings. The environment example below is for AWS S3.
• SFTP/SSH: choose sftp, set host, user, port, and key path, and configure known_hosts_file with a trusted SSH known-hosts file. Verify the server fingerprint with its operator; rclone does not validate host keys by default.

Don’t bake secrets into images. Use least-privilege credentials injected at runtime, and restrict access to any rclone configuration file. Environment variables and an unencrypted config are not secret vaults.

Env-only configuration: AWS S3 example

# Inject AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as runtime secrets.
# For temporary credentials, also inject AWS_SESSION_TOKEN.
export RCLONE_CONFIG_MYREMOTE_TYPE=s3
export RCLONE_CONFIG_MYREMOTE_PROVIDER=AWS
export RCLONE_CONFIG_MYREMOTE_ENV_AUTH=true
# Change this to the actual bucket region.
export RCLONE_CONFIG_MYREMOTE_REGION=us-east-1


3) Copy data in (and rerun after a failure)

To instance (S3 → NVMe)

# Download into a dedicated directory.
mkdir -p /data/projectA &&
rclone copy myremote:datasets/projectA /data/projectA \
--progress --transfers 4 --checkers 8


From instance (NVMe → S3)

rclone copy /data/results myremote:results/projectA \
--progress --transfers 4 --checkers 8


• After interruption: rerun the same copy command. Files that match its comparison checks are skipped. Do not assume an incomplete file resumes at its last byte after rclone exits; behavior depends on the backend and operation. Verify the completed dataset.
• Tuning: measure throughput and RAM before increasing concurrency. S3 multipart buffers scale with transfers × upload concurrency × chunk size. For example, 16 transfers × 6 chunks × 64 MiB could require about 6 GiB, before other overhead. --fast-list also uses memory. Change one setting at a time; larger chunks are not a universal starting point.

SSH/SFTP example

rclone copy /data/results sftpremote:/srv/results/projectA \
--progress --transfers 4 --checkers 8 \
--sftp-known-hosts-file ~/.ssh/known_hosts


4) Integrity: SHA‑256 manifests you can trust

Make a manifest from a stable source directory of regular files, keep it outside that directory, then copy and verify the dataset. Do not change source files during hashing or transfer. Use a new run-specific destination prefix and protect the manifest through a trusted channel or verified signature; a checksum alone does not authenticate the sender.

Create manifest at source

mkdir -p /data/manifests &&
rclone hashsum SHA256 /data/results > /data/manifests/projectA.SHA256SUMS.txt &&
rclone checksum SHA256 /data/manifests/projectA.SHA256SUMS.txt /data/results


Copy data + manifest

# Continue only after the source verification succeeds.
rclone copy /data/results myremote:results/projectA --progress &&
rclone copyto /data/manifests/projectA.SHA256SUMS.txt \
myremote:manifests/projectA.SHA256SUMS.txt


Verify at destination (downloaded)

# Obtain the trusted source manifest on this machine first.
# Replace this path with its actual location; do not create an empty file.
manifest=/trusted/manifests/projectA.SHA256SUMS.txt
test -s "$manifest" &&
rclone copy myremote:results/projectA ./projectA --progress &&
rclone checksum SHA256 "$manifest" ./projectA


Verify remote content by reading it

# Reads object contents through this machine; may incur request/egress fees.
rclone checksum SHA256 /data/manifests/projectA.SHA256SUMS.txt \
myremote:results/projectA --download


A successful rclone checksum run must exit with code 0. Without --one-way, it also reports unexpected destination files. SHA-256 is not interchangeable with MD5 or a multipart ETag. Investigate any error or missing file before treating the transfer as verified.

5) Sync vs copy, and delete safety

• copy adds or updates destination files without deleting extra files. It can overwrite an existing file, so retain a separate backup or use a new destination prefix.
• sync makes the destination match the source (including deletes). Use with care:

rclone sync /data/results myremote:results/projectA --dry-run --delete-after --progress


Keep --dry-run while reviewing every planned change. Only remove it after confirming the exact source and destination, an independent backup, and that destination-only files should be deleted. --delete-after changes deletion order; it does not make an incorrect destination safe.

6) Fewer files = faster transfers (bundle smartly)

Millions of tiny files can spend much of their transfer time on per-file work. Bundle a stable dataset when that suits how you will retrieve it; benchmark compression on representative data first.

# Requires GNU tar, zstd, and sha256sum. Keep output outside /data/run123.
# Choose a new archive name for each run.
mkdir -p /data/bundles &&
tar -C /data/run123 -I 'zstd -T1 -3' -cf /data/bundles/run123.tar.zst . &&
(
cd /data/bundles &&
sha256sum run123.tar.zst > run123.tar.zst.sha256 &&
rclone copyto run123.tar.zst myremote:runs/run123.tar.zst --progress &&
rclone copyto run123.tar.zst.sha256 myremote:manifests/run123.tar.zst.sha256
)


This example uses one zstd worker and compression level 3 as a modest starting point, not a performance guarantee. Use pigz when gzip compatibility matters, and choose bundle sizes around available disk space and the cost of retransferring a failed bundle.

7) Move data between buckets or projects

For a cross-provider copy, rclone normally streams data through the machine running the command, without keeping a complete local copy:

rclone copy awsA:bucketA/prefix gsB:bucketB/prefix --progress --transfers 4 --checkers 8


Both remotes need the required permissions. Server-side copying is possible only for supported backend/configuration combinations. Check the transfer logs, available bandwidth, and each provider’s request and data-transfer charges.

8) Bandwidth and reliability knobs

• --bwlimit 100M caps data transfer at 100 MiB/s, about 839 Mbit/s, rather than 100 Mbit/s.
• --retries 8 --low-level-retries 20 increases retry attempts but can delay failure reporting; fix persistent errors rather than relying on retries.
• --timeout 2m --contimeout 10s changes I/O-idle and connection timeouts. These values can fail slow connections sooner; tune from observed failures.
• --checksum changes file comparison to use a shared hash where available. It is not a guarantee of SHA-256 verification; use the manifest checks above.

Log the command and rclone version in your run card, but redact credentials, tokens, and signed URLs.

9) rsync when both ends are POSIX

For SSH transfers, install rsync on both ends and verify the SSH host key. This example copies the contents of the source directory and does not delete destination-only files:

rsync -avh --partial --partial-dir=.rsync-partial --progress \
/data/results/ user@host:/srv/results/projectA/


--partial-dir retains partial data for a later rsync run; keep that directory inaccessible to other users. -a does not include hard links, ACLs, or extended attributes; review -H, -A, and -X if needed and supported. Still verify against a trusted SHA-256 manifest.

10) Security basics

• Keep credentials out of images and source control. Restrict config-file access and use runtime secret injection; redact secrets from logs.
• Mount secrets at runtime; don’t commit them.
• Keep buckets private and use authenticated HTTPS requests or SFTP with host-key validation. A presigned URL grants time-limited access to an object; it does not make an independently public object private. Review bucket policies and protect signed URLs as credentials.

Methods snippet (copy‑paste)

transfers:
tool: "<actual rclone version>"
source:
type: "<actual backend>"
url: "<path or remote:bucket/prefix, no secrets>"
destination:
type: "<actual backend>"
url: "<path or remote:bucket/prefix, no secrets>"
command: |
rclone copy <src> <dst> --transfers 4 --checkers 8 --progress
manifest:
algo: "SHA256"
file: "<trusted manifest outside the payload directory>"
verified: "yes | no"
verification_exit_code: "<record actual result>"
notes: "source snapshot, compression, retries/timeouts, verification time"


Related reading

• Scientific modeling on cloud GPUs — what works, what doesn’t

Try Compute today

Choose an available GPU configuration and a compatible container image in Compute. Use custom templates to reuse your image and launch settings, and check current pricing before starting. Restart availability and retention limits apply to stopped instances; export important data before termination. If your application needs strong FP64 performance, confirm the hardware fit before committing to a run.

‍