BatchFerry copies, syncs and verifies large object sets between S3-compatible endpoints. It streams objects in parallel, resumes interrupted transfers, and checks integrity with server-side ETags — without staging anything on local disk.
$ batchferry sync s3://archive-eu/datasets s3://cold-store/datasets \
--workers 8 --verify etag --resume
Static binaries are published for Linux, macOS and FreeBSD. Install from source with Go 1.22+:
$ go install batchferry.org/cmd/batchferry@latest $ batchferry --version batchferry 0.9.4 (build 2026-07-11)
Endpoints are declared once and referenced by alias:
$ batchferry remote add archive-eu \
--endpoint https://s3.eu-central.example.net \
--region eu-central-1
$ batchferry ls archive-eu/datasets
Transfer state is journalled per batch, so an interrupted run continues where it stopped instead of restarting.
Objects are streamed endpoint to endpoint with a bounded worker pool. Nothing is written to local disk.
Optional ETag or SHA-256 verification per object, with a machine-readable report at the end of the run.
--dry-run prints the exact object set and byte count a command would move, before it moves anything.
| Flag | Default | Description |
|---|---|---|
--workers | 4 | Concurrent object transfers. |
--part-size | 16MiB | Multipart chunk size for large objects. |
--verify | etag | Integrity mode: none, etag or sha256. |
--resume | off | Reuse the batch journal from a previous run. |
--include / --exclude | — | Glob filters applied to object keys. |
Anything that speaks the S3 API, including MinIO, Ceph RGW, Backblaze B2 (S3 endpoint) and Wasabi. Non-S3 backends are out of scope.
No. Credentials may be supplied by environment variables or an external credential helper; the remote definition stores only the endpoint and region.
Runs of a few million objects are routine. The journal is a single append-only file and stays well under a gigabyte at that scale.