AWS DataSync
An AWS service that copies large amounts of files over the network, and checks they arrived intact.
AWS DataSync moves files and objects from one storage system to another over the network. The typical case is a company with years of documents, images or research data on a file server in its own office, which it wants in Amazon S3, Amazon EFS or Amazon FSx. It also works between AWS services and from other clouds, like Azure Blob Storage or Google Cloud Storage.
For on-premises storage you install a small virtual machine, the DataSync agent, near your file server. It reads from the usual protocols, such as NFS, SMB, HDFS or object storage, and sends the data to AWS. You define a task with a source, a destination and options, then run it once or on a schedule. Each run only copies what changed, encrypts data in transit and verifies that what landed matches what left.
You could copy files yourself with scripts, but at millions of files the hard parts are speed, retries, permissions and proving nothing got corrupted. DataSync uses many parallel streams to make the most of your link, and it handles those details. When the link is too slow for the amount of data, a physical transfer is the alternative.
Keep in mind
- Copies files and objects into, out of and between AWS storage services.
- Reads NFS, SMB, HDFS and object storage, plus some other clouds.
- Copies only changes on repeat runs and verifies data integrity.
- Uses an agent installed near your on-premises storage.
