AWSMigration
AWS DataSync and Snow Family
Data transfer — DataSync for online file/object transfers and Snow Family for offline, petabyte-scale physical data migration.
Overview
AWS DataSync automates and accelerates online data transfers between on-premises storage and AWS storage services, while the AWS Snow Family provides physical devices for offline, petabyte-scale migrations when network transfer is impractical.
| Feature | DataSync | Snow Family |
|---|---|---|
| Transfer method | Online (over network) | Offline (physical device shipped) |
| Best for | Ongoing sync, ≤ ~100 TB over decent bandwidth | Petabyte-scale or limited/no network bandwidth |
| Speed | Up to 10 Gbps with agent | Device capacity: 8 TB – 100 PB |
| Direction | Bidirectional | Primarily to AWS (export also supported) |
Core Concepts
| Concept | Description |
|---|---|
| DataSync Agent | VM deployed on-premises that reads from local storage and transfers to AWS |
| DataSync Task | Defines source/destination locations, transfer settings, schedule, and filters |
| Task Execution | A single run of a DataSync task; tracks bytes transferred and files synced |
| Snowball Edge Storage | 80 TB usable storage device with local compute (optional); ships to/from AWS |
| Snowball Edge Compute | Device optimised for edge computing with less storage but more CPU/GPU |
| Snowcone | Small, rugged 8–14 TB device for edge collection and migration in harsh environments |
| Snowmobile | 100 PB shipping container towed by a truck for exabyte-scale migrations |
| AWS OpsHub | GUI application for managing Snow Family devices locally |
AWS DataSync
How DataSync Works
On-Premises AWS
┌──────────────┐ ┌──────────┐ ┌─────────────────┐
│ NFS / SMB / │────▶│ DataSync │────▶│ S3 / EFS / FSx │
│ HDFS / Object│ │ Agent │ │ │
│ Storage │ │ (VM) │ │ (or another │
└──────────────┘ └──────────┘ │ on-prem NFS) │
│ └─────────────────┘
TLS encrypted
over Direct Connect
or internetDataSync Features
| Feature | Description |
|---|---|
| Incremental transfer | Only transfers changed files after initial sync |
| Scheduling | Cron-based or hourly/daily/weekly task schedules |
| Bandwidth throttling | Limit bandwidth to avoid impacting production traffic |
| Data integrity checks | Automatic verification using checksums during and after transfer |
| Task filtering | Include/exclude files based on name patterns or modification time |
| Encryption | TLS in transit; destination encryption at rest (S3 SSE, EFS, etc.) |
Supported Locations
| Source / Destination | Notes |
|---|---|
| NFS (on-premises) | Via DataSync agent |
| SMB (on-premises) | Via DataSync agent, AD-integrated |
| HDFS (on-premises) | Via DataSync agent |
| Self-managed object storage | S3-compatible APIs |
| Amazon S3 | All storage classes including Glacier |
| Amazon EFS | Cross-region or cross-account EFS sync |
| Amazon FSx | FSx for Windows, Lustre, NetApp ONTAP, OpenZFS |
| Other cloud storage | Azure Blob, Google Cloud Storage (via agent) |
AWS Snow Family
Device Comparison
| Device | Usable Storage | Compute | Use Case |
|---|---|---|---|
| Snowcone | 8 TB HDD / 14 TB SSD | 2 vCPUs, 4 GB RAM | Edge collection, small office, IoT |
| Snowball Edge Storage | 80 TB | 40 vCPUs, 80 GB RAM | Large data migration, local storage |
| Snowball Edge Compute | 42 TB (210 TB with GPU) | 52–104 vCPUs, 208–416 GB RAM | Edge ML inference, local processing |
| Snowmobile | 100 PB | N/A | Exabyte-scale DC migration |
Snow Family Workflow
1. Order device via AWS Console
2. AWS ships device to location
3. Connect device to local network
4. Load data using OpsHub / CLI / NFS interface
5. Ship device back to AWS
6. AWS loads data into S3 bucketEdge Computing on Snow
- Run EC2 instances and Lambda functions locally on Snowball Edge
- Pre-install AMIs before shipping for edge processing
- Supports AWS IoT Greengrass for IoT edge workloads
- Cluster up to 5–10 devices for increased storage and compute
When to Use Which
| Scenario | Recommended Service |
|---|---|
| Ongoing file sync to S3/EFS/FSx | DataSync |
| One-time migration ≤ few TB over good network | DataSync |
| One-time migration 10–80 TB, limited bandwidth | Snowball Edge |
| Migration > 100 TB, weeks of network transfer | Snowball Edge (multiple) |
| Petabyte-to-exabyte scale migration | Snowmobile |
| Edge data collection in harsh environments | Snowcone |
| Edge ML inference without cloud connectivity | Snowball Edge Compute |
| Cross-region or cross-account EFS/S3 sync | DataSync (agentless) |
Common Use Cases
- Data center migration — Ship multiple Snowball Edge devices to transfer hundreds of TB to S3 when network bandwidth is insufficient.
- Ongoing NFS/SMB sync — DataSync agent continuously replicates on-premises file shares to EFS or FSx for hybrid access.
- Disaster recovery data staging — DataSync scheduled tasks to keep S3 copies of on-premises data for DR.
- Edge data collection — Snowcone deployed at remote sites (ships, oil rigs, field hospitals) to collect and ship data.
- Cloud-to-cloud transfer — DataSync moves data between Azure Blob / GCS and Amazon S3 without on-premises infrastructure.
SAA/SAP Exam Tips
SAA Tip: "Transfer large amounts of data offline" or "limited network bandwidth" → Snow Family. "Ongoing file synchronization" or "scheduled data transfer" → DataSync.
Exam Trap: Snowball cannot transfer directly to Glacier. Data lands in S3 first, then an S3 Lifecycle policy transitions it to Glacier.
SAP Tip: DataSync can transfer between AWS storage services (S3 ↔ EFS, S3 ↔ FSx, cross-region S3 ↔ S3) without an on-premises agent — useful for cross-account and cross-region replication scenarios.
Cross-Cloud Equivalents
| Provider | Service / Solution | Notes |
|---|---|---|
| AWS | AWS DataSync / Snow Family | Baseline |
| Azure | Azure Data Box (offline) / Azure File Sync (online) | Data Box family mirrors Snow; File Sync ≈ DataSync |
| GCP | Transfer Appliance (offline) / Storage Transfer Service (online) | Similar split of online/offline |
| On-Premises | Rsync, Robocopy, Aspera, Signiant | Traditional file transfer tools |
Pricing Model
| Dimension | Unit | Notes |
|---|---|---|
| DataSync: data transferred | Per GB | Source → destination; charged at origin |
| Snowball Edge: service fee | Per device per job | 10-day included use; daily charges after |
| Snowball Edge: shipping | Per shipment | Varies by region and shipping speed |
| Snowcone: service fee | Per device per job | Smaller device, lower fee than Snowball |
| Snowmobile | Custom pricing | Engagement-based |
Related Services / See Also
- Amazon S3 — primary destination for Snow Family and DataSync transfers
- AWS DMS — for database-specific migrations (not file transfers)
- Amazon EFS and FSx — DataSync targets for file system synchronization
- AWS Storage Gateway — hybrid storage with caching (different from DataSync's transfer focus)
AWS Step Functions
Serverless orchestration — visual workflows that coordinate AWS services using state machines defined in Amazon States Language (ASL).
AWS DMS
Database migration — migrate databases to AWS with minimal downtime using continuous data replication and optional schema conversion.
Built by Fred Cheung @CookedRicer · Powered by Fumadocs & Github Copilot