# Migrate from DynamoDB This page explains how to fill the `source` and `target` properties of the [configuration file](https://migrator.docs.scylladb.com/stable/configuration.md) to migrate data: - from a DynamoDB table, a ScyllaDB Alternator table, or a [DynamoDB S3 export](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/S3DataExport.HowItWorks.html), - to a DynamoDB table or a ScyllaDB Alternator table. In file `config.yaml`, make sure to keep only one `source` property and one `target` property, and configure them as explained in the following subsections according to your case. ## Configuring the Source The data `source` can be a DynamoDB table, an Alternator table, or a DynamoDB S3 export. ### Reading from DynamoDB To read from DynamoDB, use the source type `dynamodb`. Here is a minimal `source` configuration: ```yaml source: type: dynamodb table: region: ``` Where `
` is the name of the table to read, and `` is the AWS region where the DynamoDB instance is located. For custom AWS-compatible endpoints such as DynamoDB Local or LocalStack, you can also provide an `endpoint`: ```yaml source: type: dynamodb table:
endpoint: host: http:// port: ``` Where `` and `` should be replaced with the host name and TCP port of your custom DynamoDB-compatible endpoint. ### Reading from Alternator To read from ScyllaDB Alternator, use the source type `alternator` and provide an `endpoint`: ```yaml source: type: alternator table:
endpoint: host: http:// port: ``` Where `` and `` should be replaced with the host name and TCP port of your Alternator instance. The `endpoint.host` value must include the protocol prefix (`http://` or `https://`) for Alternator. In practice, your source database may require authentication. For DynamoDB: ```yaml source: type: dynamodb table:
region: credentials: accessKey: secretKey: ``` Where `` and `` should be replaced with your actual AWS access key and secret key. For Alternator: ```yaml source: type: alternator table:
endpoint: host: http:// port: credentials: accessKey: secretKey: ``` The Migrator also supports advanced AWS authentication options such as using [AssumeRole](https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html). Please read the [configuration reference](https://migrator.docs.scylladb.com/stable/configuration.md#aws-authentication) for more details. Last, you can provide the following optional properties: ```yaml source: # ... same as above # Split factor for reading. The default is to split the source data into chunks # of 128 MB that can be processed in parallel by the Spark executors. scanSegments: 1 # Throttling settings, set based on your database capacity (or wanted capacity) readThroughput: 1 # Can be between 0.1 and 1.5, inclusively. # 0.5 represents the default read rate, meaning that the job will attempt to consume half of the read capacity of the table. # If you increase the value above 0.5, spark will increase the request rate; decreasing the value below 0.5 decreases the read request rate. # (The actual read rate will vary, depending on factors such as whether there is a uniform key distribution in the DynamoDB table.) throughputReadPercent: 1.0 # At most how many tasks per Spark executor? The default is to use the same as 'scanSegments'. maxMapTasks: 1 ``` ### Reading a DynamoDB S3 Export To read the content of a [DynamoDB table exported to S3](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/S3DataExport.HowItWorks.html), use the `source` type `dynamodb-s3-export`. Here is a minimal source configuration: ```yaml source: type: dynamodb-s3-export # Name of the S3 bucket where the DynamoDB table has been exported bucket: # Key of the `manifest-summary.json` object in the bucket manifestKey: # Key schema and attribute definitions, see https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_TableCreationParameters.html tableDescription: # See https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_AttributeDefinition.html attributeDefinitions: - name: type: # ... other attributes # See https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_KeySchemaElement.html keySchema: - name: type: # ... other key schema definitions ``` Where ``, ``, ``, ``, ``, and `` should be replaced with your specific values. Here is a concrete example assuming the export manifest summary URL is `s3://my-dynamodb-exports/my-path/AWSDynamoDB/01234567890123-1234abcd/manifest-summary.json`, and the table only uses a single text attribute `id` as a partition key: ```yaml source: type: dynamodb-s3-export bucket: my-dynamodb-exports manifestKey: my-path/AWSDynamoDB/01234567890123-1234abcd/manifest-summary.json tableDescription: attributeDefinitions: - name: id type: S keySchema: - name: id type: HASH ``` Additionally, you can provide the following optional properties: ```yaml source: # ... same as above # Connect to a custom endpoint instead of the standard AWS S3 endpoint endpoint: # Specify the hostname without a protocol host: port: # AWS availability region region: # Connection credentials: credentials: accessKey: secretKey: # Whether to use “path-style access” in S3 (see https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html). Default is false. usePathStyleAccess: true ``` Where ``, ``, ``, ``, and `` should be replaced with your specific values. The Migrator also supports advanced AWS authentication options such as using [AssumeRole](https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html). Please read the [configuration reference](https://migrator.docs.scylladb.com/stable/configuration.md#aws-authentication) for more details. ## Configuring the Destination The migration `target` can be DynamoDB or Alternator. ### Writing to DynamoDB To write to DynamoDB, use the target type `dynamodb`: ```yaml target: type: dynamodb # Name of the table to write. If it does not exist, it will be created on the fly. table:
# When transferring DynamoDB sources to DynamoDB-like targets (DynamoDB or Alternator), # the migrator supports transferring live changes occurring on the source table after transferring an initial # snapshot. # Please see the documentation page “Stream Changes” for more details about this option. streamChanges: false ``` Where `
` should be replaced with your specific value. Additionally, you can also set the following optional properties: ```yaml target: # ... same as above # Connect to a custom endpoint. endpoint: host: port: # AWS availability region: region: # Authentication credentials: credentials: accessKey: secretKey: # Throttling settings, set based on your database write capacity units (or wanted capacity). # By default, for provisioned tables we use the configured write capacity units, and for on-demand tables we use the value 40000. writeThroughput: 1 # Can be between 0.1 and 1.5, inclusively. # 0.5 represents the default write rate, meaning that the job will attempt to consume half of the write capacity of the table. # If you increase the value above 0.5, spark will increase the request rate; decreasing the value below 0.5 decreases the write request rate. # (The actual write rate will vary, depending on factors such as whether there is a uniform key distribution in the DynamoDB table.) throughputWritePercent: 1.0 # When streamChanges is true, skip the initial snapshot transfer and only stream changes. # This setting is ignored if streamChanges is false. skipInitialSnapshotTransfer: false ``` Where ``, ``, ``, ``, and `` are replaced with your specific values. ### Writing to Alternator To write to ScyllaDB Alternator, use the target type `alternator`: ```yaml target: type: alternator table:
endpoint: host: http:// port: streamChanges: false ``` The `endpoint.host` value must include the protocol prefix (`http://` or `https://`) for Alternator. In addition to the common DynamoDB-style properties shown above, Alternator targets also support: ```yaml target: type: alternator # ... same as above # Remove consumed capacity headers from requests. Defaults to true for Alternator. removeConsumedCapacity: true # Optional Alternator-specific routing / client tuning: datacenter: dc1 rack: rack1 activeRefreshIntervalMs: 1000 idleRefreshIntervalMs: 60000 compression: false optimizeHeaders: false maxConnections: 400 connectionMaxIdleTimeMs: 600000 connectionTimeToLiveMs: 0 connectionAcquisitionTimeoutMs: 10000 connectionTimeoutMs: 15000 ``` The Migrator also supports advanced AWS authentication options such as using [AssumeRole](https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html). Please read the [configuration reference](https://migrator.docs.scylladb.com/stable/configuration.md#aws-authentication) for more details.