Set Up a Spark Cluster with Docker¶

This page describes how to set up a Spark cluster locally on your machine by using Docker containers. It requires Docker and Git.

1. Clone the Migrator repository.

   git clone https://github.com/scylladb/scylla-migrator.git
   cd scylla-migrator
   

2. Download the latest release of the scylla-migrator-assembly.jar and put it in the directory migrator/target/scala-2.13/.

   mkdir -p migrator/target/scala-2.13
   wget https://github.com/scylladb/scylla-migrator/releases/latest/download/scylla-migrator-assembly.jar \
     --directory-prefix=migrator/target/scala-2.13
   

   Alternatively, download a specific release of scylla-migrator-assembly.jar.

3. Start the Spark cluster.

   docker compose up -d
   

4. Open the Spark web UI.

   http://localhost:8080

5. Rename the file config.yaml.example to config.yaml, and configure it according to your needs.

6. Finally, run the migration.

   docker compose exec spark-master /spark/bin/spark-submit --class com.scylladb.migrator.Migrator \
     --master spark://spark-master:7077 \
     --conf spark.driver.host=spark-master \
     --conf spark.scylla.config=/app/config.yaml \
     <... other arguments> \
     /jars/scylla-migrator-assembly.jar
   

   The spark-master container mounts the ./migrator/target/scala-2.13 dir on /jars and the repository root on /app.

   See a complete description of the expected arguments to spark-submit in page Run the Migration, and replace “<… other arguments>” above with the appropriate arguments.

7. You can monitor progress by observing the Spark web console you opened in step 4. Additionally, after the job has started, you can track progress via http://localhost:4040.

   FYI: When no Spark jobs are actively running, the Spark progress page at port 4040 displays unavailable. It is only useful and renders when a Spark job is in progress.