Migrate media between storage providers

View Markdown

Moving media between providers usually means downloading everything, re-encoding, then re-uploading. Ittybit handles the transfer and transcoding in one step — read from one connection, write to another.

“Launchpad” (a video startup like Mux) is migrating from AWS S3 to Cloudflare R2. They want to re-encode to h264 during the move to cut storage costs.

Set up connections

Register both the source and destination buckets:

# Source: AWS S3
ittybit connections add s3 \
  --name aws-source \
  --endpoint https://s3.us-east-1.amazonaws.com \
  --region us-east-1 \
  --access-key-id $AWS_ACCESS_KEY_ID \
  --secret-access-key $AWS_SECRET_ACCESS_KEY

# Destination: Cloudflare R2
ittybit connections add s3 \
  --name r2-dest \
  --endpoint https://<account-id>.r2.cloudflarestorage.com \
  --region auto \
  --access-key-id $R2_ACCESS_KEY_ID \
  --secret-access-key $R2_SECRET_ACCESS_KEY

API

Read from the S3 connection and write to the R2 connection. Transcode to h264 during the move:

ittybit video \
  -i s3://launchpad-media/originals/interview.mov \
  -o s3://launchpad-r2/media/interview.mp4 \
  --input-connection aws-source \
  --output-connection r2-dest \
  --codec h264 \
  --width 1920 \
  --format mp4
const task = {
  input: 's3://launchpad-media/originals/interview.mov',
  kind: 'video',
  options: {
    input_connection_id: 'conn_src456',
    output_connection_id: 'conn_dst789',
    codec: 'h264',
    width: 1920,
    format: 'mp4',
  },
  output: 's3://launchpad-r2/media/interview.mp4',
};

const res = await fetch('https://api.ittybit.com/jobs', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.ITTYBIT_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify(task),
});
const data = await res.json();
import requests

task = {
    "input": "s3://launchpad-media/originals/interview.mov",
    "kind": "video",
    "options": {
        "input_connection_id": "conn_src456",
        "output_connection_id": "conn_dst789",
        "codec": "h264",
        "width": 1920,
        "format": "mp4",
    },
    "output": "s3://launchpad-r2/media/interview.mp4",
}

res = requests.post(
    "https://api.ittybit.com/jobs",
    headers={"Authorization": f"Bearer {api_key}"},
    json=task,
)
data = res.json()
TASK='{
  "input": "s3://launchpad-media/originals/interview.mov",
  "kind": "video",
  "options": {
    "input_connection_id": "conn_src456",
    "output_connection_id": "conn_dst789",
    "codec": "h264",
    "width": 1920,
    "format": "mp4"
  },
  "output": "s3://launchpad-r2/media/interview.mp4"
}'

curl -X POST https://api.ittybit.com/jobs \
  -H "Authorization: Bearer $ITTYBIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d "$TASK"

Ittybit reads from S3 using aws-source credentials, transcodes to h264, and writes the result to R2 using r2-dest credentials.

CLI

ittybit video \
  -i s3://launchpad-media/originals/interview.mov \
  -o s3://launchpad-r2/media/interview.mp4 \
  --input-connection aws-source \
  --output-connection r2-dest \
  --codec h264 \
  --width 1920

Copy without re-encoding

If you just want to move files without transcoding, omit the codec and quality options:

ittybit video \
  -i s3://launchpad-media/originals/interview.mp4 \
  -o s3://launchpad-r2/media/interview.mp4 \
  --input-connection aws-source \
  --output-connection r2-dest
const task = {
  input: 's3://launchpad-media/originals/interview.mp4',
  kind: 'video',
  options: {
    input_connection_id: 'conn_src456',
    output_connection_id: 'conn_dst789',
  },
  output: 's3://launchpad-r2/media/interview.mp4',
};

const res = await fetch('https://api.ittybit.com/jobs', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.ITTYBIT_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify(task),
});
const data = await res.json();
import requests

task = {
    "input": "s3://launchpad-media/originals/interview.mp4",
    "kind": "video",
    "options": {
        "input_connection_id": "conn_src456",
        "output_connection_id": "conn_dst789",
    },
    "output": "s3://launchpad-r2/media/interview.mp4",
}

res = requests.post(
    "https://api.ittybit.com/jobs",
    headers={"Authorization": f"Bearer {api_key}"},
    json=task,
)
data = res.json()
TASK='{
  "input": "s3://launchpad-media/originals/interview.mp4",
  "kind": "video",
  "options": {
    "input_connection_id": "conn_src456",
    "output_connection_id": "conn_dst789"
  },
  "output": "s3://launchpad-r2/media/interview.mp4"
}'

curl -X POST https://api.ittybit.com/jobs \
  -H "Authorization: Bearer $ITTYBIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d "$TASK"

Batch migration

Loop over a list of files to migrate an entire bucket:

for file in interview.mov keynote.mov demo.mov; do
  ittybit video \
    -i "s3://launchpad-media/originals/$file" \
    -o "s3://launchpad-r2/media/${file%.mov}.mp4" \
    --input-connection aws-source \
    --output-connection r2-dest \
    --codec h264 \
    --format mp4
done

See also