# Batch processing with Laravel Artisan and Ittybit

Build an Artisan command to batch-process media files via Ittybit with rate limiting

You have thousands of media files in your database that need transcoding -- legacy uploads stuck in source format, a backlog from a migration, or an entire library that was never optimized. Installing FFmpeg on your server and writing a processing pipeline is a project in itself. Instead, you can write a single Artisan command that reads URLs from the database, dispatches Ittybit tasks in rate-limited batches, polls for completion, and updates each record when it finishes.

## Migration

Add columns to track processing state on your existing media table.

```bash
php artisan make:migration add_ittybit_columns_to_media_table
```

```php
// database/migrations/xxxx_add_ittybit_columns_to_media_table.php
Schema::table('media', function (Blueprint $table) {
    $table->string('ittybit_task_id')->nullable()->after('url');
    $table->string('processing_status')->default('pending')->after('ittybit_task_id');
    $table->string('output_url')->nullable()->after('processing_status');
    $table->timestamp('processed_at')->nullable()->after('output_url');
});
```

```php
// app/Models/Media.php
namespace App\Models;

use Illuminate\Database\Eloquent\Model;

class Media extends Model
{
    protected $fillable = [
        'url',
        'ittybit_task_id',
        'processing_status',
        'output_url',
        'processed_at',
    ];

    protected $casts = [
        'processed_at' => 'datetime',
    ];

    public function scopeUnprocessed($query)
    {
        return $query->where('processing_status', 'pending');
    }

    public function scopeDispatched($query)
    {
        return $query->where('processing_status', 'dispatched');
    }
}
```

## Configuration

Add your Ittybit API key to `.env` and register it in config.

```bash
ITTYBIT_API_KEY=your_ittybit_api_key
```

```php
// config/services.php
'ittybit' => [
    'api_key' => env('ITTYBIT_API_KEY'),
],
```

## The Artisan command

Generate the command scaffold, then replace it with the full implementation below.

```bash
php artisan make:command BatchProcessMedia
```

```php
// app/Console/Commands/BatchProcessMedia.php
namespace App\Console\Commands;

use App\Models\Media;
use Illuminate\Console\Command;
use Illuminate\Support\Facades\Http;
use Illuminate\Support\Collection;

class BatchProcessMedia extends Command
{
    protected $signature = 'media:batch-process
        {--batch-size=10 : Number of tasks to dispatch per batch}
        {--delay=2 : Seconds to wait between batches}
        {--poll-interval=5 : Seconds between status checks}
        {--limit=0 : Max items to process (0 = all)}
        {--dry-run : Show what would be processed without dispatching}';

    protected $description = 'Batch-process unprocessed media files via Ittybit';

    private string $apiBase = 'https://api.ittybit.com';

    public function handle(): int
    {
        $batchSize = (int) $this->option('batch-size');
        $delay = (int) $this->option('delay');
        $pollInterval = (int) $this->option('poll-interval');
        $limit = (int) $this->option('limit');
        $dryRun = $this->option('dry-run');

        $query = Media::unprocessed()->orderBy('id');

        if ($limit > 0) {
            $query->limit($limit);
        }

        $items = $query->get();

        if ($items->isEmpty()) {
            $this->info('No unprocessed media found.');
            return self::SUCCESS;
        }

        $this->info("Found {$items->count()} unprocessed media files.");

        if ($dryRun) {
            $items->each(fn ($m) => $this->line("  Would process: {$m->url}"));
            return self::SUCCESS;
        }

        // Phase 1: Dispatch tasks in batches
        $dispatched = $this->dispatchBatches($items, $batchSize, $delay);

        if ($dispatched->isEmpty()) {
            $this->error('No tasks were dispatched successfully.');
            return self::FAILURE;
        }

        $this->newLine();
        $this->info("Dispatched {$dispatched->count()} tasks. Polling for results...");
        $this->newLine();

        // Phase 2: Poll for completion
        $results = $this->pollForCompletion($dispatched, $pollInterval);

        // Summary
        $this->newLine();
        $completed = $results->where('status', 'completed')->count();
        $failed = $results->where('status', 'failed')->count();
        $this->info("Done. {$completed} completed, {$failed} failed.");

        return $failed > 0 ? self::FAILURE : self::SUCCESS;
    }

    private function dispatchBatches(Collection $items, int $batchSize, int $delay): Collection
    {
        $dispatched = collect();
        $batches = $items->chunk($batchSize);
        $batchCount = $batches->count();

        $bar = $this->output->createProgressBar($items->count());
        $bar->setFormat(' Dispatching %current%/%max% [%bar%] %percent:3s%%');
        $bar->start();

        foreach ($batches as $index => $batch) {
            foreach ($batch as $media) {
                $taskId = $this->dispatchTask($media);

                if ($taskId) {
                    $media->update([
                        'ittybit_task_id' => $taskId,
                        'processing_status' => 'dispatched',
                    ]);
                    $dispatched->push($media->fresh());
                } else {
                    $media->update(['processing_status' => 'failed']);
                    $this->newLine();
                    $this->warn("Failed to dispatch task for media #{$media->id}");
                }

                $bar->advance();
            }

            // Rate-limit: sleep between batches (skip after the last one)
            if ($index < $batchCount - 1) {
                sleep($delay);
            }
        }

        $bar->finish();
        $this->newLine();

        return $dispatched;
    }

    private function dispatchTask(Media $media): ?string
    {
        try {
            $response = Http::withToken(config('services.ittybit.api_key'))
                ->timeout(30)
                ->post("{$this->apiBase}/tasks", [
                    'input' => $media->url,
                    'kind' => 'video',
                    'options' => [
                        'width' => 1920,
                        'format' => 'mp4',
                        'quality' => 'high',
                    ],
                ]);

            if ($response->successful()) {
                return $response->json('id');
            }

            $this->newLine();
            $this->warn("API error for media #{$media->id}: {$response->status()}");
            return null;
        } catch (\Exception $e) {
            $this->newLine();
            $this->warn("Request failed for media #{$media->id}: {$e->getMessage()}");
            return null;
        }
    }

    private function pollForCompletion(Collection $dispatched, int $pollInterval): Collection
    {
        $pending = $dispatched->keyBy('id');
        $results = collect();

        $bar = $this->output->createProgressBar($pending->count());
        $bar->setFormat(' Processing  %current%/%max% [%bar%] %percent:3s%%');
        $bar->start();

        while ($pending->isNotEmpty()) {
            foreach ($pending as $id => $media) {
                $status = $this->checkTaskStatus($media->ittybit_task_id);

                if ($status === null) {
                    continue;
                }

                if ($status['done']) {
                    if ($status['status'] === 'completed') {
                        $media->update([
                            'processing_status' => 'completed',
                            'output_url' => $status['output_url'],
                            'processed_at' => now(),
                        ]);
                    } else {
                        $media->update(['processing_status' => 'failed']);
                    }

                    $results->push([
                        'id' => $media->id,
                        'status' => $status['status'],
                    ]);

                    $pending->forget($id);
                    $bar->advance();
                }
            }

            if ($pending->isNotEmpty()) {
                sleep($pollInterval);
            }
        }

        $bar->finish();

        return $results;
    }

    private function checkTaskStatus(string $taskId): ?array
    {
        try {
            $response = Http::withToken(config('services.ittybit.api_key'))
                ->timeout(15)
                ->get("{$this->apiBase}/tasks/{$taskId}");

            if (!$response->successful()) {
                return null;
            }

            $data = $response->json();
            $status = $data['status'];

            return [
                'done' => in_array($status, ['completed', 'failed', 'error']),
                'status' => $status,
                'output_url' => $data['output']['url'] ?? null,
            ];
        } catch (\Exception $e) {
            return null;
        }
    }
}
```

## Running the command

Process everything with defaults (batches of 10, 2-second delay between batches):

```bash
php artisan media:batch-process
```

Preview what would be dispatched without making any API calls:

```bash
php artisan media:batch-process --dry-run
```

Tune batch size and rate limiting for larger runs:

```bash
php artisan media:batch-process --batch-size=25 --delay=5 --limit=500
```

The progress bar shows two phases: dispatching and polling. If the command is interrupted, re-running it picks up where it left off -- items already marked `dispatched` or `completed` are skipped by the `unprocessed` scope.

## Resuming interrupted runs

If the command exits after dispatching but before polling completes, the dispatched items are still tracked in the database. Add a second command or extend the existing one with a `--resume` flag to pick those up.

```php
// Add to BatchProcessMedia.php signature
protected $signature = 'media:batch-process
    {--batch-size=10 : Number of tasks to dispatch per batch}
    {--delay=2 : Seconds to wait between batches}
    {--poll-interval=5 : Seconds between status checks}
    {--limit=0 : Max items to process (0 = all)}
    {--dry-run : Show what would be processed without dispatching}
    {--resume : Poll for already-dispatched tasks instead of dispatching new ones}';

// Add to the top of handle()
if ($this->option('resume')) {
    $dispatched = Media::dispatched()->get();
    if ($dispatched->isEmpty()) {
        $this->info('No dispatched tasks to resume.');
        return self::SUCCESS;
    }
    $this->info("Resuming {$dispatched->count()} dispatched tasks...");
    $results = $this->pollForCompletion($dispatched, $pollInterval);
    $completed = $results->where('status', 'completed')->count();
    $failed = $results->where('status', 'failed')->count();
    $this->info("Done. {$completed} completed, {$failed} failed.");
    return $failed > 0 ? self::FAILURE : self::SUCCESS;
}
```

```bash
php artisan media:batch-process --resume
```

## Scheduling

Run the command nightly to process any new uploads that accumulated during the day.

```php
// app/Console/Kernel.php
protected function schedule(Schedule $schedule)
{
    $schedule->command('media:batch-process --batch-size=25 --delay=3')
        ->dailyAt('02:00')
        ->withoutOverlapping()
        ->appendOutputTo(storage_path('logs/batch-process.log'));
}
```

## See also

- [Video upload pipeline with Laravel](/guides/video-upload-pipeline-with-laravel) -- single-file upload flow with webhooks
- [Build a user upload pipeline](/guides/build-a-user-upload-pipeline) -- multi-step processing pipeline
- [Process files from S3](/guides/process-files-from-s3) -- use S3 URLs as task inputs