Shopify Webhooks at Scale: A Laravel Queue Architecture That Handles 10K Events/Minute

Series: Shopify Type: Tutorial Meta Description: Production-grade Shopify webhook processing with Laravel Horizon, Redis queues, automatic retries, dead-letter queues, and monitoring dashboards. Handles 10,000 webhook events per minute. Keywords: Shopify webhooks, Laravel Horizon, Redis queue, webhook processing, Laravel queue architecture Word Count Target: 2500 Published: Draft — NOT for publication

---

The Webhook Challenge

Shopify apps live on webhooks. Every order created, every product updated, every customer registered — Shopify fires a webhook to your app, and you have 5 seconds to respond with a 200 OK or Shopify considers it a failure. After 19 failures over 48 hours, Shopify removes your webhook subscription entirely.

At small scale, processing webhooks is trivial. A controller receives the payload, does the work synchronously, and returns 200. But when your app serves 3,000 active merchants and a flash sale triggers 10,000 webhook events in a single minute, synchronous processing becomes impossible. Your database connection pool saturates. Response times creep past the 5-second limit. Webhooks start failing. Shopify removes your subscriptions. Your app stops receiving data. Merchants notice. They uninstall.

This tutorial shows the Laravel queue architecture we built to process 10,000 Shopify webhook events per minute with zero dropped events, automatic retries, dead-letter handling, and full observability.

Architecture Overview

Shopify Webhook POST
    |
    v
Nginx / Load Balancer
    |
    v
Laravel Controller (validate HMAC, dispatch job, return 200)
    |
    v
Redis Queue (sorted by priority)
    |
    v
Laravel Horizon Workers (autoscaled)
    |
    v
Job Processes (idempotent, retryable)
    |
    v (on failure)
Dead Letter Queue + Alert System

The critical design principle: the webhook controller does no work except validation and dispatch. It returns 200 in under 50 milliseconds. All actual processing happens asynchronously in queued jobs.

Step 1: Webhook Controller

// app/Http/Controllers/WebhookController.php
namespace App\Http\Controllers;

use Illuminate\Http\Request;
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Facades\RateLimiter;
use App\Jobs\Webhooks\ProcessWebhook;
use App\Services\WebhookHmacValidator;

class WebhookController extends Controller
{
    public function handle(Request $request, WebhookHmacValidator $validator)
    {
        $shopDomain = $request->header('X-Shopify-Shop-Domain');
        $topic = $request->header('X-Shopify-Topic');
        $apiVersion = $request->header('X-Shopify-API-Version');
        $webhookId = $request->header('X-Shopify-Webhook-Id');

        // Verify HMAC signature
        if (!$validator->isValid($request)) {
            Log::warning('Webhook HMAC validation failed', [
                'shop' => $shopDomain,
                'topic' => $topic,
                'webhook_id' => $webhookId,
            ]);
            return response('Invalid signature', 401);
        }

        // Rate limit per shop (prevent a single shop from flooding)
        $rateLimitKey = "webhook:{$shopDomain}:" . floor(time() / 10);
        if (RateLimiter::tooManyAttempts($rateLimitKey, 200)) {
            Log::warning('Webhook rate limit exceeded', [
                'shop' => $shopDomain,
                'topic' => $topic,
            ]);
            return response('Rate limited', 429);
        }
        RateLimiter::hit($rateLimitKey, 10);

        // Determine queue priority based on topic
        $queue = $this->resolveQueue($topic);

        // Dispatch the processing job
        ProcessWebhook::dispatch(
            shopDomain: $shopDomain,
            topic: $topic,
            payload: $request->all(),
            webhookId: $webhookId,
            apiVersion: $apiVersion,
        )->onQueue($queue);

        return response('OK', 200);
    }

    private function resolveQueue(string $topic): string
    {
        return match (true) {
            str_contains($topic, 'app/uninstall') => 'webhook-critical',
            str_contains($topic, 'app/subscription') => 'webhook-critical',
            str_contains($topic, 'orders/') => 'webhook-high',
            str_contains($topic, 'customers/') => 'webhook-high',
            str_contains($topic, 'products/') => 'webhook-default',
            str_contains($topic, 'inventory/') => 'webhook-low',
            default => 'webhook-default',
        };
    }
}

The HMAC validator service:

// app/Services/WebhookHmacValidator.php
namespace App\Services;

use Illuminate\Http\Request;

class WebhookHmacValidator
{
    public function isValid(Request $request): bool
    {
        $hmacHeader = $request->header('X-Shopify-Hmac-Sha256');

        if (!$hmacHeader) {
            return false;
        }

        $secret = config('shopify.api_secret');
        $calculatedHmac = base64_encode(
            hash_hmac('sha256', $request->getContent(), $secret, true)
        );

        return hash_equals($calculatedHmac, $hmacHeader);
    }
}

Step 2: The Processing Job with Idempotency

Every webhook must be processed idempotently. Shopify may deliver the same webhook multiple times. If you process an orders/create webhook twice, you create duplicate records. Idempotency keys prevent this.

// app/Jobs/Webhooks/ProcessWebhook.php
namespace App\Jobs\Webhooks;

use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Log;
use App\Models\WebhookEvent;
use App\Services\Webhooks\WebhookProcessorFactory;

class ProcessWebhook implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    // Retry configuration
    public int $tries = 4;
    public int $maxExceptions = 3;
    public int $timeout = 120;
    public bool $failOnTimeout = true;

    // Backoff increases with each retry: 10s, 60s, 300s
    public array $backoff = [10, 60, 300];

    // Discard job if it was created more than 30 minutes ago
    public int $retryUntil = 1800;

    public function __construct(
        public string $shopDomain,
        public string $topic,
        public array $payload,
        public string $webhookId,
        public string $apiVersion,
    ) {
        // Set queue based on the queue name passed during dispatch
        $this->onQueue($this->resolveQueue());
    }

    public function handle(WebhookProcessorFactory $processorFactory): void
    {
        $idempotencyKey = $this->generateIdempotencyKey();

        // Check if we already processed this exact webhook
        $alreadyProcessed = DB::table('processed_webhooks')
            ->where('idempotency_key', $idempotencyKey)
            ->exists();

        if ($alreadyProcessed) {
            Log::debug('Skipping duplicate webhook', [
                'shop' => $this->shopDomain,
                'topic' => $this->topic,
                'webhook_id' => $this->webhookId,
            ]);
            return;
        }

        // Mark as processing (atomic lock to prevent race conditions)
        $locked = DB::table('processed_webhooks')->insertOrIgnore([
            'idempotency_key' => $idempotencyKey,
            'shop_domain' => $this->shopDomain,
            'topic' => $this->topic,
            'webhook_id' => $this->webhookId,
            'status' => 'processing',
            'payload_hash' => md5(json_encode($this->payload)),
            'created_at' => now(),
            'updated_at' => now(),
        ]);

        if (!$locked) {
            // Another worker picked this up
            return;
        }

        try {
            $processor = $processorFactory->make($this->topic);
            $processor->process($this->shopDomain, $this->payload);

            // Mark as completed
            DB::table('processed_webhooks')
                ->where('idempotency_key', $idempotencyKey)
                ->update(['status' => 'completed', 'updated_at' => now()]);

        } catch (\Throwable $e) {
            // Mark as failed (will be retried by queue)
            DB::table('processed_webhooks')
                ->where('idempotency_key', $idempotencyKey)
                ->update([
                    'status' => 'failed',
                    'error_message' => substr($e->getMessage(), 0, 500),
                    'updated_at' => now(),
                ]);

            throw $e; // Re-throw to trigger queue retry
        }
    }

    public function failed(\Throwable $exception): void
    {
        Log::error('Webhook processing failed permanently', [
            'shop' => $this->shopDomain,
            'topic' => $this->topic,
            'webhook_id' => $this->webhookId,
            'error' => $exception->getMessage(),
        ]);

        // Move to dead letter queue for manual inspection
        DB::table('dead_letter_webhooks')->insert([
            'shop_domain' => $this->shopDomain,
            'topic' => $this->topic,
            'payload' => json_encode($this->payload),
            'webhook_id' => $this->webhookId,
            'error_message' => substr($exception->getMessage(), 0, 1000),
            'retry_count' => $this->attempts(),
            'created_at' => now(),
        ]);

        // Alert the team
        event(new \App\Events\WebhookFailedPermanently(
            $this->shopDomain,
            $this->topic,
            $exception->getMessage()
        ));
    }

    private function generateIdempotencyKey(): string
    {
        // X-Shopify-Webhook-Id is unique per delivery attempt
        // Combine with topic and shop for belt-and-suspenders uniqueness
        return md5($this->webhookId . $this->topic . $this->shopDomain);
    }
}

Step 3: Processor Factory and Individual Processors

The factory pattern lets us add new webhook handlers without modifying the core job:

// app/Services/Webhooks/WebhookProcessorFactory.php
namespace App\Services\Webhooks;

use App\Services\Webhooks\Processors\{
    OrderCreatedProcessor,
    OrderUpdatedProcessor,
    ProductUpdatedProcessor,
    ProductDeletedProcessor,
    CustomerCreatedProcessor,
    AppUninstalledProcessor,
    InventoryUpdateProcessor,
};

class WebhookProcessorFactory
{
    private array $processors = [
        'orders/create' => OrderCreatedProcessor::class,
        'orders/updated' => OrderUpdatedProcessor::class,
        'products/update' => ProductUpdatedProcessor::class,
        'products/delete' => ProductDeletedProcessor::class,
        'customers/create' => CustomerCreatedProcessor::class,
        'app/uninstalled' => AppUninstalledProcessor::class,
        'inventory_levels/update' => InventoryUpdateProcessor::class,
    ];

    public function make(string $topic): WebhookProcessorInterface
    {
        $processorClass = $this->processors[$topic] ?? null;

        if (!$processorClass) {
            throw new \InvalidArgumentException("No processor for topic: {$topic}");
        }

        return app($processorClass);
    }
}

// app/Services/Webhooks/WebhookProcessorInterface.php
namespace App\Services\Webhooks;

interface WebhookProcessorInterface
{
    public function process(string $shopDomain, array $payload): void;
}

An example processor for order creation:

// app/Services/Webhooks/Processors/OrderCreatedProcessor.php
namespace App\Services\Webhooks\Processors;

use App\Services\Webhooks\WebhookProcessorInterface;
use App\Models\Shop;
use App\Models\Order;
use App\Services\Analytics\OrderAnalyticsService;
use Illuminate\Support\Facades\Log;

class OrderCreatedProcessor implements WebhookProcessorInterface
{
    public function __construct(
        private OrderAnalyticsService $analytics,
    ) {}

    public function process(string $shopDomain, array $payload): void
    {
        $shop = Shop::where('domain', $shopDomain)->first();

        if (!$shop) {
            Log::warning("Shop not found for webhook: {$shopDomain}");
            return;
        }

        $orderId = $payload['id'];
        $orderNumber = $payload['order_number'];
        $totalPrice = (float) $payload['total_price'];
        $currency = $payload['currency'];
        $customerId = $payload['customer']['id'] ?? null;
        $lineItems = $payload['line_items'] ?? [];

        // Upsert the order
        Order::updateOrCreate(
            [
                'shop_id' => $shop->id,
                'shopify_order_id' => $orderId,
            ],
            [
                'order_number' => $orderNumber,
                'total_price' => $totalPrice,
                'currency' => $currency,
                'customer_id' => $customerId,
                'line_items_count' => count($lineItems),
                'status' => $payload['financial_status'],
                'raw_payload' => $payload,
                'ordered_at' => $payload['created_at'],
            ]
        );

        // Update analytics
        $this->analytics->recordOrder($shop, [
            'order_id' => $orderId,
            'total' => $totalPrice,
            'items' => count($lineItems),
        ]);

        Log::info("Order processed: {$orderNumber} for {$shopDomain}");
    }
}

Step 4: Laravel Horizon Configuration

Horizon provides the dashboard, autoscaling, and monitoring for your queue workers. Here is the production configuration:

// config/horizon.php
use Illuminate\Support\Str;

return [
    'domain' => env('HORIZON_DOMAIN'),
    'path' => 'horizon',
    'middleware' => ['web', 'auth.admin'],

    'waits' => [
        'redis:webhook-critical' => 5,
        'redis:webhook-high' => 15,
        'redis:webhook-default' => 30,
        'redis:webhook-low' => 60,
    ],

    'trim' => [
        'recent' => 60,      // Keep recent jobs for 60 minutes
        'pending' => 60,
        'completed' => 60,
        'recent_failed' => 10080, // Keep failed jobs for 7 days
        'failed' => 10080,
        'monitored' => 10080,
    ],

    'environments' => [
        'production' => [
            'supervisor-webhook-critical' => [
                'connection' => 'redis',
                'queue' => ['webhook-critical'],
                'balance' => 'auto',
                'autoScalingStrategy' => 'time',     // Scale based on wait time
                'balanceMaxShift' => 1,
                'balanceCooldown' => 3,
                'minProcesses' => 3,
                'maxProcesses' => 10,
                'balanceEvery' => 2,
                'tries' => 4,
                'timeout' => 60,
            ],
            'supervisor-webhook-high' => [
                'connection' => 'redis',
                'queue' => ['webhook-high'],
                'balance' => 'auto',
                'autoScalingStrategy' => 'time',
                'balanceMaxShift' => 2,
                'balanceCooldown' => 3,
                'minProcesses' => 5,
                'maxProcesses' => 20,
                'balanceEvery' => 2,
                'tries' => 4,
                'timeout' => 90,
            ],
            'supervisor-webhook-default' => [
                'connection' => 'redis',
                'queue' => ['webhook-default'],
                'balance' => 'auto',
                'autoScalingStrategy' => 'time',
                'balanceMaxShift' => 3,
                'balanceCooldown' => 3,
                'minProcesses' => 5,
                'maxProcesses' => 30,
                'balanceEvery' => 2,
                'tries' => 3,
                'timeout' => 120,
            ],
            'supervisor-webhook-low' => [
                'connection' => 'redis',
                'queue' => ['webhook-low'],
                'balance' => 'auto',
                'autoScalingStrategy' => 'size',    // Scale based on queue size
                'balanceMaxShift' => 2,
                'balanceCooldown' => 5,
                'minProcesses' => 2,
                'maxProcesses' => 15,
                'balanceEvery' => 5,
                'tries' => 2,
                'timeout' => 300,
            ],
        ],
    ],
];

The key settings:

• autoScalingStrategy 'time' scales up when jobs wait too long. Critical webhooks should never wait more than 5 seconds. Default webhooks can wait up to 30 seconds.
• autoScalingStrategy 'size' scales up when the queue depth grows. Low-priority webhooks do not need low latency, but they should not accumulate unboundedly.
• balanceMaxShift controls how aggressively Horizon adds or removes processes. We keep it low (1-3) to avoid oscillation.
• minProcesses ensures a baseline capacity even when traffic is zero. Critical and high queues always have warm workers.

Step 5: Dead Letter Queue and Failure Handling

When a job fails all retry attempts, it lands in the dead letter table. We built a dashboard and automated alerting around this.

Dead Letter Dashboard

// app/Http/Controllers/Admin/DeadLetterController.php
namespace App\Http\Controllers\Admin;

use Illuminate\Http\Request;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Artisan;

class DeadLetterController extends Controller
{
    public function index(Request $request)
    {
        $webhooks = DB::table('dead_letter_webhooks')
            ->when($request->shop, fn ($q, $shop) => $q->where('shop_domain', $shop))
            ->when($request->topic, fn ($q, $topic) => $q->where('topic', $topic))
            ->orderBy('created_at', 'desc')
            ->paginate(50);

        $summary = [
            'total' => DB::table('dead_letter_webhooks')->count(),
            'last_24h' => DB::table('dead_letter_webhooks')
                ->where('created_at', '>=', now()->subDay())->count(),
            'by_topic' => DB::table('dead_letter_webhooks')
                ->select('topic', DB::raw('count(*) as count'))
                ->groupBy('topic')
                ->orderByDesc('count')
                ->get(),
        ];

        return view('admin.dead-letter', compact('webhooks', 'summary'));
    }

    public function retry(int $id)
    {
        $webhook = DB::table('dead_letter_webhooks')->where('id', $id)->first();

        if (!$webhook) {
            return back()->with('error', 'Webhook not found.');
        }

        // Re-dispatch the original job
        ProcessWebhook::dispatch(
            shopDomain: $webhook->shop_domain,
            topic: $webhook->topic,
            payload: json_decode($webhook->payload, true),
            webhookId: $webhook->webhook_id,
            apiVersion: '2025-07',
        );

        // Remove from dead letter queue
        DB::table('dead_letter_webhooks')->where('id', $id)->delete();

        return back()->with('status', 'Webhook retried successfully.');
    }

    public function retryByTopic(string $topic)
    {
        $count = DB::table('dead_letter_webhooks')
            ->where('topic', $topic)
            ->count();

        $webhooks = DB::table('dead_letter_webhooks')
            ->where('topic', $topic)
            ->get();

        foreach ($webhooks as $webhook) {
            ProcessWebhook::dispatch(
                shopDomain: $webhook->shop_domain,
                topic: $webhook->topic,
                payload: json_decode($webhook->payload, true),
                webhookId: $webhook->webhook_id,
                apiVersion: '2025-07',
            );
        }

        DB::table('dead_letter_webhooks')->where('topic', $topic)->delete();

        return back()->with('status', "Retried {$count} webhooks for topic: {$topic}");
    }
}

Alerting

We use Laravel's event system to send alerts when webhooks fail permanently:

// app/Providers/EventServiceProvider.php
namespace App\Providers;

use Illuminate\Foundation\Support\Providers\EventServiceProvider as ServiceProvider;
use App\Events\WebhookFailedPermanently;
use App\Listeners\SendWebhookFailureAlert;
use App\Listeners\IncrementFailureMetrics;

class EventServiceProvider extends ServiceProvider
{
    protected $listen = [
        WebhookFailedPermanently::class => [
            SendWebhookFailureAlert::class,
            IncrementFailureMetrics::class,
        ],
    ];
}

// app/Listeners/SendWebhookFailureAlert.php
namespace App\Listeners;

use App\Events\WebhookFailedPermanently;
use Illuminate\Support\Facades\Notification;
use App\Notifications\WebhookFailureSlackNotification;

class SendWebhookFailureAlert
{
    public function handle(WebhookFailedPermanently $event): void
    {
        // Send to Slack
        Notification::route('slack', config('services.slack.webhook_url'))
            ->notify(new WebhookFailureSlackNotification(
                $event->shopDomain,
                $event->topic,
                $event->error,
            ));
    }
}

Step 6: Monitoring Dashboard

We built a simple monitoring dashboard that shows real-time webhook processing health.

// app/Http/Controllers/Admin/WebhookMonitorController.php
namespace App\Http\Controllers\Admin;

use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Redis;

class WebhookMonitorController extends Controller
{
    public function index()
    {
        $metrics = [
            'processed_last_hour' => DB::table('processed_webhooks')
                ->where('created_at', '>=', now()->subHour())
                ->where('status', 'completed')
                ->count(),

            'failed_last_hour' => DB::table('processed_webhooks')
                ->where('created_at', '>=', now()->subHour())
                ->where('status', 'failed')
                ->count(),

            'dead_letter_total' => DB::table('dead_letter_webhooks')->count(),

            'queue_depth' => $this->getQueueDepths(),

            'processing_time_avg' => DB::table('processed_webhooks')
                ->where('created_at', '>=', now()->subHour())
                ->where('status', 'completed')
                ->selectRaw('AVG(TIMESTAMPDIFF(SECOND, created_at, updated_at)) as avg_seconds')
                ->value('avg_seconds'),
        ];

        $metrics['success_rate'] = $metrics['processed_last_hour'] > 0
            ? round(
                $metrics['processed_last_hour'] /
                ($metrics['processed_last_hour'] + $metrics['failed_last_hour']) * 100,
                2
            )
            : 100;

        return view('admin.webhook-monitor', compact('metrics'));
    }

    private function getQueueDepths(): array
    {
        $queues = ['webhook-critical', 'webhook-high', 'webhook-default', 'webhook-low'];
        $depths = [];

        foreach ($queues as $queue) {
            $depths[$queue] = Redis::connection()->llen("queues:{$queue}");
        }

        return $depths;
    }
}

Step 7: Database Cleanup

The processed_webhooks table grows fast. At 10K events per minute, that is 14.4 million rows per day. We use a scheduled job to clean it up:

// app/Console/Kernel.php
namespace App\Console;

use Illuminate\Console\Scheduling\Schedule;
use Illuminate\Foundation\Console\Kernel as ConsoleKernel;

class Kernel extends ConsoleKernel
{
    protected function schedule(Schedule $schedule): void
    {
        // Clean up processed webhooks older than 7 days
        $schedule->command('webhooks:cleanup-processed', ['--days' => 7])
            ->dailyAt('03:00')
            ->withoutOverlapping()
            ->onOneServer();

        // Clean up dead letter webhooks older than 30 days
        $schedule->command('webhooks:cleanup-dead-letter', ['--days' => 30])
            ->weekly()
            ->withoutOverlapping()
            ->onOneServer();

        // Monitor webhook health and alert on anomalies
        $schedule->command('webhooks:health-check')
            ->everyFiveMinutes()
            ->withoutOverlapping()
            ->onOneServer();

        // Prune Horizon metrics
        $schedule->command('horizon:snapshot')
            ->everyFiveMinutes();
    }
}

The cleanup command uses chunked deletion to avoid locking the table:

// app/Console/Commands/CleanupProcessedWebhooks.php
namespace App\Console\Commands;

use Illuminate\Console\Command;
use Illuminate\Support\Facades\DB;

class CleanupProcessedWebhooks extends Command
{
    protected $signature = 'webhooks:cleanup-processed {--days=7}';
    protected $description = 'Remove processed webhook records older than specified days';

    public function handle(): int
    {
        $days = (int) $this->option('days');
        $cutoff = now()->subDays($days);

        $this->info("Cleaning up processed webhooks older than {$days} days...");

        $deleted = 0;
        $chunkSize = 10000;

        do {
            $affected = DB::table('processed_webhooks')
                ->where('created_at', '<', $cutoff)
                ->where('status', 'completed')
                ->limit($chunkSize)
                ->delete();

            $deleted += $affected;
            $this->info("Deleted {$affected} records (total: {$deleted})");

            // Small pause to avoid overwhelming the database
            if ($affected > 0) {
                usleep(100000); // 100ms
            }
        } while ($affected > 0);

        $this->info("Cleanup complete. Deleted {$deleted} records.");
        return self::SUCCESS;
    }
}

Performance Numbers

This architecture has been running in production for 6 months. Here are the numbers:

The 3.7% duplicate detection rate validates the idempotency approach. Shopify redelivers webhooks frequently — without the processed_webhooks deduplication table, we would process about 127,000 duplicate events daily.

Scaling Considerations

Redis cluster. At our current growth rate, single-instance Redis will become a bottleneck at around 20K events/minute. We plan to migrate to Redis Cluster with separate instances per queue priority.

Database sharding. The processed_webhooks table receives heavy write traffic. We are evaluating partitioning by date range to keep recent data fast while archiving old records.

Regional workers. Merchants are distributed globally. Deploying workers in US, EU, and Asia regions with region-specific Redis instances would reduce latency for geographically distant merchants.

This architecture is not overengineered for its current load. It is appropriately engineered for its growth trajectory. Every component — the priority queues, the idempotency checks, the dead letter handling, the monitoring — was added in response to a real production failure. Build the simple version first. Add complexity only when the simple version breaks.