Webhook delivery with retries

Goal: POST signed events to your customers' URLs. Retry with a growing delay when the receiver is down. Give up on answers that will not get better.

Runnable project

Every file below is imported from examples/webhooks, which has a Makefile and an e2e.sh that runs the whole flow. Clone the repository, cd examples/webhooks, and run make install up.

Key files

• src/WebhookHandler.php
• bin/worker.php
• public/index.php
• docker-compose.yml

View source on GitHub

Files

Redis and src/Hermes.php as in the welcome email tutorial.

{
    "require": {
        "php": ">=8.4",
        "tomaj/hermes": "dev-master",
        "predis/predis": "^2.3 || ^3.0",
        "symfony/http-client": "^7.0",
        "monolog/monolog": "^3.0"
    },
    "autoload": {
        "psr-4": {
            "App\\": "src/"
        }
    },
    "minimum-stability": "dev",
    "prefer-stable": true
}

<?php

declare(strict_types=1);

use App\Hermes;
use Tomaj\Hermes\Message;

require __DIR__ . '/../vendor/autoload.php';

// Something happened in your app, for example an order was shipped.
$event = [
    'event' => 'order.shipped',
    'order_id' => 1042,
    'shipped_at' => date(DATE_ATOM),
];

Hermes::emitter()->emit(new Message('webhook.deliver', [
    'url' => getenv('WEBHOOK_URL') ?: 'http://127.0.0.1:9000/',
    'secret' => 'whsec_change_me',
    'body' => $event,
]));

http_response_code(202);
echo "Webhook queued.\n";

<?php

declare(strict_types=1);

namespace App;

use RuntimeException;
use Symfony\Contracts\HttpClient\HttpClientInterface;
use Tomaj\Hermes\Handler\HandlerInterface;
use Tomaj\Hermes\Handler\RetryTrait;
use Tomaj\Hermes\MessageInterface;

final class WebhookHandler implements HandlerInterface
{
    use RetryTrait;

    public function __construct(private HttpClientInterface $http) {}

    #[\Override]
    public function handle(MessageInterface $message): bool
    {
        $payload = $message->getPayload() ?? [];
        $body = json_encode($payload['body'], JSON_THROW_ON_ERROR);
        $timestamp = (string) time();

        $response = $this->http->request('POST', $payload['url'], [
            'headers' => [
                'Content-Type' => 'application/json',
                // The message id stays the same on every retry. Receivers use it to drop duplicates.
                'Webhook-Id' => $message->getId(),
                'Webhook-Attempt' => (string) ($message->getRetries() + 1),
                'Webhook-Timestamp' => $timestamp,
                'Webhook-Signature' => hash_hmac('sha256', "{$timestamp}.{$body}", $payload['secret']),
            ],
            'body' => $body,
            'timeout' => 10,
        ]);

        $status = $response->getStatusCode(); // waits for the response
        if ($status >= 500 || $status === 429) {
            // Throwing makes Hermes send a copy with a later executeAt.
            throw new RuntimeException("Receiver answered {$status}");
        }

        // 2xx is delivered. Any other 4xx will not get better on a retry, so give up.
        return $status < 300;
    }

    public function maxRetry(): int
    {
        return 8; // last attempt roughly an hour and a half after the first one
    }
}

<?php

declare(strict_types=1);

use App\Hermes;
use App\WebhookHandler;
use Monolog\Handler\StreamHandler;
use Monolog\Logger;
use Symfony\Component\HttpClient\HttpClient;
use Tomaj\Hermes\Dispatcher;
use Tomaj\Hermes\Shutdown\PredisShutdown;

require __DIR__ . '/../vendor/autoload.php';

$redis = Hermes::redis();
$driver = Hermes::driver($redis);
$driver->setShutdown(new PredisShutdown($redis));

$dispatcher = new Dispatcher($driver, new Logger('webhooks', [new StreamHandler('php://stdout')]));
$dispatcher->registerHandler('webhook.deliver', new WebhookHandler(HttpClient::create()));
$dispatcher->handle();

A fake receiver for local testing. It answers 503 to the first attempt and 204 to the second:

<?php

declare(strict_types=1);

// A fake receiver for local testing: fails the first attempt with 503, accepts the next one.
$attempt = (int) ($_SERVER['HTTP_WEBHOOK_ATTEMPT'] ?? 1);
$expected = hash_hmac('sha256', $_SERVER['HTTP_WEBHOOK_TIMESTAMP'] . '.' . file_get_contents('php://input'), 'whsec_change_me');
$valid = hash_equals($expected, $_SERVER['HTTP_WEBHOOK_SIGNATURE'] ?? '');

error_log(sprintf('webhook %s attempt %d signature %s', $_SERVER['HTTP_WEBHOOK_ID'] ?? '?', $attempt, $valid ? 'ok' : 'BAD'));
http_response_code($attempt === 1 ? 503 : 204);

Run it

composer install
docker compose up -d redis
php -S 127.0.0.1:9000 receiver.php     # terminal 1: the receiver
php bin/worker.php                     # terminal 2: the worker
php public/index.php                   # terminal 3: emit one event

The receiver logs the first attempt at once and the second one 16 to 45 seconds later, with the same Webhook-Id:

webhook d8299c53-… attempt 1 signature ok
webhook d8299c53-… attempt 2 signature ok

The worker logs Handler App\WebhookHandler throws exception - Receiver answered 503 between them.

Pitfalls

• Decide which failures to retry. Throwing retries, returning false does not. A timeout, 5xx, or 429 can succeed later. A 404 or 410 will not.
• Send a stable id. The dispatcher keeps the message id on every retry copy. Send it as a header so receivers can drop duplicates, because a timeout can hide a request that did arrive.
• The retry delay is fixed by Hermes. It is retries⁴ + 15 + jitter × (retries + 1) seconds, with a random jitter from 1 to 30. With maxRetry() at 8, the last attempt is about an hour and a half after the first.
• Retries need a driver with delay. On RabbitMQ and SQS the copy is published at once, so all attempts happen within seconds. Use Predis for webhooks.
• Set a timeout. One slow receiver must not block the worker. Ten seconds is plenty, and running several workers keeps other deliveries moving.

Full source

Every file of this recipe, with its README.md, Makefile and the e2e.sh check, is in examples/webhooks on GitHub. Clone the repository, cd examples/webhooks, and run make install up, then make e2e.

FAQ

Which responses are retried?

Timeouts, 5xx, and 429 throw, so they are retried. Other 4xx return false and are not.

How do receivers drop duplicates?

By the Webhook-Id header. It is the message id, which stays the same on every retry.

How do receivers verify the signature?

Compute hash_hmac('sha256', timestamp . '.' . body, secret) and compare it with hash_equals(), as receiver.php does.

How long does Hermes keep trying?

With maxRetry() 8, about an hour and a half. See Retry for the formula.

Can I use RabbitMQ for webhooks?

Not with backoff. RabbitMQ ignores executeAt, so every retry runs at once.