Image thumbnails with RabbitMQ

Q: Why is there no retry?

RabbitMQ ignores executeAt, so a retry would run at once. A broken image returns false instead.

Q: How do I stop the worker?

Touch /tmp/hermes-thumbnails-shutdown. SharedFileShutdown works with any driver.

Goal: after an upload, make thumbnails in three sizes in a worker. This recipe uses RabbitMQ through LazyRabbitMqDriver, so you can see how the broker changes the guarantees.

Files

yaml

services:
  rabbitmq:
    image: rabbitmq:4-management
    ports:
      - "5672:5672"   # AMQP
      - "15672:15672" # management UI, guest / guest

json

{
    "require": {
        "php": ">=8.4",
        "ext-gd": "*",
        "tomaj/hermes": "dev-master",
        "php-amqplib/php-amqplib": "^3.7"
    },
    "autoload": {
        "psr-4": {
            "App\\": "src/"
        }
    },
    "minimum-stability": "dev",
    "prefer-stable": true
}

php

<?php

declare(strict_types=1);

namespace App;

use PhpAmqpLib\Connection\AMQPLazyConnection;
use Tomaj\Hermes\Driver\LazyRabbitMqDriver;

final class Hermes
{
    public static function driver(): LazyRabbitMqDriver
    {
        // Lazy: the TCP connection opens on the first send() or wait(), not here.
        $connection = new AMQPLazyConnection(getenv('RABBITMQ_HOST') ?: '127.0.0.1', 5672, 'guest', 'guest');

        // Queue "thumbnails", no extra AMQP message properties, sleep 1 second between empty polls.
        return new LazyRabbitMqDriver($connection, 'thumbnails', [], 1);
    }
}

php

<?php

declare(strict_types=1);

use App\Hermes;
use Tomaj\Hermes\Emitter;
use Tomaj\Hermes\Message;

require __DIR__ . '/../vendor/autoload.php';

if (($_FILES['image']['error'] ?? UPLOAD_ERR_NO_FILE) !== UPLOAD_ERR_OK) {
    http_response_code(422);
    echo "Upload a file as the image field.\n";
    return;
}

$name = bin2hex(random_bytes(8)) . '.jpg';
$path = __DIR__ . '/../storage/originals/' . $name;
@mkdir(dirname($path), 0o775, true);
move_uploaded_file($_FILES['image']['tmp_name'], $path);

$emitter = new Emitter(Hermes::driver());
$emitter->emit(new Message('image.thumbnails', [
    'path' => realpath($path),
    'sizes' => [160, 640, 1280],
]));

http_response_code(202);
echo "Uploaded {$name}, thumbnails are on the way.\n";

php

<?php

declare(strict_types=1);

namespace App;

use RuntimeException;
use Tomaj\Hermes\Handler\HandlerInterface;
use Tomaj\Hermes\MessageInterface;

final class ThumbnailHandler implements HandlerInterface
{
    public function __construct(private string $directory) {}

    #[\Override]
    public function handle(MessageInterface $message): bool
    {
        ['path' => $path, 'sizes' => $sizes] = $message->getPayload();

        $source = @imagecreatefromstring((string) @file_get_contents($path));
        if ($source === false) {
            // A broken upload stays broken. Return false instead of throwing, so nothing is retried.
            return false;
        }

        $base = pathinfo($path, PATHINFO_FILENAME);
        foreach ($sizes as $width) {
            $thumb = imagescale($source, (int) $width);
            if ($thumb === false || !imagejpeg($thumb, "{$this->directory}/{$base}-{$width}.jpg", 82)) {
                throw new RuntimeException("Cannot write the {$width}px thumbnail of {$path}");
            }
        }

        return true;
    }
}

php

<?php

declare(strict_types=1);

use App\Hermes;
use App\ThumbnailHandler;
use Tomaj\Hermes\Dispatcher;
use Tomaj\Hermes\Shutdown\SharedFileShutdown;

require __DIR__ . '/../vendor/autoload.php';

$directory = __DIR__ . '/../storage/thumbnails';
@mkdir($directory, 0o775, true);

$driver = Hermes::driver();
// RabbitMQ has no Hermes shutdown helper of its own. A shared file works with any driver.
$driver->setShutdown(new SharedFileShutdown('/tmp/hermes-thumbnails-shutdown'));
$driver->setMaxProcessItems(500); // GD leaks a little with big images, start fresh now and then

$dispatcher = new Dispatcher($driver);
$dispatcher->registerHandler('image.thumbnails', new ThumbnailHandler($directory));
$dispatcher->handle();

Run it

bash

composer install
docker compose up -d
php bin/worker.php                    # terminal 1
php -S localhost:8003 -t public       # terminal 2
curl -F image=@photo.jpg http://localhost:8003/
ls storage/thumbnails

Three files appear: …-160.jpg, …-640.jpg, and …-1280.jpg. make e2e uploads a generated JPEG and checks the three sizes. The RabbitMQ UI at http://localhost:15672 (guest / guest) shows the thumbnails queue.

To stop the worker after its current message, touch the shared file:

bash

touch /tmp/hermes-thumbnails-shutdown

Pitfalls

On 5.x messages are acknowledged on receive. The 5.x driver consumes with no_ack = true. If the worker dies while resizing, that upload gets no thumbnails. Hermes 6.0 acknowledges after the handler instead, so the upload is delivered again (#70).
No delay, so no real retry. RetryTrait works, but the copy is published at once and all attempts happen within seconds. The handler does not use it. A broken file returns false, and a full disk throws.
On 5.x the queue is not durable. A RabbitMQ restart drops the queue and its messages. Hermes 6.0 declares it durable. See RabbitMQ.
No priorities. setupPriorityQueue() throws NotSupportedException on this driver.
GD needs memory for the full image. A 6000 × 4000 photo needs about 100 MB. Set memory_limit for the worker, not for the web app.

Full source

Every file of this recipe, with its README.md, Makefile and the e2e.sh check, is in examples/thumbnails on GitHub. Clone the repository, cd examples/thumbnails, and run make install up, then make e2e.

FAQ

What happens if the worker dies while resizing?

On 5.x that upload gets no thumbnails, because RabbitMQ settled the delivery on receive (no_ack). Keep a command that re-emits images without thumbnails. On 6.0 RabbitMQ delivers it again.

Why is there no retry?

RabbitMQ ignores executeAt, so a retry would run at once. A broken image returns false instead.

Are messages kept when RabbitMQ restarts?

Not on 5.x, which declares a non-durable queue. From 6.0 the queue is durable and messages are persistent. See RabbitMQ.

How do I stop the worker?

Touch /tmp/hermes-thumbnails-shutdown. SharedFileShutdown works with any driver.

How much memory does GD need?

About width × height × 4 bytes for the decoded image, so roughly 100 MB for a 6000 × 4000 photo.

Image thumbnails with RabbitMQ ​

Files ​

Run it ​

Pitfalls ​

Full source ​

FAQ ​

What happens if the worker dies while resizing? ​

Why is there no retry? ​

Are messages kept when RabbitMQ restarts? ​

How do I stop the worker? ​

How much memory does GD need? ​