Photo: free stock photography (Unsplash licence) — see imprint
The admin worker lied to you
In development, tasks run because an admin browser tab is quietly processing the queue. In production nobody has the admin open at 3 a.m. — so the queue simply fills up. Tasks sit in 'queued' and nothing complains.
Run a real consumer
Run the message consumer as a supervised process (systemd, supervisor, a container) with a memory and time limit so it restarts cleanly. Disable the admin worker in production. This is the fix, and it takes fifteen minutes.
Then watch the queue depth
A consumer that dies silently looks exactly like one that is idle. Alert on queue depth and on the age of the oldest message — that catches both a dead worker and one that cannot keep up.
Beware the poison message
One malformed job that throws on every retry can block a queue forever. Configure a retry limit and a dead-letter queue so a single bad order does not stop every other order from being processed.
- Disable the admin worker; run a supervised consumer.
- Alert on queue depth and oldest-message age.
- Configure retries and a dead-letter queue.
Frequently asked questions
Because nothing is consuming the queue. Nothing is broken, nothing errors — the jobs are simply sitting there waiting for a worker that does not exist. In development you never notice, because an open admin tab quietly processes them for you. In production at 3 a.m. nobody has the admin open. Check for a running consumer first; that is the answer most of the time.
No. It only works while somebody has an admin tab open, which is a terrible thing to make your nightly jobs depend on. Turn it off in production and run the message consumer as a supervised process — systemd, supervisor, a container — with memory and time limits so it restarts cleanly instead of leaking. It is a short piece of work and it is the actual fix.
Not by looking at it — a dead consumer and an idle one look identical from the outside. Alert on two things instead: the depth of the queue, and the age of the oldest message in it. Depth catches a worker that cannot keep up; age catches a worker that is not there at all. Without those, you find out when a customer tells you.
That is a poison message: one malformed job that throws on every retry and gets retried forever. Give the queue a retry limit and a dead-letter queue, so the bad job steps aside after a few attempts and everything behind it keeps moving. Without that, a single broken order can stop every other order from being processed, and it will pick a Friday to do it.
We do this for a living — Shopware, Node.js, React, ERP integration and automation for B2B.
Talk to an engineer