alt Hacker NewsThe one use case where a DB backed queue will fail for sure is when the payload is large. For example, you queue a large JSON payload to be picked up by a worker and process it, then the DB writing overhead itself makes a background worker useless.
I've benchmarked Redis (Sidekiq), Postgres (using GoodJob) and SQLite (SolidQueue), Redis beats everything else for the above usecase.
SolidQueue backed by SQLite may be good when you are just passing around primary keys. I still wonder if you can have a lot of workers polling from the same database and update the queue with the job status. I've done something similar in the past using SQLite for some personal work and it is easy to hit the wall even with 10 or so workers.
Replies
In my experience you want job parameters to be one, maybe two ids. Do you have a real world example where that is not the case?
FWIW, Sidekiq docs strongly suggest only passing around primary keys or identifiers for jobs.
Using Redis to store large queue payloads is usually a bad practice. Redis memory is finite.
> Redis beats everything else for the above usecase.
Reminds me of Antirez blog post that when Redis is configured for durability it becomes like/slower than postgresql http://oldblog.antirez.com/post/redis-persistence-demystifie...
Interesting, as a self-contained minimalistic setup.
Shouldn't one be using a storage system such as S3/garage with ephemeral settings and/or clean-up triggers after job-end ? I get the appeal of using one-system-for-everything but won't you need a storage system anyway for other parts of your system ?
Have you written up somewhere about your benchmarks and where the cutoffs are (payload size / throughput / latency) ?