From Celery to DBOS: Scaling Dosu’s Pipelines to 20k Workflows per Hour

We just partnered with DBOS on a case study that digs into how we hardened Dosu’s ingestion and RAG pipelines. The short version: Celery got us moving, but as we started ingesting hundreds of thousands of documents per day and coordinating long, multi-step jobs, orchestration and observability became the bottleneck.

The problem: Our ingestion and RAG pipelines were getting complex. We were running Celery, juggling retries, and chasing failures across scattered logs. Shipping got slower and debugging took too long.

What we changed: We migrated the pipelines to DBOS for durable execution, queueing, and end-to-end observability. It runs within our existing Cloud Run setup and stores state in Postgres, so we eliminated the need for extra infrastructure.

What improved

• Orchestration is now code we can read and reason about
• Real-time visibility with DBOS Conductor cuts debugging time
• Scaled to roughly 20,000 workflows per hour while serving 50,000+ projects reliably

For the full DBOS customer story, including the architecture and numbers, please read on. You can also view it live in Dosu’s Public Space for DBOS to observe workflows and context flow in real-time.

Looking ahead

If you are living with Celery queues and scattered logs, this case study shows a cleaner path. DBOS provided us with durable workflows, built-in retries, and a single place to view what is happening. The payoff was fewer fire drills and faster shipping.

We’re standardizing new agents on this stack so reliability and observability come by default, not as an afterthought.