Just Use Postgres for Durable Workflows

Recent technical discussions recommend using Postgres directly for durable workflows, replacing traditional external orchestrators. This approach leverages Postgres’s scalability, reliability, and rich query capabilities, simplifying system architecture and reducing complexity.

Durable workflows involve checkpointing program progress in a database to enable recovery after failures. Traditionally, systems like Temporal, Airflow, and AWS Step Functions use external orchestrators to manage workflow execution and state. However, a growing perspective argues that this external orchestration is overly complex. Instead, by using Postgres as the core coordination point—where application servers directly communicate with and checkpoint to the database—developers can streamline workflow management.

This method involves clients submitting workflows by creating entries in a Postgres table. Application servers poll this table, dequeue workflows, and checkpoint each step’s output directly into the database. If a server crashes, others can recover workflows from the checkpoints, ensuring resilience. Postgres’s native features, such as locking mechanisms and integrity constraints, prevent duplicate processing and coordinate concurrent execution effectively.

Why It Matters

Replacing external orchestrators with Postgres simplifies durable workflow architecture, making it more accessible and easier to scale. It allows organizations to leverage well-understood database solutions for scalability, high availability, observability, and security. This approach reduces operational complexity and reliance on specialized orchestration systems, potentially lowering costs and increasing system robustness.

Given Postgres’s widespread use and mature ecosystem, this model offers a practical, scalable alternative for building reliable, durable workflows, especially for teams already invested in relational databases.

Background

Durable workflows have traditionally depended on external orchestrators that manage execution state and retries. These systems, while powerful, introduce complexity and additional points of failure. Recent discussions suggest that since checkpointing is fundamentally a database operation, it makes sense to embed this logic directly into the database layer. Postgres, with its proven scalability and rich feature set, emerges as an ideal candidate for this approach. This perspective aligns with ongoing efforts to simplify distributed systems and reduce reliance on complex orchestration layers.

“Replacing a central orchestrator with Postgres makes durable workflows fundamentally simpler and leverages existing database solutions for scalability and observability.”

— Hacker News discussion participant

“Postgres’s native features like locking and integrity constraints make it well-suited for coordinating concurrent workflow execution without external orchestrators.”

— Postgres expert

What Remains Unclear

It is still unclear how this approach performs under extremely high throughput scenarios or complex workflow dependencies. Practical implementation details and best practices are still emerging, and real-world case studies are limited.

What’s Next

Further experimentation and case studies are expected to validate this approach. Developers and organizations will likely test Postgres-based durable workflows in production environments, refining techniques for scalability and reliability. Upcoming tools or frameworks may emerge to facilitate this pattern, and community feedback will shape best practices.

Key Questions

How does using Postgres for workflows compare to traditional orchestrators?

Using Postgres simplifies architecture by removing the need for a separate orchestrator server, leveraging the database’s native features for coordination, scalability, and observability.

Can Postgres handle high-volume workflow execution?

Postgres can scale vertically to handle tens of thousands of workflows per second and can be horizontally scaled with sharding or distributed variants, but real-world limits depend on workload specifics.

What are the security implications of this approach?

Since workflow checkpoints are stored in the database, security measures like access controls, encryption, and auditing are essential to protect sensitive data, similar to other database applications.

Is this approach suitable for all types of workflows?

It is most suitable for workflows that can be managed with simple checkpointing and do not require complex external orchestration or dependencies. More complex workflows may still benefit from specialized orchestrators.

Source: Hacker News