Orchestrating complex workflows across distributed systems often comes with a recurring challenge: How do you recover gracefully from partial failures without redoing everything? The traditional model of replaying entire orchestration histories wastes resources and risks side effects from duplicating work. Many platforms either resign to all-or-nothing replay or impose operational burdens that complicate recovery.

Enter Execution Ledger — an open-source durable workflow orchestrator written in Rust, deployed on Azure Container Apps. It treats recovery as a first-class primitive, enabling operators to target replay at the granularity of individual workflow steps, not whole orchestrations. Backed by built-in idempotency and compensation patterns, this engine was designed from the ground up to address real-world durability pain points spanning ERP systems, blob storage, queue processing, and more.

In this post, we'll examine why Rust and Azure Container Apps form a strong fit for this platform, how Execution Ledger is architected for operational resilience and scalability, and what key lessons emerged from building such a backend system for durable job orchestration.

Architecture Overview

Execution Ledger runs as four Container Apps behind API Management: the scheduler (singleton), Drasi (singleton for logical replication), workers (multiple replicas with concurrency limits), and connectors. The worker replicas scale horizontally using KEDA based on Azure Service Bus queue depth, enabling parallelism while preserving sequential step execution within each intent.

Key Technical Observations

• Step-Level Targeted Replay — Unlike other orchestration frameworks that rewind workflows wholesale, Execution Ledger assigns idempotency keys per step (run ID + step ID). Operators can selectively replay only the failed step without affecting the rest of the workflow’s execution history. This surgical precision simplifies recovery and reduces risk of duplicate side effects.

• Rust for Performance and Reliability — Rust’s native compilation eliminates cold-start overhead common in managed runtimes. A minimal memory footprint (single-digit MBs per instance) combined with denial of unsafe code and stringent clippy lints safeguard reliability in a multi-tenant environment where programming errors can cost dearly.

• Azure Container Apps with KEDA Autoscaling — Leveraging built-in KEDA scalers on Azure Service Bus queue depth allows automatic horizontal scaling of worker replicas. This architecture decouples concurrency from single-instance limits, enabling parallelism across replicas while preserving sequential step execution within each intent.

• Managed Identity for Seamless Security — No shared connection strings or secrets in environment variables. Execution Ledger uses Azure Managed Identity everywhere — from Service Bus access to Key Vault integration — minimizing operational credential burden and enhancing security posture.

• PostgreSQL Connection Pooling Nuances — Each worker replica maintains its own connection pool, requiring careful sizing to avoid exhausting Azure PostgreSQL Flexible Server’s max connections. This constraint influences scaling strategies and operational monitoring.

• Doc-Comment-As-Specification for Connectors — Connector modules include detailed Rust documentation that both explains and specifies configuration contracts, expected failure modes, and compensation strategies, doubling as developer onboarding and runtime contract enforcement.

How It Works

Step 1: Job Scheduling

The scheduler, running as a singleton Container App to avoid conflicting enqueues, injects workflow intents into the system by writing records into PostgreSQL. These intents represent durable jobs to be executed and monitored.

Step 2: State and Event Replication (Drasi)

Drasi connects as a single consumer to PostgreSQL's logical replication slot, streaming changes downstream to update caches, trigger events, or communicate with other services. By isolating replication to one Container App, Execution Ledger ensures predictable event flow without sharding complexity.

Step 3: Worker Execution

Workers retrieve intents from the Service Bus queue, each intent representing a durable workflow consisting of sequenced steps that execute synchronously within that intent. To manage load, each worker replica employs a Tokio semaphore limiting concurrency (default 5 tasks per replica). KEDA monitors the queue and scales container replicas automatically based on message depth—giving elastic processing power.

A typical command for replaying a failed step looks like this:

cargo run -p execution-ledger-cli -- job-replay <tenant-id> <run-id> health

This instructs the worker to rerun only the "health" step for that execution run, leaving other steps untouched, ensuring precise incident recovery.

Step 4: Idempotency and Compensation

Each workflow step’s connector implements idempotency using the composite key of tenant-run-step. Rust’s strict error handling and fail-closed semantics prevent partial side-effect execution. Compensation logic, a first-class primitive, reallocates or rolls back external system effects if needed, reducing manual intervention.

Example Rust snippet enforcing safe error handling:

// ❌ won't compile — clippy::unwrap_used is deny
let config = load_config().unwrap();

// ✅ proper error propagation
let config = load_config().context("failed to load runtime config")?;

This approach reduces runtime panics and forces explicit error design early.

Step 5: Configuration and Secrets Management

Using Azure App Configuration tied with Key Vault references, the system boots with minimal environment variables – just the configuration endpoint and environment label. Other secrets (SAS tokens, URLs, toggles) resolve securely at startup via managed identities, locking down the deployment and simplifying audit compliance.

Quick Tips & Tricks

1. Leverage KEDA With Azure Service Bus for Rust Apps
   Rust workers can compete independently for queue messages; no additional coordination is required. Configure a sensible scale threshold (e.g., 25 messages per replica) to avoid scaling thrash.

2. Keep Your PostgreSQL Connection Pools Tight
   Monitor and limit connections per worker replica carefully, especially on smaller Azure PostgreSQL tiers with strict max connection limits. Factor in API and scheduler pools too.

3. Embrace Fail-Closed Configuration Patterns
   Use fail-fast in connectors to avoid undefined states. Missing or invalid configuration should terminate early with explicit errors rather than fallback behaviors.

4. Document Connectors Inline Using Rust Doc Comments
   Write configuration specs and failure behaviors as module-level Rust doc comments. This doubles as executable documentation and onboarding for new contributors.

5. Avoid Unsafe Rust and Panic Calls
   Enforce strict lints denying unsafe, panic!(), unwrap(), and expect() to increase runtime stability and security for multi-tenant orchestration applications.

6. Deploy Using Azure Developer CLI (azd) for Infrastructure
   Use azd provision && azd deploy to spin up the entire environment including Service Bus, PostgreSQL, Key Vault, and Container Apps with minimal operational overhead.

Conclusion

Execution Ledger fills a unique niche in the durable workflow orchestration space by making step-level recovery first-class, bringing precise incident recovery to complex multi-step jobs that span disparate external systems. By harnessing the power of Rust’s safety and performance, combined with Azure Container Apps’ managed scaling and identity integration, Luke Murray crafted a platform that excels where serverless durable functions or heavyweight Kubernetes solutions fall short.

The approach emphasizes operational transparency and reliability—idempotency and compensation patterns baked in, zero cold-start latency, and a clean developer experience with rigorous compile-time checks. Execution Ledger is particularly well-suited for ERP integrations, asynchronous queue processing, and durable REST orchestration that demand strong recovery guarantees.

As the cloud ecosystem continues evolving toward fine-grained, event-driven, resilient architectures, durable workflow engines like Execution Ledger demonstrate how language choices and cloud platform capabilities can reshape operational excellence. Expect this breed of platforms to gain traction for critical business automations where failure tolerance is non-negotiable.

References

1. Execution Ledger on GitHub — Open-source repo with code, docs, and devcontainer setup
2. Building Execution Ledger - Durable Workflow Orchestration in Rust on Azure Container Apps — Original technical blog post by Luke Murray
3. Azure Container Apps documentation — Official guides on ACA and KEDA
4. KEDA scalers for Azure — Deep dive into autoscaling strategies using KEDA
5. Azure PostgreSQL Flexible Server — Managed PostgreSQL service details

Image courtesy of Luke Murray, luke.geek.nz