Building a Workflow Engine That Handles 11.8 Million Cases

Contents

When you're processing 11.8 million cases across 384 active workflows, you learn what actually matters in workflow engine design. Not theoretical scalability - real, production-proven patterns that handle the load without collapsing.

Here's what 15 years of building SwiftCase taught us about workflow architecture.

The Problem With Most Workflow Systems

Most workflow tools fall into two traps:

1. Too rigid: Predefined workflows that can't adapt to real business complexity
2. Too flexible: "Build anything" tools that become unmaintainable spaghetti

Real businesses need both structure and adaptability. A legal firm's case management has different requirements than an insurance broker's claims processing - but both need audit trails, deadlines, and multi-party coordination.

Modular Architecture: The Foundation

SwiftCase is built with clear domain boundaries. Each module owns its responsibility:

• Workflow Engine: Core workflow execution and rules processing
• Document Generation: Templates with merge fields and conditional logic
• User Management: Multi-tenant users, roles, and permissions
• Reporting: Analytics, dashboards, and scheduled reports
• API Layer: REST APIs for integrations

Each module owns its domain. The workflow engine doesn't know how documents are generated - it just triggers document generation when needed. This separation means:

• Teams can work on different areas without stepping on each other
• Bugs are isolated to their domain
• Individual modules can be optimised without system-wide changes

Event-Driven Processing

Polling is expensive. At our scale, constantly checking "has anything changed?" would burn resources and add latency.

Instead, everything is event-driven:

1. User completes a form → Event fired
2. Workflow engine receives event → Evaluates rules
3. Rules match → Trigger actions (email, document, status change)
4. Actions complete → New events fired

This creates a reactive system where work only happens when something actually changes. A case sitting idle consumes zero processing resources.

Scheduled Events

Not everything is triggered by user action. Many workflows need time-based triggers:

• "If no response in 7 days, send reminder"
• "90 days before contract expires, create renewal case"
• "Every morning, generate summary reports"

We handle this with scheduled event processing - background jobs that evaluate time-based conditions and fire events when thresholds are crossed.

The Workflow Definition Problem

How do you define workflows that are:

• Understandable by non-developers
• Powerful enough for complex business logic
• Maintainable over years of changes

We use a configuration-based approach. Workflows are defined as structured data - conditions, actions, and transitions - rather than code. This means:

• Business analysts can understand and modify workflows
• Changes don't require code deployments
• Workflows can be versioned and rolled back

The trade-off is that some complex logic requires custom development. But for 90% of use cases, configuration is sufficient - and much safer than letting everyone write code.

State Management at Scale

A case can have dozens of fields, documents, notes, and related records. Multiply that by millions of cases, and you need careful state management.

Database Design

We use MariaDB with Doctrine ORM. Our data is fundamentally relational:

• Cases belong to clients
• Documents attach to cases
• Users have roles and permissions
• Everything has audit history

Relational databases excel at this. We optimise with:

• Careful indexing on query patterns
• Redis caching for frequently accessed data
• Read replicas for reporting queries

Audit Trail

Every action creates an audit record. With 154+ event types logged automatically, we can reconstruct exactly what happened to any case. This isn't just compliance - it's invaluable for debugging and support.

Multi-Tenancy Considerations

SwiftCase serves multiple organisations, each with their own:

• Workflows and configurations
• Users and permissions
• Data that must remain isolated

True multi-tenancy at the database level keeps data strictly separated while allowing shared infrastructure for efficiency.

Lessons Learned

1. Boring Technology Wins

We chose Symfony and MariaDB not because they're exciting, but because they're proven. Fifteen years of production use has validated that choice. The latest framework might be faster to start - but we're not optimising for week one, we're optimising for year ten.

2. Optimise for the Common Case

Most cases follow the happy path. Optimise for that, then handle edge cases gracefully. Don't let rare scenarios complicate the common flow.

3. Monitoring Matters More Than You Think

You can't fix what you can't see. Comprehensive logging and monitoring let us identify issues before users notice them. When you're processing thousands of cases daily, even small problems compound quickly.

4. Human Review Points

Not everything should be automated. Build in human review points for decisions that need judgement. The best workflow engines amplify human decision-making, not replace it.

What's Next

We're continuously improving the workflow engine. Current focus areas:

• AI-assisted routing: Using LLMs to suggest workflow paths based on case content
• Predictive SLAs: Machine learning to identify cases likely to breach deadlines
• Natural language workflow creation: Describing workflows in plain English

But the fundamentals - modular architecture, event-driven processing, careful state management - remain constant. They're proven patterns that scale.

---

Want to work on this?

We're always looking for engineers who care about building reliable systems at scale. If this kind of problem interests you:

View engineering roles | How we build