Why We Built Switchboard With Multi-Provider AI

Contents

When we built Switchboard, our AI-powered communication platform, we made two deliberate architectural choices: keep the domain-specific work in our own code, and abstract external providers so no single vendor owns our system. Here's why, and how we implemented it.

The Vendor Lock-In Problem

AI is moving fast. The best model today might not be the best model tomorrow. If your architecture is tightly coupled to one provider:

• You can't easily switch when something better emerges
• Provider outages become your outages
• Pricing changes hold you hostage
• You're stuck with one provider's strengths and weaknesses

We've seen this pattern before. Companies that bet everything on one vendor inevitably regret it when that vendor changes pricing, deprecates APIs, or falls behind competitors.

The Provider Abstraction Layer

Switchboard's AI module abstracts all provider-specific details behind clean interfaces. The core engine doesn't know or care whether it's talking to OpenAI or Anthropic - it just requests completions and gets responses.

Our Own Layer

The domain-specific parts are ours, built around the providers' models:

• Case Tools: Five tools that find, verify, read, update and open SwiftCase cases
• Verification: The caller must match a second, different detail before the agent reads or changes anything
• Agent Rules: Per-agent instructions on what the agent must and must not do

Keeping these in our own code gives us control over behaviour that we don't want to hand to a provider.

External LLM Providers

For general language understanding and generation, we support multiple providers:

• OpenAI: GPT-4 and newer models
• Anthropic: Claude models

Each provider has different strengths. Some excel at reasoning, others at following instructions. Our abstraction lets us choose the provider per agent, with a second provider set as its fallback if you want one.

Speech Processing

Voice AI requires more than just LLMs:

• Speech-to-Text (STT): Converting user speech to text
• Text-to-Speech (TTS): Converting responses to natural speech

We use Deepgram for STT (excellent accuracy, low latency) and ElevenLabs for TTS (natural-sounding voices). But these are behind abstractions too - we can swap providers without changing application code.

How the Abstraction Works

The architecture follows a consistent pattern:

Application Code → Provider Manager → Provider Implementation → External API

The Provider Manager handles:

1. Provider Selection: Choosing which provider to use for a request
2. Failover: Switching to backup providers on errors
3. Configuration: Provider-specific settings without leaking into app code
4. Streaming: Consistent streaming interface regardless of provider

Application code simply requests what it needs:

• "Complete this conversation"
• "Transcribe this audio"
• "Generate speech for this text"

The provider layer handles the rest.

Real-World Benefits

Graceful Degradation

When OpenAI had a major outage last year, services built tightly on their API went down. Our speech recognition already degrades gracefully: if Deepgram fails mid-call, it moves to ElevenLabs, then Twilio, and the call carries on. The language model can be given the same kind of fallback, agent by agent.

Cost Optimisation

Different providers have different pricing models. By abstracting providers, we can give simpler agents cheaper models and reserve expensive models for complex work. This isn't possible when you're locked to one provider.

Trying New Models

When a new model releases, we can try it on one agent without touching the others. The abstraction makes this trivial - just add a new provider implementation and point that agent at it.

Best Tool for Each Job

Transcription, language understanding, and speech synthesis are different problems. The best solution for each might come from different providers. Our architecture lets us optimise each component independently.

Implementation Details

Streaming Is Non-Negotiable

Voice AI needs real-time responses. Users expect sub-second latency - they're having a conversation, not submitting a form. Everything is streaming:

• Audio streams in from the user
• Transcription streams to the LLM
• LLM responses stream to TTS
• Audio streams back to the user

Traditional request-response patterns add unacceptable latency. Streaming everything means users hear responses while they're still being generated.

WebSockets for Real-Time

HTTP polling doesn't work for voice. We use WebSockets for bidirectional, real-time communication between clients and servers. This adds complexity but is essential for voice applications.

Circuit Breakers

When a provider starts failing, we need to fail fast rather than queue up timeouts. Circuit breakers monitor provider health and short-circuit requests to failing providers, automatically routing to alternatives.

Trade-Offs

This architecture isn't free:

1. Complexity: Abstractions add code and indirection
2. Lowest Common Denominator: Provider-specific features are harder to use
3. Testing: Each provider needs integration tests
4. Subtle Differences: Providers behave slightly differently

We accept these trade-offs because the benefits - resilience, flexibility, cost optimisation - outweigh the costs for our use case. If you're building a simple prototype, this is probably overkill. For production systems handling real customer conversations, it's essential.

What We Learned

1. Build Your Own Where It Matters

General-purpose LLMs are impressive, but they shouldn't hold the parts that must be exact. We keep case lookup, caller verification and case updates in our own tools, and use external providers for the language.

2. Design for Change

The AI landscape changes constantly. Architectures that assume stability become liabilities. Design assuming your providers will change - because they will.

3. Abstractions Should Be Thin

Heavy abstractions that try to normalise everything become their own problem. Our abstractions are thin - just enough to swap providers, not so much that they obscure what's happening.

4. Monitoring Per Model

When something goes wrong, you need to know which model or provider is the issue. Separate monitoring and logging for each makes debugging tractable.

5. Provider-Specific Optimisations Are OK

The abstraction doesn't mean treating all providers identically. We tune prompts per provider, use provider-specific features when valuable, and route based on strengths. The abstraction is for swappability, not uniformity.

The Future

We're continuing to evolve this architecture:

• More providers: As new capable models emerge
• Smarter routing: ML-based provider selection
• Cost prediction: Estimating costs before making requests
• Quality monitoring: Automated quality scoring of provider responses

The core pattern - abstraction with clean interfaces - will remain. It's proven its value in production.

---

Interested in AI infrastructure?

Switchboard is built by a small team solving hard problems in real-time AI. If this kind of architecture interests you:

View engineering roles | Learn about Switchboard