- Agentforce retains brokers working by ensuring they’ve backup AI fashions as an alternative of counting on only one.
- Agentforce makes use of a multi-layered failover design, together with gateway-level failovers, gentle failovers, and circuit breakers, to offer complete safety in opposition to outages.
Massive Language Fashions (LLMs) like OpenAI present the reasoning and language to your AI brokers. Once you join your agent on to a single LLM supplier, that supplier’s uptime and efficiency dictate yours. A delay or outage on the mannequin layer instantly impacts your brokers, making them seem damaged or unresponsive to your clients.
LLM outages occur a minimum of a few times 1 / 4. That’s so much much less typically than one 12 months in the past, however nonetheless typically sufficient to make enterprise leaders and their clients nervous concerning the reliability of brokers.
The answer: a resilient failover plan.
Right here’s how Salesforce ensures LLM outages don’t silence brokers constructed with Agentforce.
Agentforce’s failover resolution
Hold your brokers “always-on” with seamless failover. Agentforce gives automated failover to an equal LLM mannequin in Azure OpenAI, making certain resilience in opposition to API errors, latency spikes, or upstream outages.
At Salesforce, we constructed this on the gateway layer. It’s a provider-level failover system designed for scale.
This isn’t simply summary routing logic. It’s purpose-built infrastructure that quickly detects failures, intelligently reroutes visitors, and restores service — all with out requiring any adjustments on the shopper facet.
Gateway failovers: Azure OpenAI backs up OpenAI
When a request to an OpenAI mannequin fails with a 401, 403, 404, or server-side 5xx error, the gateway checks whether or not failover circumstances are met. If that’s the case, the request is retried in opposition to the identical mannequin served by way of Azure OpenAI.
This high-level failover conduct diagram reveals how Salesforce brokers will verify Azure if OpenAI fails. Finish-users conversing with our brokers want by no means know there was a failure from OpenAI, since Azure will instantly fill the void.
The gist: Spot the LLM failure quick. Reroute queries as wanted. Resume regular visitors when circumstances enhance.
Agentforce helps two failover modes:
- Comfortable failover: Retry occurs on the particular person request stage when a 4xx or 5xx error occurs from the first mannequin supplier.
- Circuit breaker: If 40% or extra of OpenAI visitors fails inside a 60-second window, Agentforce platform bypasses retries solely and routes all visitors to the equal mannequin on Azure OpenAI. The circuit resets after 20 minutes if OpenAI recovers.
This protects Agentforce throughout each remoted failures and sustained outages.
Dealing with delays: Good latency retries guarantee quick responses
Velocity issues in agentic interactions — each millisecond counts. Agentforce’s retry mechanism handles each latency and server-side errors to make sure your brokers keep responsive.
If a request stalls, Agentforce mechanically retries it, preserving your brokers responsive even when upstream fashions stall. If the request fails utterly with a 4xx or 5xx error on the server facet, failover kicks in and serves the request from a secondary LLM supplier.
A race for solutions: Delayed Parallel Retries
Conventional LLM requests depend on a lone name with no possibility however to cease and wait if there’s an issue.
However we use a way referred to as delayed parallel retries to maintain brokers quick and responsive if there’s a delay or different concern with the LLM mannequin.
Right here’s the way it works: When your agent sends a request to an LLM, a major callout begins. If that callout doesn’t get a response inside a sure time, that delay triggers a second, parallel callout. Each requests then “race” to complete, and we use whichever one returns a solution first.
This diagram outlines Agentforce’s delayed parallel retry mechanism that improves agent availability. If the first callout stalls, it triggers a concurrent retry callout. This “race” between requests ensures quick responses and boosts agent availability, even throughout upstream mannequin delays.
Our methodology solves two issues that delay conventional single-thread efficiency: strict sequential processing and lengthy waits.
We bypass sequential processing by creating parallelism. Reactive frameworks mix the first and delayed calls, and cancel the slower one the second a response arrives.
We finish lengthy waits with a separate, “elastic” thread that schedules the retry timer and manages the “race.”
Agentforce boasts 99.99% availability because of this failover design.
Observability: Log and monitor all errors
We use observability instruments to grasp and enhance how our programs carry out.
Agentforce screens each flip of the agentic interplay, capturing the whole lot from a person’s preliminary request to the AI agent’s last response.
- We log each error that happens, together with its error path.
- We monitor failover occasions and correlate them with planning and reasoning conduct. This helps us regulate our system’s thresholds. For instance, we decide what number of dialog turns to take care of and cross to the brand new LLM to make sure a seamless transition with out dropping context.
- We use the circuit breaker mechanism to keep away from flip-flopping throughout partial restoration.
- We monitor model-specific Service Degree Indicators (SLIs) to detect degradation in a particular AI mannequin as early as doable, and swap to a backup mannequin.
All of those mannequin interactions are seen to clients of their audit logs.
Agentforce has failover safety constructed proper in
LLM failures, whether or not as a result of API errors, latency, or upstream outages, aren’t simply noise. They block activity development and undermine the system contract, making clients query their belief in your brokers.
Profitable brokers rely on real-time, multi-turn LLM execution to drive planning and reasoning selections and system actions.
What we’ve constructed proper into Agentforce is failure-aware routing, scoped to identified failure modes and tuned for actual latency and error conduct.
It isn’t a general-purpose abstraction layer that merely returns a generic failover reply when issues go incorrect. Our resolution is a proactive, focused infrastructure to make Agentforce dependable beneath real-world failure circumstances.
Mannequin-level volatility is not any excuse for system-level fragility.
Jayesh Govindarajan
EVP, Software program Engineering at Salesforce
Our failover resolution at present handles OpenAI and Azure OpenAI fashions. We’ll quickly increase it to different suppliers, together with Anthropic and Gemini. Ultimately, you’ll be capable of use adaptive mannequin fallback and vendor choice for much more management. However this core precept holds: Mannequin-level volatility is not any excuse for system-level fragility.
At Salesforce, Belief is our primary worth. We received’t let LLM failures undermine your clients’ belief in your AI brokers. Agentforce bakes within the reliability you want by making certain your brokers work even when LLM suppliers fail.
What’s your agentic AI technique?
Our playbook is your free information to turning into an agentic enterprise. Study use circumstances, deployment, and AI abilities, and obtain interactive worksheets to your crew.
Often Requested Questions (FAQs)
Agentforce’s failover resolution addresses the issue of AI brokers turning into unresponsive or showing damaged as a result of delays or outages from a single Massive Language Mannequin (LLM) supplier.
Agentforce gives automated failover to an equal LLM mannequin in Azure OpenAI on the gateway layer, quickly detecting failures and intelligently rerouting visitors with out client-side adjustments.
Agentforce helps “gentle failover,” which retries particular person requests upon errors, and “circuit breaker,” which routes all visitors to a backup mannequin if a big share of requests fail inside a particular time-frame.
Delayed parallel retries provoke a second, parallel name to an LLM if the first name is delayed. Each calls “race” to complete, and the primary response is used, stopping lengthy waits and enhancing agent pace.
Agentforce makes use of observability instruments to log each error and its path, monitor failover occasions, and monitor model-specific Service Degree Indicators (SLIs). All these mannequin interactions are seen to clients of their audit logs.
