Does the Flower Shop Need an AI Gateway?
Should your agents sit behind an AI gateway? I'll answer that for the flower shop agents, the open-source example I use to show how I build worker-agent systems. Gateways are showing up in a lot of architecture diagrams this year, and plenty of vendors will tell you every agent needs one.
This post is for two people. One is deciding whether to pay for a gateway, whether that's an owner or the person who signs off on an AI budget. The other is the engineer who would have to set it up and keep it running. I'll use the flower shop as the worked example, compare the main tools, and finish with how I put gateways in for clients.
For the flower shop, my answer is no, not yet.
What a gateway is
An AI gateway is a proxy between your code and the model providers. Your agents send their model requests to the gateway instead of straight to Anthropic or anyone else, and the gateway forwards them.
Because every request passes through one place, the gateway can apply rules to all of them: routing between providers and models, falling back to another model when one fails, caching, rate limits, budgets per team or per key, logging, and sometimes content filters or redaction.
If you're buying: a gateway is how you find out who is spending what on AI, and how you put a ceiling on it, without asking every team to build that themselves.
If you're building: most gateways now accept the Anthropic Messages API format, so moving an existing app behind one is usually a change to the client's base_url and API key. The harder work is deciding the keys, budgets and routing rules.
A gateway only sees model traffic: the prompt going out and the reply coming back. It doesn't run your tools and it doesn't know which tool calls need a person's sign-off. In the flower shop, the approval gate and the MCP server sit on the other side of the agent loop, and a gateway replaces neither.
What the flower shop does today
The agents make every model call through one client, AsyncAnthropic, to one provider. The model comes from SHOP_AGENTS_MODEL. It runs as a single process when the owner asks for a morning brief or a specific task.
Cost is capped per run in code. Each worker gets at most eight turns and 1,500 output tokens per turn, and the coordinator gets the same turn limit with 2,000. A runaway loop stops on its own. There's no monthly ceiling in the code, and for one shop on one API key the spend limit in the provider's console covers that. The Anthropic SDK already retries failed requests twice, with backoff.
There's also already a place to change where requests go. On the local-model branch, --local points the same client at llama.cpp's server by setting base_url. Pointing it at a gateway is the same one-line change, so waiting costs nothing.
Checking the usual reasons
Multiple providers. The shop uses one.
Budgets per team or key. There's one owner and one key. Per-team budgets matter when several apps or departments share a provider account and someone has to answer for whose agent spent what.
Fallback. This is the one I thought about longest. Valentine's week is the shop's busiest week, and a provider outage then would be bad timing. But when a run fails, the owner just doesn't get a morning brief, and nothing gets sent, because everything risky was waiting for approval anyway. The shop keeps working the way it did before the agents.
A fallback model also has a cost. If the gateway quietly swaps models, the workers are now running on one my evals never tested. A quote written by an untested model is a worse outcome than a late brief.
Caching. Exact-match caching rarely hits for agents, because each request carries the latest tool results. Semantic caching, which serves a stored answer to a request that looks similar, worries me more. Two inventory questions can look almost the same while the stem counts behind them differ, and a cached answer about yesterday's peonies is wrong in a way nobody notices.
Logging. The shop already keeps an audit log of every automatic run, approval and rejection. For more detail on the model calls themselves, I'd add OpenTelemetry tracing in the agent code before adding a proxy.
When a gateway is worth it
The answer changes as soon as more than one team, app or client is involved, and the reasons are usually organizational.
- Several agents or apps share provider accounts, and finance wants spend reported per team, product or client. Virtual keys with their own budgets and model allow-lists are what you're paying for here.
- A security team wants one place where model traffic leaves the network, so it can be logged, filtered or blocked. That's much easier at one proxy than across every service that calls a model.
- The system really does use different models for different jobs, and the routing rules shouldn't be copied into every agent.
- Uptime matters enough that you'll pay for a tested fallback model, including the evals to prove it behaves.
If the flower shop grew into several locations, each with its own agents and its own budget, I'd add one then.
Comparing the tools
These are the gateways I see most often. All of them can sit in front of Anthropic models, and they move fast, so check current docs before you commit.
| LiteLLM | Portkey | Cloudflare AI Gateway | Kong AI Gateway | |
|---|---|---|---|---|
| How you run it | Open source, self-hosted proxy | Open-source gateway plus a hosted control plane; VPC option for enterprise | Managed, on Cloudflare's network | Plugins on Kong's API gateway, self-hosted or Kong's cloud |
| Budgets and keys | Virtual keys with per-key and per-team budgets and model access lists | Per-key budgets and rate limits; RBAC on paid tiers | Spend limits per user or team via request metadata | Token- and cost-based rate limits per consumer |
| Fallback and routing | Fallbacks and load balancing across providers | Fallbacks, retries, conditional routing | Dynamic routing and fallbacks, set up in a dashboard | Load balancing and fallback across providers and models |
| Caching | Exact and semantic caching (Redis, Qdrant and others) | Simple and semantic caching | Exact-match caching at the edge | Semantic cache plugin |
| Guardrails | Hooks for guardrail providers | Built-in and partner guardrails, PII redaction | Content moderation on prompts and responses | Prompt guard and related plugins |
| Good fit | Teams that want full control and will run their own infrastructure | Teams that want a dashboard and governance without building it | Teams already on Cloudflare who want something running today | Organizations already running Kong for their APIs |
For buyers, the first column of that table matters most. Self-hosting LiteLLM costs nothing to license, but someone on your team now operates a service that every AI feature depends on. Managed options cost money each month and you're handing request data to another vendor, so ask for their security documentation the way you would for any processor. If you already pay for Cloudflare or Kong, the cheapest gateway may be one you already have.
For builders, look closely at how each one handles streaming and tool-use responses with your SDK, how budgets are keyed (per key, per team, per metadata tag), and where the logs end up. Turn off semantic caching for agent traffic unless you've tested it against real tool results. And whatever you pick, the gateway is one more network hop and one more thing that can be down, so give it the same timeouts and alerting as any other dependency.
On AWS there's a fifth option: if your models already run through Amazon Bedrock, some of these controls come built in, and teams often run LiteLLM in front of Bedrock with a fallback to Anthropic's own API.
What a gateway doesn't change
With or without a gateway, the rules from the original post still apply. The list of tools that run without approval stays in the orchestrator's code. Each worker keeps its allow-list. A queued action still reports itself as not done. A gateway can cap what you spend on tokens. It can't stop a worker from sending a $4,000 quote, and nothing about your safety should depend on it.
How I'd put a gateway in for you
If you're past the point where one key and a console spend limit are enough, this is how I usually do the work.
1. Map the model traffic. I find every place your systems call a model: which apps, which keys, which providers, and roughly what each one costs.
2. Choose the gateway with you. We decide between self-hosted and managed based on where your data is allowed to go, what you already run (Cloudflare, Kong, AWS), and who on your team will own it after I leave. You get a short written recommendation with costs, so the decision isn't buried in a Slack thread.
3. Set up keys and budgets that match how you're organized. One virtual key per app or team, with model allow-lists and spend limits, and alerts before a budget runs out instead of after.
4. Move apps over one at a time. Usually it's a base_url change per app. I start with the least critical one, compare cost and latency before and after, and only then move the rest.
5. Test the fallback before you need it. If we configure a fallback model, it runs through the same evals as the primary in CI. I use DeepEval for this, grading tool calls and outcomes, not only the wording of replies. If the fallback can't pass, it doesn't go in.
6. Trace it and hand it over. OpenTelemetry traces connect gateway logs to your agents' own audit trail, so one request can be followed end to end. You get a runbook covering budget changes, adding a new app and what to do when the gateway or a provider is down.
For a small business this is often a managed gateway and a handful of keys. For an enterprise team it's a project with a security review, per-team budgets and deployment in your own cloud. Either way your approval rules stay in your code, where they belong.
If you're trying to work out whether you need a gateway, or you already know you do and want it done properly, get in touch at wildflowersoftware.com. We'll start by looking at where your model traffic goes today.