The Core Problem

You see the monthly bill from OpenAI: $2,400. You know something happened. You don't know what. Or which agent did it. Or whether to shut it down or optimize it. Every major LLM observability tool misses this.

The Three Incumbents (And Why They're Not Enough)

Portkey
"Production observability for your LLM stack. 250+ models. One gateway."
$99–$2,000+/mo
What it does well
  • 250+ models across 12+ providers
  • SOC2, HIPAA, GDPR, ISO27001
  • Routing, fallbacks, caching, load balancing
  • Per-model, per-request cost visibility
The problem
  • Shows spend by model, not by agent
  • Air-gapped deployments add $1,000s/mo
  • No per-agent budgets or spend limits
  • Request-level attribution only
Bottom line: Portkey shows you how much you spent on GPT-4, Claude, and Gemini. It doesn't show you which agent spent it. When your orchestrator spawns Agent A → Sub-Agent B → 3 LLM calls, Portkey just sees "3 requests." Cost tracking at the request level, not the agent level.
Helicone
"Open-source observability. Obsessively affordable."
$20/user/mo or $1/10K req
What it does well
  • Fully open-source (MIT license)
  • Cheapest SaaS pricing in the market
  • One-line proxy swap integration
  • Caching, threat detection, rate limiting
The problem
  • Cost per request, not per agent
  • Shallow evaluation tools
  • Limited prompt management on free tier
  • No per-agent spend attribution
Bottom line: Same agent-level blind spot as Portkey. You can segment by user or model, but "how much is my data pipeline agent costing vs. my customer support agent?" — no answer.
LangSmith
"The Datadog for AI. Debug. Test. Evaluate. Deploy."
$39/user/mo
What it does well
  • Deep LangChain/LangGraph integration
  • Best-in-class evaluation tools
  • Prompt versioning (Git for prompts)
  • Framework-agnostic SDK tracing
The problem
  • Routes by function call, not agent identity
  • Per-seat pricing scales painfully
  • Only 14-day or 400-day retention (no middle)
  • Deeply LangChain-tied UX
Bottom line: LangSmith traces beautifully but doesn't aggregate costs by agent. A 10-person team = $390/month minimum. If you're not on LangChain, the UX assumes you are. Priciest per-user option in the category.

The Blind Spot (All Three Miss This)

The question none of them answer: Which of your agents is most expensive?

Portkey
"I don't know."
I can tell you about models.
Helicone
"I don't know."
I can tell you about requests.
LangSmith
"I don't know."
I can tell you about function calls.
What You Actually Need
  • "Agent A spent $500 this month. Agent B spent $200. Agent C spent $50."
  • "Agent A's cost per request is 3× Agent B's. Something's wrong."
  • "If I shut down Agent C, I save $50/month. The ROI is negative."
  • "Agent B spawned 12 sub-agents. What did they cost? Which should I keep?"

This isn't a small feature gap. It's structural. These tools were designed before autonomous agents became the primary way teams run LLMs. Portkey's gateway architecture assumes humans are making the calls. Helicone's request-based model does too. LangSmith's trace model assumes you care about function-level visibility, not agent-level attribution.

They all miss that teams now need to budget like they have employees.

Feature Comparison: The Real Picture

Capability Portkey Helicone LangSmith What You Need
Free Tier 10K req/mo 10K–100K req/mo 5K traces/mo Generous free tier
Cheapest Paid $99/mo $20/user or $1/10K $39/user/mo Predictable pricing
Model Coverage 250+ ✓ All providers ✓ All providers ✓ Multi-provider
Cost per Request ✓ ✓ Traces only Cost visibility
Cost per Agent ✗✗ ✗✗ ✗✗ UNMET
Per-Agent Budgets ✗ ✗ ✗ UNMET
Evaluation Tools Basic Basic Best-in-class Quality > Cost
Caching ✓ ✓ ✗ Cost optimization
Open-Source Option Enterprise only ✓ Full Enterprise only Sovereignty
Proxy Integration ✓ ✓ SDK only Easy setup
LangChain Integration ✗ ✗ ✓✓ Nice-to-have

The takeaway: These tools solve the "what happened" problem. None solve the "which agent spent it" problem.

Who Should Use What (Today)

Use Portkey if:
  • You need 250+ model support
  • Enterprise compliance (HIPAA, GDPR, SOC2)
  • On-premises / air-gapped deployment
  • Fine with $99+/mo for a managed gateway
Skip it if: You have multiple agents and need per-agent spend tracking.
Use Helicone if:
  • Cost is your primary constraint
  • You want open-source (self-host free)
  • Request-level tracking is enough for now
  • You value fast community iteration
Skip it if: You need evaluation tools, prompt management, or agent-level cost attribution.
Use LangSmith if:
  • 100% committed to LangChain/LangGraph
  • Debugging complex chain behavior
  • Small team (1–5 people)
  • Need best-in-class evaluation tools
Skip it if: You run heterogeneous agents, have a big team, or need agent-level cost tracking.

The Unmet Market Need

Autonomous agents are moving from research projects to production systems. As teams run 5, 10, 20 agents simultaneously, cost accountability becomes critical.

No incumbent has solved per-agent cost attribution because it wasn't a priority when they launched. Portkey launched as a gateway. Helicone launched for request-level observability. LangSmith launched for LangChain tracing.

Agent-level cost tracking requires:

  1. Agent identity as a first-class concept — not bolted on to request-level data
  2. Cost attribution across nested agent calls — agents spawn sub-agents spawn tool calls
  3. Per-agent budgets and spend limits — like autoscaling guardrails for costs
  4. Multi-provider spend aggregation — know the cost-per-token across OpenAI + Anthropic + Google
  5. MCP-first cost tracking — as agents use external tools, track what they cost
What the market needs next
  • Cost-conscious startups: "I have 3 agents. Which one is expensive? I don't want to pay $39/user/month."
  • AI Ops teams: "I need chargeback cost attribution per agent per team. Hard spend limits that auto-pause agents."
  • Multi-agent orchestrators: "My agents call other services, APIs, and MCP tools. I need end-to-end cost tracking."
  • Autonomous workflows: "20 agents. Some are profitable. Some cost more than they generate. Tell me which is which."

These needs are real today. No tool solves them.

The Bottom Line

Portkey, Helicone, and LangSmith are all excellent tools. Use them if their strengths match your needs:

But understand what they can't do: track spend per autonomous agent.

That gap exists because the market didn't demand it until now. The tool that owns agent cost attribution wins the next 18 months.