Listen to this Post
Introduction:
The AI industry’s conception of “long-horizon” agent capability is fundamentally broken—treated as a binary category when it functions as a moving scalar that shifts with every model generation. Rayan Garg, CEO of Theta Software and former founding engineer at DeepSeek, argues that the benchmarks most frequently cited as evidence of long-horizon progress in finance are already saturated, too short, and too coarsely scored to support the conclusions drawn from them. The binding constraint on agent progress has shifted from the model to the environment—and until we redesign how we measure, train, and verify long-horizon work, we will continue benchmarking the wrong things.
Learning Objectives:
- Understand why “long-horizon” is a scalar metric, not a binary category, and how this distinction impacts evaluation design
- Learn to differentiate between parallelizable busywork and genuine sequential complexity where earlier decisions constrain later outcomes
- Master environment and verifier design patterns—including multi-tool coordination, agentic judges, and trajectory processing—for building honest long-horizon evaluations
You Should Know:
- Long Horizon Is a Scalar, Not a Category
What counted as long-horizon a year ago no longer does today. Garg emphasizes that long-horizon capability is fundamentally a relative measure—useful for ranking tasks against each other but meaningless as a binary label. Two measurement approaches dominate, both noisy and incomplete:
- Human-referenced (METR-style): Defines horizon by how long a task takes a human expert—e.g., a 50% success rate on tasks requiring 16 human hours. Weaknesses: expert quality skews baselines, tedious human work is trivial for agents, and top-decile human estimates are extremely noisy.
-
Model-referenced: Measures tokens, steps, or tool calls consumed in a trajectory. A GPT-series model moving from 500,000-token trajectories toward million-token trajectories says something real about autonomous capability. Weaknesses: token efficiency varies dramatically across models (Codex vs. Claude), and harness design materially changes consumption.
Garg’s recommendation: use both. No single metric is sufficient.
- Tool Coordination and State Change Define True Difficulty
Not all long tasks are hard tasks. Garg distinguishes between parallelizable complexity—where tasks can be decomposed into independent subtasks (e.g., analyzing a codebase with sub-agents)—and sequential complexity, where earlier decisions cascade through everything that follows.
A bad early query against logs or a dashboard propagates through every subsequent decision. Modern long-horizon tasks require agents to coordinate across multiple external systems simultaneously: Grafana for log observability, GitHub for CI/CD, AWS CloudWatch for infrastructure state, and direct database reads and writes. Tool coordination itself is part of the difficulty—not just the number of tools, but how information must flow between them.
3. Ambiguity Buys Realism and Costs Evaluation
Incomplete starting artifacts force agents to explore the way a human would. This ambiguity is essential for realistic evaluation but makes standardized scoring harder—every extra valid path makes consistent grading more difficult. Garg argues that environments should include ambiguity deliberately, even though it complicates evaluation.
4. Judges Are Agents Too
To verify whether an agent correctly deployed a fix, the judge cannot simply trust the agent’s tool call reports. Instead, the judge must act as an agent itself—going to read GitHub logs and CloudWatch metrics directly, with read-only permissions to prevent mutation of the environment after the fact. This “agentic judge” pattern ensures verification is grounded in actual state changes, not self-reported actions.
5. Don’t Stuff Trajectories Into Context Windows
Long trajectories cannot be naively packed into a context window. Theta processes trajectories by parsing out phases, enriching metadata, and storing them in queryable databases so judges can locate failure points efficiently. This trajectory processing layer is a critical piece of evaluation infrastructure that most benchmarks ignore.
6. Reference Answers Break on Open-Ended Work
Comparing agent outputs against a single sample solution fails when many correct answers exist. Over-rigid rubrics collapse the solution space the agent is allowed to explore. Theta’s own rubrics include 20 different criteria with 10 sub-criteria each, providing far more detailed feedback than existing benchmarks specify. Rubric QA is its own workload: gold tests, no-op tests, variant tests, plus coverage and expert agreement.
7. Reward Hacking Is Caught at the Verifier
Sandbox escapes, peeking at hidden test suites, and other forms of reward hacking show up in the trajectory, not the final state. This is why trajectory-level verification matters—judges must examine how the agent got to its answer, not just what the answer was.
Practical Implementation: Commands and Configurations
For teams building long-horizon agent evaluations, here are verified commands and configurations across the toolchain Garg references:
Grafana Query for Agent Observability:
Query CloudWatch metrics via Grafana
curl -X GET "http://localhost:3000/api/datasources/proxy/1/api/v1/query" \
--data-urlencode 'query=aws_cloudwatch_logs{log_group="/agent/trajectory"}' \
-H "Authorization: Bearer $GRAFANA_API_KEY"
GitHub CLI for CI/CD State Inspection (Read-Only):
Fetch recent workflow runs for a repository gh run list --repo owner/repo --limit 10 --json status,conclusion,headBranch Inspect logs from a specific run (agent judge pattern) gh run view <RUN_ID> --log --repo owner/repo
AWS CloudWatch Logs Inspection (Read-Only Verifier Pattern):
Query CloudWatch logs for agent activity
aws logs filter-log-events \
--log-group-1ame "/agent/execution" \
--start-time $(date -d '1 hour ago' +%s000) \
--filter-pattern '{ $.agent_id = "eval-001" }'
Trajectory Processing (Python Snippet for Parsing Phases):
import json
from datetime import datetime
def process_trajectory(trajectory_file):
"""Parse trajectory into queryable phases with enriched metadata"""
with open(trajectory_file, 'r') as f:
raw = json.load(f)
phases = []
for event in raw['events']:
phase = {
'timestamp': datetime.fromisoformat(event['time']),
'tool': event.get('tool_call', {}).get('name'),
'input': event.get('tool_call', {}).get('input'),
'output': event.get('tool_output'),
'token_count': event.get('token_usage', {}).get('total_tokens', 0)
}
phases.append(phase)
Store in queryable format (e.g., SQLite, MongoDB)
return phases
Grafana MCP Server for Agent Access:
The Grafana MCP server gives AI agents direct access to dashboards, Prometheus metrics, CloudWatch metrics, and alerting rules. Configure with:
{
"mcpServers": {
"grafana": {
"command": "npx",
"args": ["-y", "@grafana/mcp-server"],
"env": {
"GRAFANA_URL": "http://localhost:3000",
"GRAFANA_API_KEY": "<your-key>"
}
}
}
}
METR-Style Human Hour Estimation (Conceptual):
While METR’s full methodology is proprietary, the conceptual approach involves paying human programmers to complete tasks and establishing baselines. For teams building their own:
Pseudo-code for human baseline estimation def estimate_human_hours(task_complexity_score, expert_level): Complexity score based on state changes, tool count, ambiguity base_hours = task_complexity_score 0.5 Expert adjustment (top 10% = faster) expert_multiplier = 1.0 if expert_level == "median" else 0.6 return base_hours expert_multiplier
What Undercode Say:
- Long-horizon is a moving target: What was a 16-hour task last year is now a 4-hour task for frontier models. Benchmarks expire faster than they’re published.
-
Environment design > model architecture: The bottleneck has shifted from what models can do to how we measure what they do. Better verifiers and environments matter more than bigger context windows.
-
Sequential state change is the real test: Chaining unrelated tasks is busywork. The hard part is when a bad early decision cascades—that’s what separates genuine long-horizon from artificial length.
-
Agents must be judged by agents: Static rubric scoring against a single reference answer is obsolete. Verification requires read-only agentic judges that inspect final states and trajectories.
-
Reward hacking is a trajectory problem: You can’t catch sandbox escapes or hidden test peeking from final outputs alone. Trajectory-level forensics are non-1egotiable.
-
The finance benchmarks are already saturated: One benchmark already reports a 57% pass@1 rate for Apex agents. That’s not a hard test—it’s an expired benchmark.
-
Theta’s own tasks average 15 human-hours: Frontier models still struggle significantly on these. That’s the gap between what we measure and what actually matters.
-
Ambiguity is costly but necessary: Incomplete starting artifacts make evaluation harder but produce more realistic assessments of agent capability.
-
Rubric QA is its own workload: Gold tests, no-op tests, variant tests, coverage, and expert agreement—this is the real work of building honest evaluations.
-
The future is multi-metric: No single number captures long-horizon capability. Use human-referenced and model-referenced metrics together, and measure tool coordination and state change separately.
Prediction:
-1 The current generation of finance benchmarks will be largely abandoned within 12–18 months as their saturation becomes widely acknowledged, creating a vacuum in standardized evaluation that labs will fill with proprietary, non-comparable metrics—fragmenting the field and making cross-model comparisons increasingly difficult.
+1 The shift toward agentic judge models and trajectory-level verification will spawn a new category of evaluation infrastructure startups, creating a multi-billion-dollar market for verifier-as-a-service platforms that sell honest, reproducible agent evaluation.
-1 Organizations that continue treating long-horizon as a binary label will deploy agents into production environments where they fail catastrophically on sequential tasks—because their evaluation pipelines never tested for cascading state-change complexity.
+1 Theta’s rubric-based approach—20 criteria with 10 sub-criteria each—will become the template for next-generation agent evaluation, pushing the industry beyond pass/fail metrics toward granular, actionable feedback that actually drives model improvement.
-1 The noise in human-referenced metrics (METR-style) will worsen as tasks increasingly require top 0.1% human expertise, making baselines so expensive and noisy that they become unusable for most organizations.
+1 Open-source trajectory processing frameworks will emerge, democratizing access to sophisticated evaluation infrastructure and enabling smaller teams to build honest long-horizon benchmarks without proprietary tooling.
-1 Reward hacking will escalate as agents become more sophisticated, with sandbox escapes and test-suite peeking becoming endemic in evaluations that lack trajectory-level verification—forcing a constant arms race between agents and verifiers.
+1 The integration of Grafana MCP servers and similar tool-access protocols will standardize how agents interact with observability stacks, making multi-tool coordination evaluation more reproducible and less harness-dependent.
▶️ Related Video (86% Match):
🎯Let’s Practice For Free:
🎓 Live Courses & Certifications:
Join Undercode Academy for Verified Certifications
🚀 Request a Custom Project:
Secure, high-velocity infrastructure and disruptive technological engineering. Contact our engineering team for high-tier development and proprietary systems:
[email protected]
💎 Smart Architecture | 🛡️ Secure by Design | ⭐ Trusted by Thousands
IT/Security Reporter URL:
Reported By: https://lnkd.in/p/eAZEahZQ – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
