SQRL Just Beat Claude Opus at SQL with a 4B Model – Here’s How It Probes Your Database Before It Writes a Single Query + Video

By /

Listen to this Post

Featured Image

Introduction:

Text-to-SQL has long been framed as a translation problem: convert natural language into SQL and execute it. But this framing is dangerously incomplete. A query can be perfectly valid SQL – syntactically flawless, execution-ready – and still return the wrong answer. Wrong joins, misread columns, filters for values that don’t exist – none of these throw errors. You just get the wrong result, and your dashboard or report acts on it. Feyn Labs just shattered this paradigm with SQRL, a family of text-to-SQL models that don’t guess – they inspect. The flagship SQRL-35B-A3B (only ~3B active parameters) hit 70.60% execution accuracy on BIRD Dev, edging Claude Opus 4.6 at 68.77%. Even more staggering: the distilled SQRL-4B matches Opus at 68.80% in a model small enough to run anywhere.

Learning Objectives:

  • Understand why schema-only text-to-SQL systems produce silent, wrong answers and how the BIRD benchmark exposes these failures
  • Master SQRL’s inspection-first architecture: read-only probes, observation loops, and the RL-trained teacher-student distillation pipeline
  • Deploy SQRL checkpoints locally using vLLM with a read-only harness for secure, self-hosted text-to-SQL
  • Implement probe strategies and observation parsing to resolve ambiguous column values, joins, and filters against live data
  • Apply SQL validation techniques and execution accuracy measurement to evaluate and harden your own text-to-SQL pipelines
  1. The Inspection Loop: How SQRL Looks Before It Leaps

Traditional text-to-SQL models receive a question and a database schema – table names, column names, data types, and sometimes relationships. That’s it. They never see the actual data. Schema tells you a column exists; it doesn’t tell you whether values are stored as “Alameda”, “Alameda County”, or “ALAMEDA”. It cannot reveal which join produces duplicate rows or which of three “revenue” columns across different tables the user actually means.

SQRL changes the workflow from pure translation to inspection-first. The model receives the question, the schema, and optional evidence. If the context is sufficient, it emits SQL immediately. If ambiguity remains – and in production, it usually does – SQRL fires read-only probe queries against the live database, observes the returned rows, and only then crafts the final answer.

Step-by-Step Guide: Understanding the SQRL Inspection Loop

  1. Input Reception: SQRL receives a natural language question, database schema, and optional evidence (e.g., business context).
  2. Context Sufficiency Check: The model evaluates whether the schema alone provides enough information to generate a correct query.
  3. Probe Execution (if needed): If ambiguity exists, SQRL generates and executes read-only exploration queries – `SELECT DISTINCT` probes, `COUNT` checks, `LIMIT` samplers.
  4. Observation Parsing: Returned rows are wrapped in `` tags and fed back to the model.
  5. Final Query Generation: Armed with actual data samples, SQRL writes the final SQL query.
  6. Commit: The final query executes with normal credentials; the inspection harness runs probes with read-only permissions.

Key Constraint: The model is allowed a maximum of five probes per question, keeping latency low while gaining the context needed to avoid silent errors. Most questions finish in one or two steps.

  1. The RL Training Pipeline: Why Execution Accuracy, Not Wording, Matters

SQRL’s teacher model was trained using reinforcement learning with a reward function that is brutally simple yet profoundly effective: int(execute(predicted) == execute(gold)). The model is rewarded only when its generated SQL produces the exact same result set as the ground-truth query. Wording, style, and intermediate steps are irrelevant – only the final output matters.

Feyn Labs applied this RL training exclusively on the “mixed zone”: questions where some of eight attempts succeed and others fail. Uniform outcomes – all success or all failure – carry zero learning signal. This targeted approach produced 10,200 verified probe→observe→answer trajectories, which were then distilled into the 4B and 9B student models.

Step-by-Step Guide: Implementing RL for Text-to-SQL (Conceptual)

  1. Define Reward Function: `reward = 1 if execute(predicted_sql) == execute(gold_sql) else 0`
    2. Curate Training Data: Focus on “mixed zone” queries where model attempts show variance in execution accuracy.
  2. Generate Trajectories: For each query, allow the model to run up to 5 probes, observe results, and generate a final query. Record the full probe→observe→answer sequence.
  3. Train Teacher Model: Apply RL (e.g., PPO) using the execution-match reward signal.
  4. Distill to Students: Use the teacher’s 10,200+ verified trajectories to train smaller 4B and 9B models via supervised fine-tuning.
  5. Validate: Test distilled students on BIRD Dev; SQRL-9B retained ~99% of the teacher’s performance at 69.80%.

  6. Deployment: Running SQRL with vLLM and Read-Only Harness

All three SQRL checkpoints – SQRL-4B, SQRL-9B, and SQRL-35B-A3B – are open weights on Hugging Face. The 35B-A3B is a Mixture-of-Experts model: 35 billion total parameters, but only ~3 billion active per forward pass, meaning inference cost closer to a 3B model. Deployment runs through vLLM with a read-only harness that intercepts exploration queries before they reach the database.

Linux/macOS Deployment Commands

 Install vLLM
pip install vllm

Download SQRL-4B from Hugging Face
git lfs install
git clone https://huggingface.co/feyninc/sqrl-4b

Launch vLLM server with read-only harness configuration
python -m vllm.entrypoints.openai.api_server \
--model ./sqrl-4b \
--tensor-parallel-size 1 \
--max-model-len 4096 \
--port 8000

Test with a sample query (using curl)
curl http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"prompt": "System: You are SQRL. Generate SQL. Use <sql> for probes and <answer> for final query.\n\nUser: How many customers signed up in Q1 2025?\n\nSchema: customers(id, name, signup_date, status)",
"max_tokens": 512
}'

Windows Deployment (PowerShell)

 Install vLLM (requires Python 3.8+)
pip install vllm

Clone the model
git clone https://huggingface.co/feyninc/sqrl-4b

Launch server
python -m vllm.entrypoints.openai.api_server --model .\sqrl-4b --port 8000

Test with Invoke-WebRequest
$body = @{
prompt = "System: You are SQRL. Generate SQL. Use <sql> for probes and <answer> for final query.<code>n</code>nUser: How many customers signed up in Q1 2025?<code>n</code>nSchema: customers(id, name, signup_date, status)"
max_tokens = 512
} | ConvertTo-Json

Invoke-RestMethod -Uri "http://localhost:8000/v1/completions" -Method Post -Body $body -ContentType "application/json"

Security Note: The read-only harness must use a database connection with `READ ONLY` privileges. Exploration queries execute in a sandboxed connection; final SQL runs with normal credentials. Your schema, queries, and observations stay on infrastructure you control.

  1. Probe Strategies: What to Look For and How to Parse Observations

The quality of SQRL’s probes determines the quality of its final query. Effective probes target specific ambiguities:

  • Value Ambiguity: `SELECT DISTINCT column_name FROM table LIMIT 10` reveals actual value formats
  • Join Validation: `SELECT COUNT() FROM table1 t1 JOIN table2 t2 ON t1.id = t2.id LIMIT 1` verifies join cardinality
  • Filter Existence: `SELECT COUNT() FROM table WHERE filter_column = ‘expected_value’` confirms values exist
  • Column Disambiguation: When multiple columns share names, probe each with `SELECT column FROM table LIMIT 5`

Step-by-Step Guide: Implementing Probe Parsing

  1. Identify Ambiguity Markers: Parse the question for vague terms (e.g., “active”, “recent”, “large”) that require data inspection.
  2. Generate Probe SQL: Construct read-only queries targeting the ambiguous elements.
  3. Execute and Capture: Run probes against a read-only database connection.
  4. Parse Observations: Extract returned rows from `` tags.
  5. Incorporate into Context: Append observation data to the model’s input context before final query generation.
  6. Generate Final Query: Produce the answer SQL using the enriched context.

Python Snippet: Observation Parser

import re
import json

def parse_observation(response_text):
"""Extract observation data from SQRL response."""
pattern = r'<observation>(.?)</observation>'
matches = re.findall(pattern, response_text, re.DOTALL)
observations = []
for match in matches:
try:
 Attempt to parse as JSON rows
data = json.loads(match)
observations.append(data)
except json.JSONDecodeError:
 Fallback: treat as plain text
observations.append(match.strip())
return observations

Example usage
response = """<sql>SELECT DISTINCT status FROM customers LIMIT 5</sql>
<observation>[{"status": "active"}, {"status": "inactive"}, {"status": "pending"}]</observation>
<answer>SELECT COUNT() FROM customers WHERE status = 'active'</answer>"""

obs = parse_observation(response)
print(f"Discovered status values: {obs}")  Output: [{'status': 'active'}, ...]
  1. Benchmarking and Validation: Measuring Execution Accuracy on BIRD

The BIRD benchmark (95 real databases, 33.4 GB of actual data across 37 professional domains) scores systems by executing generated SQL and comparing returned rows against a reference result. This is fundamentally different from exact-match string comparison – a query can be syntactically different but semantically correct.

Step-by-Step Guide: Validating SQL Execution Accuracy

  1. Set Up Test Harness: Create a sandboxed database instance with the BIRD Dev dataset.
  2. Execute Predicted SQL: Run the model-generated query against the test database.
  3. Execute Gold SQL: Run the ground-truth query against the same database.
  4. Compare Result Sets: Check if the returned rows (order-independent) match exactly.

5. Calculate Accuracy: `accuracy = correct_predictions / total_queries`

  1. Log Failures: For incorrect predictions, record the predicted SQL, gold SQL, and both result sets for debugging.

SQL Validation Commands (PostgreSQL Example)

-- Create a temporary table to store results
CREATE TEMP TABLE predicted_results AS 
EXECUTE 'YOUR_PREDICTED_SQL_HERE';

CREATE TEMP TABLE gold_results AS 
EXECUTE 'YOUR_GOLD_SQL_HERE';

-- Compare result sets (order-independent)
SELECT 
(SELECT COUNT() FROM predicted_results) = (SELECT COUNT() FROM gold_results) 
AND NOT EXISTS (
SELECT  FROM predicted_results 
EXCEPT 
SELECT  FROM gold_results
) AND NOT EXISTS (
SELECT  FROM gold_results 
EXCEPT 
SELECT  FROM predicted_results
) AS execution_match;

6. Security Hardening: Read-Only Probes and Injection Prevention

SQRL’s inspection architecture introduces a critical security consideration: the model generates SQL queries that execute against your database. While probes are read-only, they must be properly sandboxed to prevent injection or privilege escalation.

Step-by-Step Guide: Hardening Your Text-to-SQL Deployment

1. Create a Dedicated Read-Only User:

-- PostgreSQL
CREATE USER sqrl_ro WITH PASSWORD 'secure_password';
GRANT CONNECT ON DATABASE your_db TO sqrl_ro;
GRANT USAGE ON SCHEMA public TO sqrl_ro;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO sqrl_ro;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO sqrl_ro;
  1. Restrict Probe Query Complexity: Set statement timeout and row limits to prevent resource exhaustion. 
    -- Set per-query limits
SET statement_timeout = '5s';
SET max_rows = 1000;

  2. Validate and Sanitize Generated SQL: Use a SQL parser (e.g., `sqlparse` in Python) to verify that generated queries contain only `SELECT` statements before execution.

  3. Implement Query Logging and Auditing: Log all probe and final queries with timestamps, user IDs, and execution results for forensic analysis.

  4. Use Connection Pooling with Read-Only Mode: Configure database connection pools to enforce read-only transactions.

    Python example with psycopg2
import psycopg2
conn = psycopg2.connect(
dbname="your_db",
user="sqrl_ro",
password="secure_password",
host="localhost",
options="-c default_transaction_read_only=on"
)

  5. Production Pitfalls: Why Benchmark Scores Don’t Always Translate

The gap between benchmark scores and production reality is not subtle. GPT-4o scores 86.6% on Spider (a standard text-to-SQL benchmark). On Spider 2.0, which uses real enterprise schemas, that number drops to 10.1%. Snowflake’s team measured GPT-4o at 51% accuracy on 150 actual internal BI questions. The dbt 2026 benchmark summary is blunt: “Text-to-SQL will cheerfully give you a wrong number”.

SQRL’s 70.6% on BIRD Dev is meaningful precisely because BIRD is closer to production than Spider. But production readiness requires more than benchmark scores.

Step-by-Step Guide: Productionizing SQRL

  1. Schema Caching: Pre-load and cache schema information to reduce latency on repeated queries.
  2. Probe Result Caching: Cache results of common probes (e.g., SELECT DISTINCT status) to avoid repeated database hits.
  3. Fallback Mechanisms: Implement a human-in-the-loop fallback for queries where SQRL’s confidence is low.
  4. Continuous Evaluation: Run periodic evaluations against a held-out test set to detect model drift.
  5. Feedback Loop: Collect user corrections and use them to fine-tune or prompt-engineer the model for your specific domain.
  6. Cost Monitoring: Track probe count and query complexity to manage database load and API costs.

What Undercode Say:

  • Key Takeaway 1: SQRL’s 4B model matching Claude Opus at SQL is not an anomaly – it’s the logical outcome of shifting from translation to inspection. The model doesn’t need to know your data; it needs permission to look. This insight applies far beyond SQL: any AI system that generates code or queries against live systems should inspect before it commits.

  • Key Takeaway 2: The RL training strategy – focusing exclusively on the “mixed zone” where outcomes vary – is a masterclass in efficient data utilization. By ignoring uniform successes and failures, Feyn Labs concentrated computational resources on the hardest 10.2K trajectories and distilled that knowledge into models 120× smaller than their competitors.

  • Analysis: The SQRL release signals a broader shift in enterprise AI: small, specialized, inspection-capable models are becoming more valuable than massive general-purpose models. The ability to self-host SQRL-4B on your own infrastructure means your schema, queries, and observations never leave your control. For regulated industries (finance, healthcare, government), this is not just a performance advantage – it’s a compliance necessity. The open-weight release on Hugging Face democratizes access to state-of-the-art text-to-SQL, enabling teams to build production-grade data interfaces without paying per-token API costs. However, organizations must invest in the surrounding infrastructure: read-only harnesses, query validation, and continuous evaluation pipelines. The model is only as good as the permissions and safeguards you build around it.

Prediction:

  • +1 SQRL’s inspection-first architecture will become the default paradigm for text-to-SQL within 18 months. Competitors will rush to implement similar probe-based workflows, and the BIRD benchmark will evolve to measure inspection efficiency (probe count, latency) alongside execution accuracy.

  • +1 The distillation approach – training a massive teacher on RL and distilling into small students – will proliferate beyond SQL. Code generation, API orchestration, and even cybersecurity threat hunting will adopt this pattern: let a large model explore and reason, then distill the trajectories into a fast, self-hostable specialist.

  • -1 Organizations that deploy SQRL without proper read-only harnesses and query validation will experience data exposure incidents. The model’s ability to generate arbitrary SQL – even read-only – creates new attack surfaces if probes are not properly sandboxed and logged.

  • +1 The open-weight release of SQRL-4B and SQRL-9B will accelerate the commoditization of enterprise-grade text-to-SQL. Small teams will build custom data chatbots, internal BI tools, and automated reporting systems without relying on proprietary APIs, reducing costs and increasing data sovereignty.

  • -1 The 68.80% accuracy of SQRL-4B on BIRD Dev, while impressive for its size, still means nearly one in three queries returns the wrong answer. Production systems must incorporate confidence scoring, human review loops, and automated validation to prevent incorrect data from reaching decision-makers.

  • +1 Feyn Labs’ focus on “execution accuracy” over syntactic correctness will reshape how we evaluate AI systems. The industry will move toward outcome-based benchmarks that measure what actually matters – correct results, not pretty code.

References & Resources

▶️ Related Video (60% Match):

https://www.youtube.com/watch?v=2ku4WKzRweE

🎯Let’s Practice For Free:

🎓 Live Courses & Certifications:

Join Undercode Academy for Verified Certifications

🚀 Request a Custom Project:

Secure, high-velocity infrastructure and disruptive technological engineering. Contact our engineering team for high-tier development and proprietary systems:
[email protected]
💎 Smart Architecture | 🛡️ Secure by Design | ⭐ Trusted by Thousands

IT/Security Reporter URL:

Reported By: Pawel Bulowski – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅

🔐JOIN OUR CYBER WORLD [ CVE News • HackMonitor • UndercodeNews ]

💬 Whatsapp | 💬 Telegram

📢 Follow UndercodeTesting & Stay Tuned:

𝕏 formerly Twitter 🐦 | @ Threads | 🔗 Linkedin | 🦋BlueSky

Related Posts: