Before Rushing to Train Models When AI Product Performance Stalls: 13 Practical Talks Curated by Hamel Husain

Hamel Husain (a leading practitioner in LLM evaluation and one of the core instructors of the AI Evals course) hosted 13 sessions on "AI Product Engineering," distilling 9.5 hours of discussions into notes that can be read in 20 minutes. This explainer covers all 13 sessions in full; complete recordings for each session are available on YouTube or Maven, with links included at the end of each section.

Turning large language models into effective products offers numerous knobs to tune. These sessions recommend the following sequence of priorities:

1. Evaluation is the foundation. An evaluation consists of a fixed set of test cases paired with explicit scoring rubrics. Only when system performance can be measured accurately will you know whether subsequent changes represent progress or regression.
2. Improve retrieval and context first. This is where effort pays off fastest: ensure the material fed into the model is correct first.
3. Refine the system and harness next. The harness refers to the software layer wrapping the model that manages tool invocation and orchestration; this layer also encompasses inference latency, hardware economics, and execution runtime environments.
4. Reserve post-training for last. Only consider modifying the model itself after exhausting the preceding interventions. Modifying evaluations, retrieval, and the harness involves altering code, prompts, and configurations—efforts that are inexpensive, yield rapid returns, and can be easily reverted if things break. Post-training requires data curation, compute expenditure, and significant time. Without dependable evaluations, there is no way to tell after training whether the model actually improved or simply degraded in other areas.

If you are facing a specific challenge, you can jump directly to the relevant section:

• Inconsistent results or unsure where failures originate: Error Analysis, Lessons from an Education Platform
• Excessive API costs running large models on classification tasks: Model Cascades
• Data analysis Agents repeatedly making errors when querying databases: Data Analysis Agents
• RAG retrieval dropping critical details or missing targets: Multi-Vector Retrieval, Choosing Embedding Models, Search Agents
• Layout and parsing breakdowns on PDFs and scanned documents: Choosing OCR
• Agents dropping critical decisions when rewriting long documents: Subtext
• Slow generation and long user wait times: Inference Latency
• Planning to deploy open-source models but struggling to calculate VRAM requirements: The Economics of Open-Source Models
• Security risks or unexpected crashes when Agents execute code: Sandboxing Agents
• Considering model fine-tuning but unsure whether it is warranted: Post-Training

Evaluation: Pinpointing Where Failures Occur

This tier answers the question: "How do you know if a change actually helped?" Every time you tune a prompt or modify business logic, you rerun a fixed benchmark suite to confirm via objective scores whether the system advanced or regressed. The following four sections cover: what a good benchmark should look like, how to uncover real errors first, how to reduce operational costs while maintaining target accuracy, and the hard-earned lessons from a production team.

Data Analysis Agents: Failures Stem Primarily from Planning, Not Data Retrieval

Data analysis Agents connect to corporate data warehouses, write their own queries or scripts, and answer business questions such as "which customer cohort has the highest churn rate?" Tasks of this nature originally belonged to data analysts and represent a substantial fraction of knowledge work, yet few benchmarks exist that rigorously evaluate Agent performance on them.

Shreya Shankar presented the Data Agent Benchmark (DAB). Real-world enterprise data warehouses are messy, and DAB deliberately incorporates four categories of practical complications:

• Multi-database integration: Data required for a single task is scattered across at least two distinct database systems.
• Non-standard join keys: For instance (illustrative), the same user might be recorded as USER_1001 in the user table, but as usr-1001 or 1001 in the orders table; the Agent must infer that these refer to the identical entity before it can join the tables.
• Unstructured text transformation: Crucial metrics are buried inside free-form text fields and must be extracted before analysis.
• Domain knowledge: Solving the task requires understanding domain-specific industry terminology.

The paper compared five existing benchmark categories (Text-to-SQL, Table QA, Data Science & Engineering, Semantic Query Processing, and General Tool Calling). DAB is the only benchmark covering all four challenges; neither of the first two challenges is covered by any of the other five categories.

While DAB is a general benchmark that addresses questions distinct from the specific evaluations you construct for your own products, its failure analysis offers critical insights for engineering data Agents: Agents fail most frequently due to flaws in planning and execution logic, rather than selecting the wrong data. Agents often draft an initial plan at the start of a task; halfway through, when incoming data contradicts the assumptions in the plan, they stubbornly continue executing the original plan anyway. When debugging your own data Agents, this failure mode warrants special scrutiny. DAB openly accepts submissions, and the results are tracked on the DAB Leaderboard, which the team updates as new models are released.

Shreya's full presentation on DAB is available to watch on YouTube.

Model Cascades: Route Confident Queries to Small Models, Escalate the Rest to Large Models

Many production LLM use cases are fundamentally classification tasks, such as tagging customer support tickets by urgency or moderating content for policy violations. Because large models are so convenient, teams routinely funnel every request to the largest, most expensive model; when traffic scales, costs quickly become unsustainable.

BARGAIN (presented by Shreya, paper arXiv:2509.02896) aims to offload the vast majority of classification workloads to smaller models while strictly preserving a user-defined accuracy target. The scenario demonstrated on the whiteboard involved binary yes/no ticket triage: evaluating directly with gpt-5.6 sol is accurate but prohibitively expensive. BARGAIN routes incoming tickets to the economical gpt-5-nano first; along with its prediction, the small model outputs a confidence score (a number between 0 and 1 indicating how certain it is, where 0.98 signifies high confidence and 0.55 is essentially a guess). A threshold τ is established: if the confidence exceeds τ, the small model is considered sufficiently certain, and its prediction is accepted directly; if confidence does not exceed τ, the request is escalated to the large model for re-evaluation.

Raising the threshold τ routes more samples to the large model, yielding higher accuracy at higher cost; lowering it retains more samples with the small model, cutting costs but incurring more small-model errors. Note that accuracy here measures the proportion of final outputs that agree with the large model's verdict, not agreement with human ground-truth labels. The objective of a model cascade is to replicate the large model's answers at lower cost; whether the large model itself is correct is a separate question addressed by different evaluations.

Prior to deployment, the threshold τ is calibrated against a validation split through the following steps:

1. Sample roughly 500 examples, run them through the small model, and record each prediction and confidence score.
2. Evaluate those same 500 examples with the large model to serve as the reference standard.
3. Validate calibration on this sample set: verify whether higher confidence scores from the small model genuinely correlate with higher agreement with the large model. If the small model frequently disagrees with the large model even when highly "confident," its confidence metric is uncalibrated, and routing by confidence breaks down.
4. Test every confidence score observed in the sample set as a candidate threshold, simulate the resulting agreement rate and routing cost, and pick the lowest-cost threshold among those that meet the target (such as 95% agreement with the large model).

Evaluated across 8 datasets, the paper reports that BARGAIN achieves cost savings up to 86% greater than existing baseline methods.

The follow-up Task Cascades paper introduces three further optimizations: rewriting prompts into simplified queries that cheaper models can handle; reading only the most relevant passages of each document instead of full texts to conserve input tokens; and searching across diverse cascade combinations to identify the lowest-cost configuration that satisfies the accuracy threshold. Compared to standard cascades like BARGAIN, this delivers an additional 48.5% average cost reduction.

For the full presentation, see the recording of Shreya's talk on model cascades.

Error Analysis: Inspect the Data Before Writing Rubrics

When a system underperforms, a common mistake is drafting an evaluation rubric before inspecting any production data. Failure modes conceived purely in the abstract frequently fail to materialize in production, while the actual issues causing real-world breakdowns are omitted entirely.

Instead, one should start with error analysis: manually reviewing individual execution traces (the step-by-step inputs, outputs, and tool calls generated while completing a task) and clustering recurrent errors into a failure taxonomy—such as the lesson plan assistant generating "two learning objectives simultaneously," detailed in the next section. This is the most labor-intensive phase of evaluation. The Error Discovery skill presented by Shreya eliminates much of this friction: as you annotate traces, it dynamically updates the failure taxonomy, and whenever a new failure pattern is identified, it retroactively scans previous traces for matching instances. Applying active learning principles, it bypasses uniform random sampling and instead prioritizes traces most likely to expose novel failure modes or those requiring human judgment.

Its workflow operates in three steps:

1. Spin up a local review UI: Launch a local server using Python's standard library to serve a single-file web application with zero external dependencies. The interface uses distinct colors, spacing, and typography to separate roles, hierarchy, and code from prose, rendering lengthy execution traces legible.
2. Select the initial batch of samples: Extract features and cluster traces into 6–10 groups, selecting 15–25 highly diverse samples to begin inspection—with 60%–70% drawn as cluster centroids and 30%–40% sampled randomly, with duplicates removed.
3. Alternating human-Agent loop: The human reads samples, highlights text, adds notes, and accepts or rejects Agent suggestions; the Agent updates the failure taxonomy based on these annotations and samples the next batch of traces back into the interface.

The Agent operates in two modes: Breadth Mode discovers previously unseen failure modes by proposing diverse new samples; Depth Mode investigates a specific failure mode exhaustively, dispatching a dedicated sub-Agent per failure category to search across traces for matching instances. The end result is a structured failure taxonomy, a verified set of reviewed samples, and a summary of coverage and convergence metrics.

To see this skill in action, check out Shreya's Error Discovery demo recording.

Lessons from an Education Platform: Inter-Annotator Agreement Lower Than Random Chance

Lucas Machado Rocha leveraged systematic evaluation to optimize the lesson-planning assistant at Nova Escola, a Brazilian non-profit educational platform serving approximately 1 million monthly active users and 200,000 teachers. His team encountered two major missteps:

First, they drafted grading rubrics prior to conducting error analysis. Consequently, the vast majority of criteria in their rubric addressed failure modes that never actually occurred in production outputs, wasting significant labeling effort on unusable data. Second, they built a dedicated automated evaluator for an issue that was trivially resolved via prompt engineering. The assistant occasionally generated two learning objectives when standard curriculum guidelines required exactly one; once the team tweaked the prompt, the issue vanished completely. Hamel discusses the trade-offs regarding when a failure mode justifies an automated evaluator in his Evals FAQ.

Annotation efforts exposed an even deeper flaw: when two human annotators independently scored the same batch of lesson plans, their rate of agreement was lower than random chance. That two diligent annotators reviewing the same lesson plan could produce near-zero consensus indicated the problem lay not with the annotators, but in the team's failure to explicitly define "what constitutes a good lesson plan," leaving each person to evaluate according to their subjective assumptions. The team collaborated with pedagogical experts to overhaul the rubric and re-annotated the dataset from scratch.

Once failure modes were identified through error analysis, Lucas used eval skills to automate downstream stages, including authoring LLM-as-a-judge evaluators (programs prompting an LLM to score outputs against the rubric) and calibrating evaluator predictions against human annotations. An evaluator is itself a model; without auditing against human labels, its reliability remains unknown. Following deployment, they routed 2% of live production traffic through this evaluation suite daily to catch regressions early—such as degraded output quality following a prompt change or model upgrade.

To explore Lucas's end-to-end methodology in detail, watch the recording of his talk.

Retrieval and Context: Often the Problem Lies in What You Feed the Model

When a model provides an incorrect answer, the culprit is often not a lack of model intelligence, but defective context: information was missed, granular details were lost in compression, or formatting was corrupted. Interventions at this layer deliver the fastest return on investment. The following five sections detail how to preserve granular detail in retrieval, evaluate and diagnose search Agents, select embedding models, parse PDFs correctly, and rewrite documents without dropping critical decisions.

Multi-Vector Retrieval: Retain a Vector for Every Token, Then Drive Down the Cost

Conventional dense retrieval compresses an entire document into a single vector (an array of numbers capturing semantic meaning), scoring relevance by measuring proximity between query and document vectors. By analogy, this is like condensing an entire novel into a single-sentence blurb: the overarching gist remains, but specific numbers, proper names, and technical parameters are smoothed away—creating an information bottleneck.

The multi-vector retrieval approach outlined by Marek Galovic bypasses this bottleneck by preserving a separate vector for every token in both query and document. Relevance is computed via MaxSim: each token in the query searches across all document tokens to find the single most semantically similar match, recording that maximum similarity score; the document's total score is then the sum of these maximum scores across all query tokens. In this manner, every word in the query finds its exact counterpart in the document, preventing granular details from being lost during compression.

Marek's slides highlighted that this methodology achieves state-of-the-art results in cross-domain and long-document retrieval. The catch is cost: every token embedding in the corpus must be stored, and during search, every query token must be compared against every document token, scaling comparison volume quadratically with sequence lengths. Marek estimates that MaxSim requires approximately 2000 times the compute of a single dense vector dot product and demands 10 to 100 times more storage, which long kept it confined to academic papers.

To make it practical at enterprise scale, Marek presented the primary optimization: SMVE. Its core strategy is to filter out the vast majority of documents using an inexpensive index first, running exact MaxSim calculations only on the small surviving subset. Specifically, each token embedding is projected onto a large set of random directions, keeping only the top 8 dimensions with the largest values; summing these across all tokens in a document yields a sparse vector with mostly zero entries, which is indexed in an inverted index. An inverted index functions much like a book index: you jump directly to the pages containing the indexed term, ignoring pages where the term never appears. During retrieval, documents sharing zero active dimensions with the query are pruned immediately, confining expensive MaxSim scoring to a handful of candidate documents.

Marek reported that at a scale of 1 billion documents, p99 latency remains under 100 ms—meaning 99% of queries return within 100 ms. His team also released Iso-ModernColBERT: a calibrated checkpoint of GTE-ModernColBERT-v1 whose vector geometry aligns cleanly with SMVE's random projections, running roughly 3× faster under bf16 precision with virtually no degradation in ranking quality.

Marek's full session on multi-vector retrieval is hosted on Maven.

Search Agents: The Bottleneck Is Frequently the Retriever

A search Agent does not answer from a single retrieval step; it iteratively decides what to query, analyzes retrieved content, identifies gaps in its knowledge, and issues follow-up queries over multiple turns before synthesizing an answer. Engineering such Agents encounters three common hurdles: lacking realistic benchmark queries, inability to pinpoint where trajectories derail, and ambiguity over whether failures stem from the retriever or the generator.

Nandan Thakur's session presented three projects directly addressing these three bottlenecks.

ORBIT: Synthesizing multi-hop benchmark queries at zero cost. In the absence of human-annotated test sets, a standard shortcut is prompting an LLM to generate questions from source documents; however, questions generated this way can usually be answered in a single search hop. ORBIT inverts this process: it describes multiple attributes of an entity without naming it, verifying that the question is sufficiently challenging to require multi-hop exploration. For example (illustrative, not an actual benchmark question): "A writer was born in a coastal town, won a debut award for his first novel, and later transitioned to directing films. What was the title of his second film?" Answering this correctly requires first discovering the person's identity and then querying their filmography.

Because synthetic questions can contain factual errors, every query undergoes strict validation: a search Agent must verify each factual claim in the question against source documents, while an independent judge model, provided solely with those documents, must arrive at the identical answer. The ORBIT pipeline generated 20,000 verified questions with zero expenditure on commercial search APIs or proprietary LLMs. The entire workflow is fully synthetic, accessible even to teams lacking human-annotated data, and readily applicable to proprietary corpora.

Hawkeye: Visualizing step-by-step trajectory execution. Hawkeye is a visual analytics interface that unrolls every step of a search Agent's execution trace. A dataset from the presentation slides highlights: across 781 runs, 444 succeeded, 46 failed with incorrect answers, and 291 yielded no answer; successful runs averaged 18.9 search rounds, whereas runs yielding no answer averaged 42.2 rounds. In other words, runs that drag on through dozens of search rounds have almost certainly lost the scent. Armed with these insights, teams can configure an optimal maximum search budget, terminating fruitless loops once the limit is reached. With Hawkeye's paper currently under review and no public tool released yet, you can instruct your daily coding Agent to implement a comparable inspection view following this design pattern.

BrowseComp-Plus: Given the right documents, the model almost always succeeds. BrowseComp-Plus is a reproducible reproduction of OpenAI's BrowseComp. BrowseComp consists of hard-to-find factual queries: the answers are brief and straightforward to verify, but require extensive search iterations to locate. Its central finding: the primary bottleneck capping accuracy is the retriever, not the reasoning model. Testing the same GPT-4.1 under two conditions revealed that when relying on BM25 (a traditional keyword-frequency ranking algorithm) to retrieve materials independently, it answered correctly only 14.6% of the time; when directly provided with the documents containing the answer, its accuracy jumped to 93.5%. The model remained identical; the performance gulf stemmed almost entirely from whether the correct documents were retrieved.

Nandan's full session on search Agents is available on Maven.

Choosing Embedding Models: Look Beyond Leaderboards, and Don't Overlook Fine-Tuning

In RAG pipelines, the embedding model converts document chunks and queries into dense vectors; whether retrieval succeeds depends heavily on this step. When selecting models, engineers often consult the MTEB Leaderboard, yet MTEB ranks models solely by quality and rarely accounts for operational cost.

Radu Gheorghe outlined several critical engineering trade-offs. First, precision: this denotes how many bits a computer uses to store a numerical value. Fewer bits conserve memory and accelerate computation at the expense of slight rounding coarseness—analogous to storing 3.14159 as 3.14.

• Model quantization (the precision used to store model weights, primarily impacting compute speed): On CPUs, INT8 (8-bit) models run roughly 3× faster than FP32 (32-bit) with negligible quality degradation; on GPUs, switch to FP16.
• Embedding precision (the bit-width used to store output vector dimensions, primarily impacting storage footprint): Switching from FP32 to binary embeddings compresses each 32-bit float into a single 1-bit binary flag (0 or 1), cutting storage to 1/32; adopting bfloat16 (16-bit) cuts storage in half with no measurable loss in retrieval quality.
• Matryoshka representation learning: These nesting-doll models are trained to concentrate the most salient semantic signal into the leading vector dimensions, allowing trailing dimensions to be truncated directly. This preserves most of the ranking quality while slashing index storage and accelerating nearest-neighbor search.
• In-domain evaluation: Generic leaderboards cannot predict performance on your domain data; testing against internal corpora is non-negotiable, as domain-specific rankings shuffle dramatically. Common failure points include multilingual support and maximum context length.

When off-the-shelf models still fall short on your data, fine-tuning the embedding model is a frequently overlooked alternative: it is drastically simpler than fine-tuning an LLM and often delivers far greater retrieval gains. Radu built VespaEmbed, an open-source, no-code fine-tuning tool under the Apache 2.0 license: select a base model from Hugging Face, upload training text pairs where each pair contains an anchor query and its relevant passage (positive)—such as the interface example where the anchor is "What is machine learning?" and the positive is a passage explaining machine learning—pick a loss function, and initiate training. You can also try it directly on Hugging Face.

Radu's complete session on selecting embedding models is hosted on Maven, and his presentation slides detailing model selection and fine-tuning are publicly available.

Choosing OCR: Most Teams Should Use APIs, and Models Must Be Picked Around PDF Pitfalls

To enable language models to digest PDFs and scanned documents, the foundational step is OCR—extracting textual content and document geometry from page images. Joe Barrow introduced a decision matrix based on two scoping questions: Do you need plain text blocks, or full document structure (headers, tables, cell bounding boxes)? Should you call managed APIs or self-host your own models?

Intersecting these two dimensions yields four quadrants:

• Hyperscaler cloud APIs (AWS Textract, Google Cloud Vision): Output text blocks only; cost-effective ($0.60–1.50 per 1,000 pages); provide word-level bounding boxes, useful when product features require highlighting citations on source pages.
• Document-processing startup APIs (Reducto, Datalab, Extend): Output rich document structure; feature-complete, but premium-priced ($5–20 per 1,000 pages).
• Open-source pipelines (Tesseract, PaddleOCR): Output plain text blocks; fast and economical, but constrained in capability.
• Open-source vision-language models (LightOnOCR, Chandra, Docling): Output rich structural elements; require GPU infrastructure; feature-complete for core layouts.

Joe's guidance is straightforward: only about 5% of engineering teams should self-host. If your core mission is product development, managed APIs are almost universally the superior choice.

Even when using managed APIs, model selection remains critical because PDFs break extractors in countless unexpected ways. Joe highlighted several notorious failure modes:

• TeX-compiled PDFs contain no whitespace characters: TeX (the typesetting system favored in academic publishing) positions individual glyphs at explicit geometric coordinates without encoding whitespace characters; naive text extractors end up returning an unbroken, space-free string of fused characters.
• Two-column layouts lack explicit reading order: Line-by-line extraction tools read horizontally across both columns, feeding a scrambled, interleaved block of text to the LLM. Major cloud services like Textract are susceptible to this failure.
• Line-by-line dumps strip semantic structure: Headings, tables, and cell boundaries vanish, depriving downstream retrieval pipelines of necessary structural context.
• Degraded scan quality induces hallucinated text from cloud APIs: Textract has been observed returning hundreds of repeated "the" tokens on noisy, degraded scans.
• Blank pages trigger hallucinated boilerplate in vision-language models: Feeding an empty page into VLMs can result in hallucinated headers or running titles.
• VLM-generated Markdown lacks word-level bounding boxes: If your product needs to highlight exact citation text within the original PDF viewer, standalone Markdown output cannot support it because word coordinates are absent.
• Scrutinize model licensing terms: Chandra and Surya are free only for organizations with annual revenue under $2 million that do not compete with Datalab; enterprises exceeding this threshold must secure commercial licenses. GLM-OCR itself is MIT-licensed, but bundles an Apache-licensed Paddle layout model, requiring compliance with both licenses.

To appreciate how pathological PDFs can become, browse WTF PDF, a repository of edge-case documents that break the vast majority of extractors. Joe's evaluation recipe is concise: answer the two scoping questions; curate 50–100 representative pages; execute each candidate model across the identical sample set; and inspect raw extraction outputs side-by-side to observe how each fails before deciding. Artifacts like missing whitespace in TeX or column interleaving will never show up in an aggregated aggregate benchmark score.

For the full session, watch the recording of Joe's talk on OCR selection; his research write-up evaluating open-source OCR models is equally worth reading.

Subtext: Footnotes for Agents, Preserving Critical Decisions During Rewrites

When prompting an Agent to summarize, polish, or rewrite long-form documents, it condenses clauses, prunes text, and paraphrases phrasing. Unstated premises and qualifications behind original statements—such as "which specific round does 'the next round' refer to" or "what exact stipulations govern this decision"—are lost during rewriting if they were not explicitly encoded into the text.

Bryan Bischof and Adam Conway are AI engineers at Theory Ventures. Investors at the venture firm write detailed investment memos outlining critical decisions, which risked being omitted whenever an Agent rewritten them. Their solution is Subtext: a lightweight format attaching metadata to individual sentences. The source sentence remains untouched, while author clarifications and unresolved inquiries are stored as metadata in self-describing JSON (contextdoc/columnar-v1). Functioning as footnotes written specifically for Agents, it anchors rewritten content to source nuances.

To take an illustrative example (not actual Theory Ventures content): a memo might include the sentence "We recommend pausing follow-on investment in ACME." The metadata attached to this sentence could specify: "'ACME' refers to the logistics software startup from our Series A portfolio, not the publicly traded company of the same name"; "'Pausing' denotes waiting for next quarter's renewal metrics before deciding, not abandoning the position." When an Agent rewrites the memo, reading this metadata ensures critical qualifications are never omitted.

The repository provides two skills:

• subtext-annotate: Ingests raw source text (emails, notes, Markdown) into a subtext document and poses clarifying questions from the perspective of an uninformed reader (probing unexplained acronyms, ambiguous referents, vague dates, or missing background), binding the author's clarifications directly to relevant sentences.
• subtext-read: Parses .subtext.json files, interpolating explanations inline at points of potential ambiguity and cataloging unresolved questions that the author left unanswered.

Human readers benefit as well. Adam demonstrated a chat UI prototype where tickets and call recordings associated with the sender appear directly alongside each message, sparing the reader from hunting down context across disparate tools.

Bryan and Adam's full session on Subtext is hosted on Maven, with their presentation slides and code repository publicly accessible.

Systems: Latency, Economics, and Runtime Environments

Once evaluation and retrieval are dialed in, attention turns to systems engineering challenges: resolving slow generation times, calculating memory footprints and compute costs when hosting open-source models, and containing blast radiuses when Agents execute arbitrary code.

Inference Latency: Diagnose Whether the Bottleneck Is Reading Input or Generating Output

When response generation is slow, the first step is determining whether time is spent ingesting input or emitting output; the architectural solutions for the two are entirely different.

Abi Aryan's talk decomposed an inference pass into two distinct phases:

• Prefill: The model processes the complete prompt input. By analogy, this is like reading an exam prompt before answering: the entire sequence is ingested concurrently across GPU cores in parallel, yielding high token-per-second throughput.
• Decode: The model generates output tokens autoregressively, one by one. Like handwriting an answer one word at a time, each new token must be appended to the sequence before calculating the next, preventing parallel generation and making per-token decoding an order of magnitude slower than prefill. Abi emphasized that decoding throughput is primarily memory-bandwidth-bound rather than compute-bound. For every token generated, the GPU must stream the entire model's weights and the accumulated KV cache out of VRAM, even though the arithmetic operations per weight are minuscule; consequently, memory transfer speed forms the bottleneck, leaving most of the GPU's compute capacity idle.

Her Colab notebook evaluates the same model across three distinct request shapes, measured using TinyLlama-1.1B-Chat-v1.0 (a small ~1.1B parameter model) on a single Tesla T4 GPU (~15.6 GB VRAM). While swapping in larger models or faster hardware changes absolute latency figures, the relative distribution between phases remains consistent. Below are the figures from an individual benchmark run in the notebook:

Long-input, short-output (RAG, classification, information extraction): The most favorable request profile. Despite the lengthy prompt, prefill parallelizes efficiently across cores. In empirical testing with 1,205 input tokens and 25 output tokens, total latency was 1,087.5 ms: prefill took 386.4 ms (~3,119 tokens/sec), while decode took 701.1 ms (~36 tokens/sec).

Short-input, long-output (Agent execution traces, code generation, long-form composition): The most problematic profile; latency is dominated almost entirely by sequential token decoding. In testing with 37 input tokens and 256 output tokens, total latency reached 7,952.4 ms—of which decode accounted for 7,918.8 ms (~32 tokens/sec).

Short-input, short-output (single conversational turns, concise Q&A): In testing with 25 input tokens and 32 output tokens, total latency was 944 ms, with decode taking 903.5 ms. Furthermore, 25 tokens is too small a batch to saturate parallel GPU tensor cores, leaving hardware compute units largely underutilized during prefill. Inference engines like vLLM address this via continuous batching, packing small requests from multiple concurrent users into shared execution steps to keep GPUs fully occupied.

Abi's recommendation: If your bottleneck is decoding, shorten the generated response first; only then explore speculative decoding (using a draft model to speculate upcoming tokens for batch verification by the target model), downsizing the model, or upgrading hardware. If the bottleneck is prefill, batch concurrent requests and truncate context windows.

This notebook is well suited for hands-on experimentation, with three practical benchmarks worth running:

• Batch size: Run identical request profiles under batch sizes of 4 and 8 to observe per-request throughput scaling and VRAM ceiling limits.
• Inference runtime: Replace the Hugging Face pipeline with llama.cpp or vLLM, which produces the most dramatic impact on decode throughput.
• Quantization: Benchmark INT8 or INT4 model variants to measure reduced memory footprint and corresponding increases in token throughput.

Abi's full presentation on inference latency is available to watch on YouTube.

The Economics of Open-Source Models: A VRAM Estimation Rule of Thumb and Deployment Cautions

Zach Mueller walked through several prominent open-source model families, sharing his hands-on impressions from daily operational workflows:

• Qwen: The 27B dense model is his top recommendation for day-to-day assistant-style Agents handling personal productivity tasks (checking emails, browsing websites, invoking APIs), distinct from code generation. His own background Agent has operated on this model for months.
• GLM 5.2: His primary workhorse for coding; reliable and steady, rarely distracted, and rarely makes small mistakes.
• DeepSeek V4 Flash: A viable drop-in replacement for Claude Haiku.
• Kimi: His favorite model for long-form writing, though serving it at full precision demands an 8-GPU B200 node, making self-hosting impractical for most teams.

Before committing to a model, verify whether it fits inside your available GPU memory. Parameter count reflects the total number of weights stored in the network; 27B denotes 27 billion parameters. A practical rule of thumb for estimating weight VRAM is:

Weight footprint (GB) ≈ Parameter count (in billions, B) × N × 0.5

The rationale behind this formula: at 16-bit precision, each parameter occupies 2 bytes, meaning 1 billion parameters require approximately 2 GB. The term N × 0.5 corresponds directly to the byte footprint per parameter across varying precisions:

• 4-bit precision: N = 1 (equivalent to 0.5 bytes per parameter)
• 8-bit precision: N = 2 (equivalent to 1 byte per parameter)
• 16-bit precision: N = 4 (equivalent to 2 bytes per parameter)
• 32-bit precision: N = 8 (equivalent to 4 bytes per parameter)

VRAM capacity cannot merely accommodate static weights. During autoregressive generation, intermediate key-value projections for every historical token must reside in VRAM to avoid recomputing the entire past sequence with each new token; this KV cache grows linearly with context length, and intermediate activations consume additional memory during forward passes. For example, a 27B model quantized to 4-bit requires roughly 27 × 1 × 0.5 = 13.5 GB for weights; reserving headroom for KV cache and activations allows it to fit onto a single 16 GB GPU. For exact formulas, refer to EleutherAI's guide to transformer math.

Zach also noted several practical deployment caveats:

• Avoid routing proprietary codebase context through third-party proxy aggregators like OpenRouter. Proxy routers multiplex requests across arbitrary upstream hosting providers whose data privacy policies and retention guarantees vary widely.
• Beware of routers that switch upstream providers mid-conversation. Many providers cache long shared prompt prefixes (such as extensive system instructions), making subsequent requests faster and cheaper; switching providers mid-stream invalidates that prompt cache. If you employ a router, benchmark its actual impact on latency and cost systematically rather than relying on intuition.
• When leasing GPU compute, he prefers renting a single high-spec box over multi-node H100 clusters. H100 nodes may only be 10%–20% cheaper, but distributed multi-node setups require models to shuttle tensors over network interconnects, creating a severe communications bottleneck.

Zach's full talk on the economics of open-source models can be viewed on YouTube.

Sandboxing Agents: Isolation, Sub-Second Startup, and Decoupling Control from Execution

Adam Azzam explored the architectural rationale for equipping Agents with sandboxes. Running a local coding Agent on your personal workstation rarely exposes the urgent need for isolation; however, as generation speeds accelerate, multiple Agents execute concurrently, or multi-tenant collaboration begins, sandboxing becomes essential. A sandbox is an ephemeral, isolated execution environment that can be torn down immediately after use; any catastrophic failure in Agent code remains strictly quarantined within that boundary.

Many teams initially attempt to repurpose existing CI/CD pipelines (which run automated build and test suites upon code commits), but CI/CD is ill-suited: pipelines are slow to boot and have brief lifecycle timeouts, whereas an Agent may labor on a complex task for hours or days while dynamically mutating dependencies within its environment. Furthermore, Agent-generated code runs unreviewed; a destructive command could crash the host machine or exfiltrate environment credentials.

As Agent concurrency grows, container startup latency becomes a crippling bottleneck. Ramp overcomes this by rebuilding base VM images (pre-configured snapshots containing the operating system and toolchains) every 30 minutes. When an Agent launches, it mounts a pre-warmed instance and executes a quick git pull for recent diffs, reaching operational readiness in under one second.

Another core architectural tenet is decoupling the Agent from its execution tools, ensuring that tool failures cannot crash the Agent process itself. If a tool runs out of memory while loading an oversized CSV file, that failure must not terminate an Agent trajectory that has been executing for hours. In Adam's architecture diagram, the Agent operates on the control plane (managing reasoning and orchestration) while tools reside on the data plane (handling physical execution), bridged across an isolated network boundary.

Adam currently works at Modal. Workloads on Modal execute remotely with local-like responsiveness. A minimal "hello world" example initializing a sandbox on Modal is shown below; see the Modal Sandboxes Guide for detailed documentation:

import modal

app = modal.App.lookup("sandbox-hello-world", create_if_missing=True)
sb = modal.Sandbox.create(app=app)
process = sb.exec("echo", "hello")
print(process.stdout.read())
sb.terminate()

Adam's full talk on Agent sandboxes is available on Maven.

Post-Training: The Final Step—Audit Benchmarks and Runtime Environments First

When evaluation scores stall, the instinct of many practitioners is to hoard data and train models. Post-training refers to updating model parameters directly on top of pre-trained foundations using domain data—typically via supervised fine-tuning (SFT) or reinforcement learning (RL).

Will Brown and Florian Brand from Prime Intellect caution that fine-tuning should be treated as a last resort when trying to raise evaluation scores, because benchmark gains can frequently be achieved without modifying model weights at all. They recommend executing these three steps in sequence first:

1. Read execution traces to understand precisely where and why failures occur.
2. Fix the evaluation harness and runtime execution environment.
3. Improve retrieval, context formatting, and agentic harnesses.

The second recommendation may sound surprising, but subtle artifacts in evaluation setups can drastically skew scores. To prevent models from looping indefinitely, benchmarks often enforce aggressive execution timeouts; in those setups, faster inference engines or beefier hardware allow the model to complete more reasoning steps before timing out, artificially inflating benchmark scores. As a concrete example, they demonstrated scaling task timeouts by 5× on Terminal-Bench 2:

• GPT-5.2 high: 52.8% → 60.67%, an increase of 7.87 percentage points.
• GPT-5.2 xhigh: 46.3% → 60.97%, an increase of 14.67 percentage points.

The underlying models remained identical; only a single parameter in the benchmark harness was adjusted. Under the original timeout, high outperformed xhigh by 6.5 percentage points; after relaxing timeouts, xhigh edged ahead by 0.3 percentage points, completely reversing the ranking. Relying solely on the initial scores would have yielded an entirely flawed conclusion.

They cataloged several prevalent pitfalls when configuring evaluation environments:

• The model is trained sampling at temperature 1, but forced to temperature 0 during evaluation. Temperature governs generation entropy: temperature 0 deterministically greedily selects the top-ranked token at each step, while temperature 1 samples proportional to the predicted probability distribution. Imposing temperature 0 can degrade reproducibility against reported benchmarks.
• Maximum token limits or turn budgets are clamped too tightly, prematurely truncating reasoning chains before the model finishes.
• Allocating insufficient CPU cores or RAM to Agent sandboxes.
• Bloated scaffolding harnesses: simplify to a minimalist harness allowing the model to execute bash commands directly rather than burdening it with bespoke tool wrappers.
• Before concluding that a model cannot accomplish a task, inject task-specific skills into its context window first.

Post-training is justified only when two preconditions are met simultaneously:

1. Ground truth can be automatically verified, such as evaluating correctness via unit tests or deterministic verification scripts. Reinforcement learning relies on exploring thousands of trajectory rollouts; each attempt must be scored automatically to supply the credit assignment signal steering policy updates.
2. The model's baseline benchmark score is strictly greater than 0 and less than 100. A score of 0 indicates the model never succeeds, leaving zero positive trajectories to reinforce; a score of 100 leaves zero headroom for improvement.

To explore practical implementations, leverage Prime Intellect's verifiers library to define tasks and reward functions in Python; their hosted reinforcement learning platform executes training runs from a concise configuration file, accompanied by pre-configured environments ready for immediate deployment.

Will and Florian's complete session on post-training is hosted on Maven.