Six "small things" investment managers do every day

These six things are second nature to investment managers — judged in a second. But the moment you have to explain "how exactly did I decide that," people get stuck — which is precisely why they're hard to teach an AI. The team broke them out one by one into a benchmark.

The first three are classification tasks (evaluated on accuracy + F1 score), the last three are localization tasks (evaluated on exact-match accuracy). The original post notes that internally there are many more tasks like these, and the pattern holds throughout: frontier models generally underperform their own trained models on this kind of work.

Both touch on politics and finance — why is one relevant and the other not

Take the "financial article relevance" task as a real example. The two headlines below both brush against geopolitics and finance — yet for a macro investor, one is worth reading and the other should be swiped away without a second thought. Take a guess which one's relevant.

Tuning prompts every which way, AI still caps out at 78.2%

The team first took the cheapest route: rescuing the results through prompt engineering. Experts rewrote the instructions to reflect real task descriptions, and even redefined the tasks — for instance, splitting article classification from "relevant / not relevant" into three categories: "relevant and interesting," "relevant but uninteresting," and "not relevant," since a minor IPO news item is technically financially relevant but lacks the macro significance a macro investor cares about.

Prompting pulled accuracy from coin-flip level up to the 70s, but hit a ceiling beyond that — automated prompt optimization couldn't squeeze out any more headroom. The original post also notes: "more expensive doesn't mean more accurate." GPT 5.4 costs 43% more than 5.2, yet only shows a small accuracy gain — meaning newer models aren't making much progress on this kind of task, especially per dollar spent.

The labeled data itself was wrong: how to route only truly hard samples to experts

Once prompting hit its ceiling, the team turned to fine-tuning. But the first obstacle wasn't training — it was data: these judgments only have value if they've passed through an investment expert's eyes. The labels the team initially bought from a vendor were non-expert, and training on them still produced a poor model. Only when they looked through the model's reasoning traces did they discover the labels in the dataset were often simply wrong.

Having experts relabel everything was too expensive. So the team came up with a clever workaround: let the model itself flag "suspicious" samples, and only send those to experts. The logic is straightforward — if a sample doesn't even match its own training-set label, either the question is genuinely hard, or the original label is wrong. Either way, it's worth an expert's eyes.

This way, expert labor gets spent only on genuinely disputed samples, while the rest stays as-is — cleaning the data while keeping costs in check. The final evaluation, meanwhile, runs on a completely independent held-out set that never went through the cleaning process, avoiding any self-grading bias.

Three combined moves push accuracy from 73% to 84.7%

With clean data, training began. The team picked Qwen3-235B — the most thoroughly studied open-source model in academia — as the base, and ran all training on Tinker, with no need to manage GPU infrastructure themselves. Step one, standard GRPO reinforcement learning fine-tuning, laid the foundation: accuracy jumped from the base model's 44.8% to 73.48% — still short of the 80% threshold. What actually pushed it over the line were three improvements stacked on top.

The staircase shows three real data points from the original post: base 44.8% → GRPO 73.48% → full recipe 84.66%. The 11-point jump in the middle comes from the combination of the three improvements below.

What CISPO and "online policy distillation" actually do

Of the three moves above, CISPO and online policy distillation have the most intimidating names. You don't need to understand reinforcement learning — what they're solving is really the same basic problem: don't let the model overshoot or drift, and give it a solid teacher to learn from.

Online policy distillation + dynamic teacher promotion

The student model practices on its own while also referencing the answer distribution of a "teacher model" — drifting too far from the teacher gets penalized (the original post uses a penalty term: the bigger the gap between student and teacher, the more the reward is docked). The key is that the teacher isn't static: every 20 steps, the system checks accuracy on the validation set, and the moment the student sets a new high, the student itself is promoted to be the new teacher — never regressing by keeping a weaker model in the teaching role.

Error rate down nearly 30%, inference cost cut to 1/14

With all the improvements stacked together, the custom model raised average accuracy from the best frontier model's 78.2% to 84.7% — a level the team considers good enough for everyday use. The bigger savings are in cost: with a far smaller model footprint, per-task inference cost drops to just 1/13.8 that of the corresponding frontier model.

The original post says this conclusion holds well beyond the 6 tasks made public here — the pattern is consistent across a large number of similar internal tasks. The methodology also isn't tied to finance: a workflow of "route hard samples via model disagreement, have experts clean the data, then reinforcement-learning fine-tune" is, in the authors' view, applicable to other organizations' own specific judgment tasks. They call this "differentiated intelligence": custom models tuned to an organization's specific needs outperforming general frontier models on their own work.

The team first took the cheapest route: tuning prompts (instructions written for the AI) on general-purpose large models as hard as they could. But no matter how much they optimized, the best still capped out at 78.2% — short of the 80% credibility threshold investors require. So they switched tactics: use expert-labeled data to fine-tune (retrain the model on task-specific data) a custom small model.

How does a smaller open-source model beat the strongest large model (the "frontier model," i.e. the most advanced general-purpose model)? Not by being smarter — but through two moves: clean up the data first, then get the training recipe right.

Put those two moves together as a pipeline and it clicks. Follow along with a scenario: XiaoHu has a thousand news articles, cheaply outsourced labeling has plenty of mislabels mixed in, and training directly on that just teaches the model to learn the mistakes.

The official figure is "cost down 13.8×." In plain terms: to triage the same batch of documents, a frontier model's inference bill runs about fourteen times what this custom model costs.