Why "move everything to one place" never works

The hard part of finding things inside a company was never the search itself — it's that the material is inherently scattered. Every quarter, someone proposes the same "brilliant plan": record everything on one platform from now on, and make it the single source of truth. In practice, that almost never works.

The reason is that information gets created wherever it's most convenient: a suggested edit in a doc, a thread in Slack, a code reference on GitHub, a status field in Jira. Each of these platforms has spent years being refined for its own use case. Discussing a pull request in Google Docs would be a miserable experience.

So Cerebras's first decision was to design a system that asks almost no one to change their existing habits. On the ingestion side, that means going directly to each platform and pulling the data out.

The core of the whole thing is a single table

The knowledge base offers three things externally: a place to store internal data, a place to query it, and a layer handling authentication, access control, auditing, and usage stats.

At the core is one Postgres table holding vectors, raw summaries, and metadata side by side, regardless of source. The system keeps pouring in data from across the company to stay continuously queryable. The data interface is deliberately simple: whether it comes from a Slack thread or a netlist file used in chip design, it lands in the same vector table — and once it's in that table, it's queryable through the same interface.

Every data source only needs to define three things: what the data is, how to connect to it, and how often to pull it. Every row it produces has exactly the same field structure.

This design also leaves a door open for other teams. Some teams already run their own databases and don't want to move that data into Slack or a docs system just to get it into the knowledge base. The workaround is to treat that kind of source as a plugin script: the team submits a pull request with a small Python module that reads from its own system, emits rows matching the table's schema, and registers a data-source record. As long as it writes into the same store using the same fields, nothing else in the pipeline needs to change.

The hardest part of Slack is telling which line actually matters

Slack is the data source they care about most and find hardest to handle — it's where the company's freshest engineering discussions happen. They initially tried the cheapest approach: embed the raw text directly and search it. It quickly became clear that vectors alone couldn't catch everything.

The problem comes down to three traits of chat logs. First, information density varies wildly — "okay thanks" and a full paragraph explaining a kernel implementation both look like a single message to the system. Second, messages vary a lot in length, and short messages often beat out longer, more detailed ones on vector similarity. Third, a message's meaning often depends on the conversation around it.

Their fix is to have the same content covered by several retrieval methods at once, each patching a different weak spot. Same query, same four candidates — switch the scoring method and who gets boosted and who gets buried changes every time (figures taken from the original post's example).

Chat logs don't go straight into embeddings

Raw Slack text becomes keyword-searchable the moment it lands, because they maintain a full-text index on the original text in Postgres. But making embedding search actually useful takes one more processing step.

That step is called distillation: a model reads the entire thread and extracts four things — a one-line question an engineer would actually search for, a short summary, the eventual resolution, and the systems and code symbols mentioned along the way. Those four things get embedded; the verbatim chat log never goes into the vector store. In testing, normalizing threads into this fixed structure produced a clear accuracy gain, and the extra metadata gave semantic matching more to work with.

The message a long thread's summary leaves out

At this point Slack retrieval already worked, but they kept running into the same problem: an important message buried in a long thread doesn't always survive into that thread's summary.

The fix is called bursting. A burst is a run of consecutive messages from the same person. They embed each burst on its own, with the thread's overall topic prepended as context. That way, an answer buried in some tangential sub-thread — whose wording never made it into the thread-level summary — can still surface on its own.

To keep low-value content from flooding the database, each burst has to clear a weighted threshold before it gets embedded, judged on three criteria: it contains a term rare enough across the company's corpus, with a rarity cutoff of IDF ≥ 4.0; it's at least 200 characters once merged; or a message in it got an emoji reaction, counted as a coworker's stamp of approval.

Keeping a 40GB codebase fresh

Whether to embed the codebase at all was something they went back and forth on. With command-line tools like Claude Code around, building a vector index for code feels almost counterintuitive — grep seems like it should be enough. After talking to peers in the industry and reading Cursor's writeup on semantic search over large codebases, they decided to try it.

The real concern was something else: the company has many internal repos, some over 40GB each, and the question was how to keep an index current at an acceptable cost.

They ended up on CocoIndex, an open-source framework built specifically for codebase embedding. Chunking follows each language's syntactic boundaries, trying coarse-to-fine: start at a high-level boundary like a class; if the resulting chunk is still too big, fall back to method boundaries; if that's still too big, fall back further to smaller blocks of code. The same file ends up producing embeddings at several granularities — one at the file level, one per function, and so on.

The original post's chunking example. On the left, the highlighted regions in the code are the identified semantic chunks, with line-end annotations showing which level each was cut at: it tries the class first, descends into the three methods LoadManifest(), WarmShardCache(), and Restore(), and falls back further to the if/for level where a method is still too big; the private-fields block at the end forms its own chunk. The two boxes on the right show the two approaches from the comparison table above (01 NAIVE WINDOWS — fixed-length chunking, 02 LANGUAGE-AWARE — syntax-boundary chunking, with the latter selected in the diagram); in the legend at the bottom, SEMANTIC NODE refers to a semantic chunk and FIXED SPLIT to a fixed-length cut point. Source: Cerebras

Updating is where this approach really pays off. CocoIndex also stores its sync metadata in Postgres, so each commit only re-processes the chunks that actually changed, instead of recomputing the whole repo. Having sync state live in the same store as the vectors turned out to suit them especially well.

As more codebases got connected, onboarding a repo became a config file that teams submit themselves, with allow-lists and block-lists settable by file path.

What happens after a question comes in

Every incoming question first goes through a short planning pass: a model decides which tools and data sources the question likely needs, then dispatches the selected tools in parallel.

On the MCP side, tools get handed straight to the agent to orchestrate

In the web UI, the pipeline above runs end to end: planning picks the tools, execution gathers evidence in parallel and normalizes it into a unified format carrying scores, timestamps, and source hints, and a final model pass produces an answer with citations and caveats. From the user's side, it's just asking a question and getting an answer.

The MCP integration takes the opposite approach: it exposes retrieval as individual, directly callable tools rather than hiding everything behind one big "answer my question" interface. These tools are deliberately kept simple, with as little model involvement as possible, so a client can query them fast and cheap. Each MCP tool maps to one bare retrieval action, with narrow, structured, stable inputs and outputs that any client or agent can call directly. Most of them run a single retrieval path — semantic search, literal search, or a straight ripgrep call — apply a light scoring pass, and return the raw evidence rows.

That leaves orchestration to MCP-capable agents like Claude Code: which tools to call, in what order, and how to assemble the results into a final answer or a code change is all up to the agent. The retrieval layer keeps working regardless of what any of those model judgment calls decide.

Merging six rankings into one

A search result can land at the top purely because it happens to share words with the query, while actually answering a different question. Worse, the scores from the six retrievers aren't directly comparable — a vector's cosine similarity and a full-text relevance score are simply not the same ruler.

Their approach is to ignore the raw scores and look only at rank position. The algorithm is called reciprocal rank fusion, or RRF: every time a document appears on a ranked list, it gets a score bump of 1 divided by (60 plus its rank on that list), with a default weight of 1.0; that 60 is a smoothing constant.

Walking through the top-ranked document makes the mechanism clear. CLASS: CheckpointLoader appears on five of the six lists, ranking 1st, 2nd, 3rd, 5th, and 5th:

Three more steps after merging

The fused ranking doesn't go straight to answer-writing. First, duplicate chunks from the same source that got split apart are merged back together; then a cap limits how many results a single file can contribute, producing a more diverse top twenty. Those twenty, along with the original question, go to a small reranking model that scores each one 0 to 10 and keeps the top ten.

Once the ranking is settled, the surrounding context gets restored. If a hit lands in some paragraph of a wiki page, the paragraphs immediately before and after get pulled in too, so a heading, a precondition, or a caveat that got separated during chunking doesn't go missing. What reaches the reader is a complete citation, not an isolated fragment missing its premise.

By the end of this pipeline, what retrieval hands off is a package of evidence: pulled from different retrievers, fused, deduplicated at the source level, reranked against the actual question, and finally padded out with surrounding context.

Searching the whole company makes results worse

As the corpus kept growing, "search everything" stopped being useful pretty fast. A compiler engineer doesn't want data-center runbooks cluttering up their results — and vice versa.

Their fix was to add a layer called project. A project is a named bundle of data sources — a set of Slack channels, code repos, internal databases, and doc spaces organized around a team or a project. Projects are deliberately lightweight: the same data source, like a company-wide incident channel or a central platform repo, can be referenced by multiple projects at once without duplicating it.

New hires get guided into picking or creating a default project closest to their actual work — training infrastructure, compilers, data center ops. That default lives in the user's profile, and every query after that is automatically scoped to it. The upshot: new engineers don't have to spend time figuring out which Slack channels, repos, or doc spaces are relevant to them — the results they get from day one are already relevant.

This knowledge base works because it meets people wherever the information already lives, instead of forcing everything into one rigid system. Cerebras, "How we built our knowledge base"

Every concrete figure mentioned in the post

Cerebras's first decision was a reversal: instead of asking anyone to change how they write things, write a data-pulling program for every platform, normalize whatever comes back into one format, and drop all of it into the same table. Three months after launch, it's fielding more than 15,000 questions a day, asked by employees as well as automated scripts and agents.

The payoff of one shared table is that adding a new data source requires zero changes on the query side. But a new headache shows up immediately: Slack is the hardest source to crack — "okay thanks" and a full paragraph explaining a kernel implementation both look like a single message to the system. How do you tell which one actually matters?

Their answer is to trust no single ruler: run the same batch of candidates through four scoring methods at once, each covering the others' blind spots. The diagram below walks through a real question — XiaoHu posts one question to the channel, "restore hangs after manifest load," the database holds four candidates, and we watch what each of the four rulers boosts and what it buries.

"Improved retrieval accuracy" doesn't mean anything you can feel. Translate it into one concrete scenario instead: a new hire inherits the training cluster, restore gets stuck, and they need to find the answer — "set CKPT_PREFETCH=4" — using three different approaches.