My agents' search reads 25 times less to find one fact

On 28 questions, my search tool plus reading the cited lines took a median of about 3,500 tokens of text. Plain rg plus opening the top 3 files took about 84,800. That is about 25 times less. Search also read the answer file 18 times out of 28. The rg route read it 3 times.

That is a scripted test. When I gave the same choice to real AI agents, the gap shrank to about a quarter fewer tokens. Both are below, with how the search works, one part at a time.

Scripted routes, 28 questions: search reads about 25 times less and finds the answer file six times as often.

The problem: whole files for one fact

My agents work from a few thousand Markdown files: rules, procedures, specs and research notes. Most questions need one fact from one passage, such as “what effort does this worker run at?”

The default loop is to search by text, then open files. Each file the agent opens goes into its context in full, even if the answer is three lines. Context costs money and time, and a context full of near misses makes worse answers. So the goal is to find the right lines and read little else.

The tour of my agent workspace shows the search tool from above. Here are its parts, each with a tiny example.

Part 1: files become passages

The tool never returns whole files. It splits each file at every heading. A section longer than 1,800 characters is cut into windows that overlap by 200 characters, so a sentence on the edge is not lost. Each passage keeps its file path, heading and line range.

Each file also gets a fingerprint, called a hash: a short code computed from the file’s bytes that changes if any byte changes. The tool shows the first 12 characters of its SHA-256 hash. Later, it tells the agent whether the file changed since the search.

One file becomes small passages with exact line ranges, all sharing one file hash.

Part 2: ranking by words (BM25)

BM25 is a classic formula that scores a passage by the words it shares with the question. Two ideas drive it. A word that appears in fewer passages counts more (its inverse document frequency, or idf), because it says more about the passage. And repeating a word helps less each time. My tool uses the BM25 built into SQLite’s full-text index, which lets each field carry its own weight. A word in a heading or title counts twice as much as one in the body. The index also matches word forms, so “workers” matches “worker”.

A rare word in the question counts about three times more than a common one.

Part 3: ranking by meaning (embeddings)

Words fail when the question and the file use different words. An embedding model turns text into a list of numbers, so that texts with similar meaning get similar numbers. Search then compares the question’s list to each passage’s list by the angle between them, called cosine similarity, where 1 means the same direction.

Mine is EmbeddingGemma, a small Google model that gives 768 numbers per passage. It runs locally through Ollama, so nothing leaves my Mac.

The passage that shares no words with the question still wins on meaning.

Part 4: merging the two rankings

The two searches score on different scales, so the tool ignores raw scores and merges by position. This is called reciprocal rank fusion. Both lists rank files, by each file’s best passage. Each list gives a file 1 / (k + rank), and the scores add up. The paper used k = 60. My tool uses k = 20, which favours top places more.

Results near the top of both lists rise above a result that tops only one.

Merging is worth it. On my 180-question search test set, the right file was in the top five 62% of the time with both searches, against 49% for words alone and 48% for meaning alone. A reranker, a second model that re-reads the top results, could add more. An older test on a different question set found about 5 more points in the top five, at about 4 seconds a search. I have not built one.

Part 5: one search, end to end

A question becomes at most 8 passages with exact lines, then the agent reads only those lines.

The last step matters. Another agent may edit the file after the search, and new passages get their meaning numbers in small batches. So the agent opens the original, reads just the cited lines, and checks the hash. If it changed, the agent reads the file again before it trusts the lines.

The measurement

I took 28 questions from my search test set whose answers sit in the public parts of the system: rules, roles, model notes, connector docs and my evals notes. Each has a known answer file. A script ran each route once.

Search route. Run the search with its defaults (8 results), then read the cited lines of the answer file if it is in the top five, or of the first result, plus 10 lines either side. Two tool calls.
rg route. Take up to four keywords from the question by a fixed rule: the longest words of four or more letters, minus common words. Run rg, rank files by how many keywords they match, and open the top 3 whole. I capped the rg listing at 30,000 bytes, as agent tools cut long output, and skipped worker task folders. Four tool calls.

“Found” means the answer file was in the top five results for search, or among the files opened for rg. Tokens are bytes divided by 4. The raw data, script and method are in my records.

Search finds the answer six times as often in half the tool calls, but each question takes about 10 times longer.

Per question, the middle half of search runs read 11,100 to 16,200 bytes. The middle half of rg runs read 275,000 to 408,000. With 3 results instead of 8, search read about half as much and still found the answer 14 times.

The rg route misses because common words match long research notes. On a question about evals, for example, it opened two long source notes and a long summary file. Those rank above the short rule file that holds the answer. With worker task folders left in, it found nothing at all.

With real agents, the gap is smaller

A fixed keyword rule is a weak stand-in for an agent. So I gave 10 of the questions to Claude Haiku twice: once with only Grep, Read and file listing, and once with only my search tool and Read. I counted the tokens it reported, its tool calls and its time, and checked that its answer named the right file.

A real agent with search used about a quarter fewer tokens in about the same time, but named the exact answer file less often.

The search agent used a median of 23% fewer input tokens. But it named the exact answer file 6 times out of 10, against 8 for the Grep agent. Counting other files that also hold the answer, both got 9 of 10. The agent with Grep did far better than my fixed rule, because it picks its own words and follow-up searches. And most of each agent’s tokens are its own fixed instructions, which it re-reads on every turn. So the saving per question is real but modest.

What this does not show

Small samples. 28 scripted questions and 10 agent questions, each run once. Times were not repeated.
Mostly direct questions. 23 of 28 ask in the file’s own words. Only 5 use different words, where meaning search helps most.
Found means the file, not the fact. The script checks that the answer file was read, not that the cited lines held the answer.
The search tool has been tuned on some of these questions. So its found rate may be a little high.

When plain rg is still the right call

I still use rg every day. It wins when the text is exact, like a function name or an error message. It wins when you need every match, for example to rename something. And it is the only option for code, because my index covers Markdown only.

The rule my agents follow is short: search first, then open the original and check the lines before relying on them. For exact strings, use rg.

All writing