Adding Live Web Search to a Local LLM Without Making the Terminal Wait

The nishad.ai terminal began as a deliberately closed knowledge system. Qwen could answer from the structured ontology, selected public pages, published articles, recent tab conversation, and its pretrained general knowledge. It could not discover information published after the model was trained or retrieve external evidence for questions outside the site corpus.

That limitation was useful for privacy and predictability, but it became visible in questions such as:

What changed recently in llama.cpp prompt caching?

Using public benchmarks, what might a comparable Twitter engineering role
in Seattle have paid around 2021?

The local model could explain what was documented and clearly label unknowns. It could not responsibly claim that it had searched the web.

The new design adds Exa Lite search through the Cloudflare Worker, but preserves the most important user-experience rule: the visitor should see the local answer first. Search happens in parallel, then a second Qwen call streams a clearly separated, cited web addendum.

This post explains the architecture, the security boundary, the routing changes, the browser fallback, and the measured latency from a ten-prompt benchmark.

Search is an addendum, not a loading screen.

Why not give Qwen unrestricted browsing?

The model should not choose arbitrary URLs and ask the Mac gateway to fetch them. That would create an SSRF surface capable of probing:

127.0.0.1
private LAN addresses
router administration pages
cloud metadata endpoints
file URLs
credential-bearing URLs

It would also mix model-generated instructions with networking authority.

Instead, Qwen produces one bounded natural-language search query. A narrow Worker endpoint calls one known provider:

https://api.exa.ai/search

The Worker never fetches the URLs returned by Exa. It validates and returns only bounded metadata and cached highlights.

A dedicated web-intent inference

The first implementation added web_search and web_query fields to the existing ontology source router. That router also received a long catalog of careers, geography, articles, pages, projects, and lore.

The result was unreliable. A question could explicitly say “search current public sources,” but the small Qwen router would notice a locally relevant article and set web_search:false.

The solution was not a keyword algorithm. The application first performs normal ontology source and passage routing. It then gives a compact excerpt of that selected authoritative evidence to a separate schema-constrained Qwen web-intent call:

{
  "web_search": true,
  "web_query": "recent llama.cpp prompt caching changes"
}

Its prompt contains no source catalog. It only decides whether the current question materially requires fresh external evidence and, if so, writes one self-contained query.

Ontology source selection remains a separate call:

{
  "needs_site_data": true,
  "source_ids": [
    "blog-inside-nishad-ai-ontology-kv-cache"
  ],
  "intent": "answer"
}

For a compensation question, the selected career passage might identify:

Software Engineer II at Microsoft
Bengaluru, India
May 2024–present

The intent model uses those facts to produce a category query such as:

Microsoft Software Engineer II Bengaluru India compensation

rather than searching Nishad's name and retrieving unrelated namesakes.

The source router can select up to three complementary sources. Broad career-fit questions expand the selected portfolio with career, skills, projects, site architecture, and generated descriptions of published technical writing. This lets the planner incorporate AI/ML infrastructure, LLM inference, WebGPU, KV caching, search augmentation, and other work that may not appear in a single employment entry.

This creates three initial inference phases:

1. Local source and passage routing
2. Web-intent classification using the distilled local evidence
3. Final answer generation

Web-enabled questions add a fourth model call later for web synthesis.

Progressive orchestration

After routing completes, the browser starts two branches:

local Qwen answer
Exa Lite search

They run concurrently.

The local answer uses only the ontology and approved nishad.ai evidence. Its system prompt says that a separate web phase may follow and that it must not claim web access.

When the local answer completes, the terminal shows:

nishad.ai🌐> Loading estimates/data from web...

If Exa has already returned, the message advances almost immediately. Otherwise it remains visible until the bounded search request finishes.

The second Qwen prompt contains:

• the original question;
• the local answer;
• the original grounded system prompt;
• up to 2,400 characters of sanitized Exa evidence;
• a policy that treats web excerpts as untrusted evidence;
• an instruction to add new external context rather than repeat the local answer.

The result streams into a new terminal line:

nishad.ai🌐> ...

Source links and retrieval time are rendered by the browser from validated Worker data. Qwen does not invent URLs.

The Worker search boundary

The Exa key is stored as a Cloudflare Worker secret:

EXA_API_KEY

It is not committed to Git, copied into the Mac gateway, or sent to the browser.

The Worker accepts:

POST /search

with:

{
  "query": "one natural-language public search query"
}

It calls Exa Lite with:

maximum 5 results
moderation enabled
cached highlights
no live page crawling
10-second timeout

The returned public contract is provider-neutral:

{
  "schema": "nishad.web-search.v1",
  "query": "...",
  "retrieved_at": "...",
  "results": [
    {
      "title": "...",
      "url": "https://...",
      "snippet": "...",
      "published_at": null
    }
  ]
}

The model proposes words. The Worker owns network authority.

Validation and quotas

The Worker rejects:

• unapproved browser origins;
• non-POST methods;
• malformed JSON;
• request bodies above 4 KiB;
• queries shorter than 3 or longer than 300 characters;
• additional unexpected request fields;
• non-HTTPS result URLs;
• credential-bearing URLs;
• localhost, local-domain, and IP-literal results;
• titles above 200 characters;
• snippets above 900 characters;
• malformed provider responses.

Best-effort per-isolate quotas begin at:

20 searches per visitor per day
500 searches globally per day

Provider failures are sanitized into stable errors such as:

search_not_configured
search_quota_exceeded
search_rate_limited
search_budget_exhausted
search_timeout
search_unavailable

The local answer remains visible when search or synthesis fails.

Search privacy

Exa receives the generated public search query. It does not receive:

• the complete ontology;
• browser cookies;
• recent conversation history;
• the gateway token;
• the Exa key in browser-visible form;
• Spotify tokens or identifiers;
• private analytics data.

The terminal displays the exact derived query:

Public search query sent through Cloudflare to Exa: ...

This makes external data flow visible rather than silently changing the privacy boundary.

SmolLM browser fallback

Search is independent of the Mac tunnel. If the Mac is unavailable but WebGPU works:

SmolLM source routing
→ SmolLM web-intent call using selected evidence
→ browser-local answer
→ Exa search through Cloudflare
→ SmolLM web addendum

The fallback uses a much smaller evidence budget:

900 web-evidence characters
256 web-addendum tokens
2,048-token model context

SmolLM should be treated as an extractive summarizer, not as equivalent to Qwen. If WebGPU is unavailable, the application can still evolve toward showing raw validated search cards without model synthesis.

Exa Lite cost

Exa Lite currently costs:

$4 per 1,000 searches

The account includes $10 in monthly credits, enough for roughly 2,500 Lite searches.

At a maximum of 100 terminal requests per day:

3,000 requests/month

If one quarter require web search:

750 searches/month

That remains inside the monthly credit.

Even if every request searched:

3,000 searches
≈ $12 gross
≈ $2 after the monthly credit

Cloudflare Worker usage remains negligible at this scale. Outbound Exa calls are Worker subrequests and are not separately billed as Worker requests.

Benchmark design

The committed harness is:

GATEWAY_TOKEN='...' npm run benchmark:web-enrichment

It tests ten prompts ranging from 72 to 508 characters.

Each scenario measures:

1. local ontology source routing;
2. web-intent Qwen latency using distilled evidence;
3. local Qwen answer latency;
4. live Exa Lite search latency;
5. second Qwen web-synthesis latency;
6. total end-to-end time.

The local answer and Exa search run in parallel. Therefore:

parallel phase duration
≈ maximum(local Qwen duration, Exa search duration)

not their sum.

Ten-prompt results

Aggregate latency:

The internet was not the slow part.

What the measurements reveal

Exa search completed in under 3.1 seconds for every scenario. Because it ran alongside the local answer, it was fully hidden by Qwen generation in all ten cases.

The dominant costs were:

• web-intent inference;
• one or two ontology routing passes;
• local Qwen generation;
• the second Qwen synthesis.

The second synthesis had a 16.58-second median and became the largest single phase for many prompts. The longest request needed 36.20 seconds for synthesis and 80.31 seconds end to end.

Prompt length was not the only factor. Some short questions selected long local articles and triggered passage routing. Some longer questions needed no large local source. Cache state, output length, and selected evidence also affected latency.

User-experience conclusion

Progressive rendering still helps:

local answer first
→ web status
→ external addendum

The visitor receives useful local output after approximately 3–24 seconds instead of waiting 24–80 seconds for the entire enriched result.

However, the benchmark shows that web enrichment should remain selective. Running four model phases for every question would degrade throughput and user experience.

Next optimizations

Reduce local source routing for web-first questions

If a question is purely about current external data, the application may not need long local article passage selection. It can still include the compact ontology but skip unnecessary local retrieval.

Three cache-specialized prompt families

Production now uses:

slot 0 → ontology source and passage routing
slot 1 → local answers, web synthesis and reports
slot 2 → web-intent and search-query generation

The web-intent system message is stable. Distilled ontology context and the current question are placed in its final user message, improving prefix reuse. The three-slot server uses a 12,288-token total context pool and measured about 4.32 GiB RSS in the production process after all slots were exercised.

Shorter enrichment output

The synthesis prompt currently permits up to 256 tokens. Many web addenda can be limited to:

80–150 tokens

reducing total latency.

Search-result caching

Identical public searches can be cached for one to six hours. This saves Exa credits, although it will not remove the larger Qwen synthesis cost.

Evidence-first rendering

For simple queries, the browser could show validated search-result cards immediately while Qwen prepares the prose addendum.

Dedicated small intent model

The separate Qwen intent call is reliable but takes several seconds. A smaller browser or edge classifier could make the decision faster, provided it is benchmarked against the current schema-constrained Qwen result.

The broader lesson

Adding web search was not mainly a search-provider problem. Exa Lite was fast, inexpensive, and easy to bound. The harder work was preserving:

• local-first responsiveness;
• explicit provenance;
• secret isolation;
• source validation;
• failure-safe behavior;
• small-model routing reliability;
• bounded context;
• useful output when the web phase fails.

The architecture now treats the web as another evidence source—not as an authority that can execute instructions or silently overwrite the ontology.

That separation is what lets a static personal site, a local Mac-hosted Qwen model, a browser fallback, and a public search API cooperate without becoming one opaque agent with unrestricted network access.