Any organization that wants reliable latency, data processing guarantees (it's a big leak vector for company secrets), etc., will use such a search provider over the solutions you outlined.

Especially for chat-like use-cases (in contrast to coding), latency is a lot more critical, especially if you kick off a small single query question. On antother axis, scrape-reliabilty is important for a deep research job that will fire ~10+ web searches and ~100+ web fetches.

e.g. in our chat platform[0], we offer first party MCP servers for web search[1] and web access[2] which work decently well, but some of our customers that prioritize quality over cost, they go with dedicated providers like LinkUp[3]

[0]: https://github.com/EratoLab/erato

[1]: https://github.com/EratoLab/web-search-mcp

[2]: https://github.com/EratoLab/web-access-mcp

[3]: https://www.linkup.so/

I have a searxng instances but stopped using it after doing a comparison vs actual search provider APIs.

searxng gets rate limited fast and the results are inconsistent depending on what providers it can reach at any given time / isn't captcha'd out.

Meanwhile there are multiple providers with generous free tiers so no point in feeding LLMs inferior data

Also did a couple of side by side tests asking LLMs directly which feed results it prefer for some sample technical searches. Exa and tavily #1, brave #2, searx last

I have built my own search engine, which is really great btw. I self-host it.

I’ve combined multiple paid search providers and some that don’t have APIs. These API-less providers I’ve written wrappers for that scrape the normal html pages and return objects from the results.

These wrapped search providers very often block the engine from getting results. Either through rate limits or by blocking my server’s IP.

I am the only user and still get blocked. Let alone if multiple people would hit let’s say DuckDuckGo from the same office IP all day or in a long research session that requires many look ups.

An obvious solution is using a proxy with automatic rotation or some sort of captcha breaking system, which is fine for an agent, but in a search engine scenario might up costing too much in latency.

I am currently using Perplexity for https://veruscite.com/. Verify citations actually exist -- so get in a list of 100, half I can check with crossref, but the other half I need to do a web search and fetch to make sure they exist.

So I need higher rate limits to not make the tool come to a crawl. I probably will not consider DeepSeek API here (I have tried DeepSeek hosted at US providers, like Perplexity). The other search/fetch tools I try do not have great web coverage of Asian journals, but the fact DeepSeek API is not zero data retention I cannot get over.

The only reason this exists today is because normal search providers haven’t been completely disintermediated.

As soon as 100% of traffic to Google searches is not human eyes, they can’t fund it by ad rev, and you will need a service like this

For OpenWebUI and also OMP with local models. Searxng is, uh, playing fast and loose[1]. More engines are implementing anubis or other challenges, which searxng attempts to implement/bypass, but they are also making these services inactive by default to limit load[2].

Once this cat-and-mouse game goes on a bit longer I expect the quality of free searxng results to crater. My advice to anyone who has any budget at all is to skip over this whole issue and spend a few bucks on a proper search API.

[1] https://github.com/searx/searx/issues/1237 [2] https://github.com/searxng/searxng/pull/6696/changes#diff-e9...

Hosted search API from your LLM's provider is much easier to use, faster and less error prone. Also I believe that LLM may work better with its provider's search api, as it was definitely trained on it. I mean, it is just more convenient to use your provider's search.

And if we talking about LLMs, there is no particular point for search to be absolutely free if you still pay for the LLM. It would have more sense if search AND llm would be completely free.

How do you deal with fcking cloudflare blocking access to the websites ?

[flagged]