There are really only two usable search engines actually indexing the entire Internet: Google and Bing. Yandex also does but I’ve never seen it recommended for anything other than Russian language content (the company itself seems to be falling down a mineshaft at the moment). Baidu also does some although every Chinese exchange student I talked to about it (admittedly not many) advised only using it when Google is blocked. Every other engine is just wrapping Google or Bing (yes that includes Yahoo and DDG)
This is the kind of ugly truth of the search engine business. It’s a duopoly at least in part because the indicies are expensive to scrape, build, and run. You need to continuously run a large number of servers loading web pages and often running scripts. You need to be large enough to negotiate with content providers not to block you. Keep in mind paying them may bankrupt you as your margins will be thin. Google has a huge advantage here they own a good chunk of the online advertising industry and can afford to throw money around in a way a search only company wouldn’t be able to (this is why the European and Canadian link tax schemes ironically cement the existing monopolies). You need to continuously run large linear aglebra transforms on the results (PageRank is expensive). You need to store all your indicies on large expensive servers with a lot of memory as hitting disk may take too long. Results need to be fast and you will make next to nothing on each search.
DuckDuckGo’s results are a compilation of “over 400” sources according to itself, including Bing, Yahoo! Search BOSS, Wolfram Alpha, Yandex, and its own web crawler (the DuckDuckBot); but none from Google.[69][7][70][71][72] It also uses data from crowdsourced sites such as Wikipedia, to populate knowledge panel boxes to the right of the search results.[71][73] During a Bing API outage in 2024, DuckDuckGo stopped showing results, indicating that Bing provided a substantial portion of DuckDuckGo’s results.[74][75]
There are really only two usable search engines actually indexing the entire Internet: Google and Bing. Yandex also does but I’ve never seen it recommended for anything other than Russian language content (the company itself seems to be falling down a mineshaft at the moment). Baidu also does some although every Chinese exchange student I talked to about it (admittedly not many) advised only using it when Google is blocked. Every other engine is just wrapping Google or Bing (yes that includes Yahoo and DDG)
This is the kind of ugly truth of the search engine business. It’s a duopoly at least in part because the indicies are expensive to scrape, build, and run. You need to continuously run a large number of servers loading web pages and often running scripts. You need to be large enough to negotiate with content providers not to block you. Keep in mind paying them may bankrupt you as your margins will be thin. Google has a huge advantage here they own a good chunk of the online advertising industry and can afford to throw money around in a way a search only company wouldn’t be able to (this is why the European and Canadian link tax schemes ironically cement the existing monopolies). You need to continuously run large linear aglebra transforms on the results (PageRank is expensive). You need to store all your indicies on large expensive servers with a lot of memory as hitting disk may take too long. Results need to be fast and you will make next to nothing on each search.
Can you link any sources on this?
I think that’s my big hesitance to believe it, there just doesn’t seem evidence (besides other commenter mentioning an anecdotal tank man reference).
In the wikipedia article: