This blog post is part of a small series of blog posts on the topic of internet search engines.
Search engines are an integral part of many people’s everyday lives these days. Whether you want to find out about general world events, check the latest weather forecasts, search for the quickest route to a location or need instructions of any kind – most people turn to their trusted search engine.
For most people, this means using “Google”. The market share of this search engine has been well over 90% in many Western countries for decades. Google is so integrated into the everyday lives of many people that the verb “to google”, i.e., to search in the search engine, found its way into the German Duden dictionary in 2004 and into the Oxford English Dictionary at the end of the 1990s. In common usage, the term is often used even when another search engine is being used. [1, 2, 3]
However, in addition to Google, there is a whole range of other search engines, not all of which pursue similar goals or function completely differently from a technical perspective. We would like to give you a brief overview of the different variants and functions of modern internet search engines.
The Search Engine Index
Put simply, the most popular general search engines work by comparing terms in search queries from users with an internal database. This so-called search index can basically be understood as a gigantic database, which is continuously fed with website content and corresponding keywords by “web crawlers”. These “crawlers” are basically more or less complex programs that constantly search the Internet for new websites and content and add these to the search index of their search engine.
In a constantly growing Internet, web crawling is naturally an increasingly demanding task. On the one hand, the number of websites is constantly increasing, and on the other, existing websites are often becoming more and more complex, loading content dynamically, for example, or even actively trying to prevent web crawlers from reading their content. The operation of such programs therefore requires constant development, the implementation of new methods or constantly increasing computing power and Internet bandwidth. This is one of the reasons why building new search engine indexes is extremely difficult and time-consuming. Providers of small independent search engines in particular are running behind the big existing tech giants such as Google with its search engine of the same name or Microsoft’s “Bing”. [4, 5]
As a result, many search engines that appear to be “independent” at first glance actually use search indexes of larger search engines, such as Microsoft’s Bing, Yahoo! or search indexes of other well-known search engines. These third-party search engines pay for access to their own search index as a separate business line. The “independent” search engines in turn have to cover these costs through advertising revenue or other financing models. A cycle that ultimately strengthens the “big” search engines.
A side aspect that always stands out here is that almost all major search indexes are operated by tech giants in the United States of America. In Europe, on the other hand, there is no competition with a significant market share. Mojeek, a search engine from the UK with its own search index, is one of the few European offerings that does not use an external search index at all. The German search engine Ecosia recently partnered with the French search engine Qwant to jointly create another European search index (both currently use Bing’s search index, among others). However, it will still be a few years before this is usable. [4, 5, 6]
Meta Search Engines
Another type of search engine is the so-called “metasearch engine”. As the name might suggest, such search engines forward the user’s query to several other search engines and then present a list of results from several search indexes.
The best-known metasearch engines in Germany include MetaGer, which is operated by a non-profit association, and the open source software SearXNG, which can be operated by users themselves. Some of these search engines use official interfaces to other search engines for their queries and, depending on the agreement, also compensate them (such as MetaGer, which discontinued its free service last year and now only works on a paid model), or “scrape” the results of search queries to other search engines without explicit permission or compensation. [7, 8]
Metasearch engines are often able to display a broader spectrum of search results as they obtain their results from numerous sources. Although they too, like the supposedly “independent” search engines from the chapter above, are therefore dependent on other search engine offerings, they do not rely on a single source. Many metasearch engines also allow their users to set exactly which search indexes should be used and which should not.
Conclusion
The search engine offering is much broader than we often perceive. The market power of the tech giants does not necessarily go hand in hand with better quality results or a generally better user experience. Thinking outside the box and trying out smaller search engines can be quite interesting and may also show that there are usable and good options alongside the large commercial platforms.
Sources
- Doll, M. (2021): Suchmaschinen in Deutschland. URL: https://www.luna-park.de/ressourcen/seo-ratgeber/suchmaschinen-in-deutschland/
- Duden: “Googlen”. URL: https://www.duden.de/rechtschreibung/googeln
- Oxford English Dictionary: “Google” – verb. URL: https://www.oed.com/dictionary/google_v2?tab=factsheet#10568538
- Mojeek (2013): Growing the Index. URL: https://blog.mojeek.com/2013/09/growing-the-index.html
- Long, J. (2022): Mojeek – Now 6 Billion Pages. URL: https://blog.mojeek.com/2022/10/mojeek-now-six-billion-pages.html
- Lomas, N., Mehta, I. (2024): Ecosia and Qwant, two European search engines, join forces on an index to shrink reliance on Big Tech. URL: https://techcrunch.com/2024/11/11/ecosia-and-qwant-two-european-search-engines-join-forces-on-building-an-index-to-shrink-reliance-on-big-tech/
- Branz, M. (2024): Eine Ära geht zu Ende. URL: https://suma-ev.de/eine-aera-geht-zu-ende/
- Branz, M. (2024): Eine Ära ging zu Ende – und eine neue Zeit beginnt! URL: https://suma-ev.de/eine-aera-ging-zu-ende-und-eine-neue-zeit-beginnt/
Jan is co-founder of ViOffice. He is responsible for the technical implementation and maintenance of the software. His interests lie in particular in the areas of security, data protection and encryption.
In addition to his studies in economics, later in applied statistics and his subsequent doctorate, he has years of experience in software development, open source and server administration.