Internet search engines have become an integral part of the modern web. Today, there are a variety of very different services with different goals and techniques to help users find exactly the information they are looking for. Starting with simple lists of web services and important files, through simple index engines to the recently emerged AI-based search services.
This blog article is part of a series of blog posts we are going to release over the coming months and is intended to give a brief introduction to the exciting topic of search engines and explain some of the problems with the search engine market and the way we use them.
In principle, there are a large number of different search engine concepts and specialized niche software for a wide variety of use cases. This article is limited to public web search platforms.
A brief history of search engines
Search engines began their campaign shortly after the birth of the public internet. It became clear early on that an information network only makes sense if users are able to find exactly the information they need. The first “search engines” were more like reference pages about which web servers were operated by whom, which files could be downloaded from where, or even consisted merely of manually generated lists of web services that users could browse through.
It was not until the mid-1990s that such services emerged, which we would understand as “search engines” today. The first commercially operated search engines that are still active today include the Lycos search engine and Yahoo!, which were created in 1994 and quickly became very popular.
In contrast to the previously manually created lists of web services or files, search engines were already using special programs (so-called “web crawlers”) from the mid-1990s, which look up content and websites across the entire Internet and add them to the search index of the operating search engine.
With the constant growth of internet services, methods quickly became necessary that could sort websites and content in search engine indices according to different quality measures. So-called “page ranking” quickly became established for many websites and is still a hotly debated topic today. This is because search engine operators gain power over which results users are shown first when they search for a term – and which results from certain sources are not shown at all (or are shown very far down). [1]
It was precisely this circumstance that Google, among others, took advantage of to mix targeted advertisements into search results, which could be bought by other companies, thus starting the commercialization of search engines. [1]
While the search engine market was initially highly competitive, the Google search engine finally prevailed around the turn of the millennium and still holds a large share of the market today, namely around 90% worldwide. [2, 3]
However, it is precisely this market dominance that is currently in risk of being overturned. While providers of various search engine concepts have been unable to take significant market share away from Google for decades, the AI-based search engines and chatbots that have emerged in recent years could potentially do just that. [4]
At the same time, both economists and data protectionists have been criticizing the increasing centralization of market power and the power to determine what people worldwide can search, see, buy or understand for years. If a single platform in the hands of a market-oriented player is used by a large part of humanity to read news, find out about products, network with other people and pursue their own hobbies and interests, this gives that platform enormous power, which not only has an economic impact, but can also have a strong effect on our society. [5, 6]
The optimization paradox
The quality of search results is the yardstick by which a search engine must be measured. Which results are displayed to users at the top of an endless list of search results determines whether the search engine is perceived as “good” or “bad”.
The perception of the quality of search results is, of course, highly subjective and prone to strong habituation effects. The lock-in effect can also occur here if users become accustomed (usually subconsciously) to how they have to formulate search queries to a particular search engine in order to obtain satisfactory results. If you then use a different search engine with the same expectations, which prepares or sorts results in a different way, this often leads to frustration among users.
Although most major search engines can typically display several million results for simple search queries, users often only look at the first few entries. These places are therefore highly coveted. Search Engine Optimization (SEO) describes the process and methods used to provide the web crawlers of search engines with exactly the right information to ensure that the respective website appears as high up in the search engines as possible.
This optimization frequently includes minimizing loading times, providing machine-readable summaries of the website content and keyword collections to describe the page. However, it can also include methods that deliberately attempt to fool search engines into believing content that does not exist in order to illegally appear at the top of search results. The advent of AI-generated content has also changed the way search engines should look for content. Generative AI can also be used to flood search engine web crawlers with a gigantic mass of content “optimized” for them. [7, 8, 9]
However, SEO is also criticized in other ways today. The constant development and secrecy of search engine ranking algorithms means that SEO is a constantly changing arms race. The more website operators are willing to invest in SEO, the more likely it is that they will appear high up in the results for general search queries. This reinforces the centralization effect of the internet, as the most relevant results from a multitude of different, small websites no longer appear, but rather content from large platforms is displayed much more prominently in search results. [10]
The era of chatbots
However, beyond the gigantic platform economy of search engine giants such as Google and the centralization of the internet through SEO and AI-generated content, there seems to be an interesting development in the search engine market since the emergence of chatbots controlled by generative AI. The development and improvement of large language models (LLMs), which form the basis of the generative AI behind the new chatbots, is leading to a noticeable turnaround. [11, 12]
More and more people seem to be using chatbots to get information on all kinds of topics. Instead of searching Google and the like from a variety of sources, users now often ask their questions to chatbots, which provide them with answers interactively in a kind of extremely energy-intensive “conversation”. It apparently doesn’t matter that the AI’s answers are sometimes completely made up and it is generally difficult for users to understand where the chatbots’ information actually comes from. [11, 12, 13, 14]
The end of classical search engines?
Despite the unbroken market power of a few search providers, the disruptive power of AI-based search platforms and the seemingly hopelessly commercialized market landscape, there is currently no sign of the end of traditional search engines. Nevertheless, the field of search engines is currently in a state of transition and will be subject to major changes sooner or later. For people with an affinity for data protection or the environment, in particular, this can mean some adjustment effects.
Over the coming months, we would like to examine the complexity of this topic several times from different angles and with a changing focus on sub-aspects in further blog articles. In doing so, we want to provide ideas and information that will enable people to find exactly the information they need in a better and more data protection-friendly way in future.
Sources
- Brin, Sergey and Page, Lawrence (1994): The Anatomy of a Large-Scale Hypertextual Web Search Engine. URL: http://infolab.stanford.edu/~backrub/google.html
- Gandal, N. (2001). The Dynamics of Competition in the Internet Search Engine Market. UC Berkeley: Department of Economics. Retrieved from https://escholarship.org/uc/item/0h17g08v
- Statcounter (2024): Search Engine Market Share Worldwide. URL: https://gs.statcounter.com/search-engine-market-share#yearly-200901-202410
- Pierce, David (2023): The AI assistant revolution is more than 50 years in the making. URL: https://www.theverge.com/23888985/ai-assistant-you-chatgpt-bard-siri-alexa-vergecast-podcast
- Scholz, Nina (2024): Monopolexperte Ulrich Müller: „Google muss zerschlagen werden“. URL: https://www.freitag.de/autoren/nina-scholz/ulrich-mueller-von-lobbycontrol-google-muss-zerschlagen-werden
- ZDF heute (2024): Zerschlägt die US-Regierung Google?. URL: https://www.zdf.de/nachrichten/wirtschaft/unternehmen/google-us-regierung-zerschlagung-monopol-100.html
- Zeller, Tom (2006): A New Campaign Tactic: Manipulating Google Data. URL: https://www.nytimes.com/2006/10/26/us/politics/26googlebomb.html
- Elliott, Kirsten (2024): Google Scholar is not broken (yet), but there are alternatives. URL: https://blogs.lse.ac.uk/impactofsocialsciences/2024/10/22/google-scholar-is-not-broken-yet-but-there-are-alternatives/
- Kra, Alex (2021): The mermaid is taking over Google search in Norway. URL: https://alexskra.com/blog/the-mermaid-is-taking-over-google-search-in-norway/
- Brereton, Dimitri (2022): Google Search Is Dying. URL: https://dkb.io/post/DEPR_google-search-is-dying
- Edwards, Benj (2024): OpenAI launches ChatGPT with Search, taking Google head-on. URL: https://arstechnica.com/ai/2024/10/openai-launches-chatgpt-with-search-taking-google-head-on/
- Love, Julia and Alba, Davey (2024): Google’s AI Search Gives Sites Dire Choice: Share Data or Die. URL: https://www.bloomberg.com/news/articles/2024-08-15/google-s-search-dominance-leaves-sites-little-choice-on-ai-scraping
- Slater, Joe and Humphries, James and Townsen Hicks, Michael (2024): ChatGPT Isn’t ‘Hallucinating’—It’s Bullshitting! URL: https://www.scientificamerican.com/article/chatgpt-isnt-hallucinating-its-bullshitting/
- Sloan, Karen (2023): A lawyer used ChatGPT to cite bogus cases. What are the ethics? URL: https://www.reuters.com/legal/transactional/lawyer-used-chatgpt-cite-bogus-cases-what-are-ethics-2023-05-30/
Jan is co-founder of ViOffice. He is responsible for the technical implementation and maintenance of the software. His interests lie in particular in the areas of security, data protection and encryption.
In addition to his studies in economics, later in applied statistics and his subsequent doctorate, he has years of experience in software development, open source and server administration.