If you ask “what is a search engine?” you will see so many different answers.
The goal of this lesson is to give you a clear understanding of search engine, how it works, top search engines in the market, and how search engine add new websites to its database.
Search Engine Definition
This is how I will define search engine.
It is a platform where you can search for something you want to learn more about, and it will present you with relevant information it has from its database.
That sounds simple, right? Just like a directory.
However, web search engines are complicated because their database is the Internet (World Wide Web) and they need to figure how to show accurate and the best search results to a query.
Let’s look at some facts to understand the complexity:
- More than 1.1 billion websites exist on the internet and you can just imagine the total number of web pages! (NetCraft, 2021)
- Search engines contain at least 5.55 billion indexed pages (WorldWideWebSize, 2021)
- Google alone processes over 40,000 search queries per second, and that’s almost 3.5 billion a day. (InternetLiveStats, 2021)
Here are some potential problems search engines face due to this:
- How do we find all the webpages on the Internet?
- How do we store all web page information in our database?
- How do we find and retrieve data from our database for a search query?
- In what order should we display the results?
- How to present the results for a great user experience?
- What’s the best way to manage all users simultaneously?
A search engine has to find a way to answer all the above questions to be effective.
How Does a Search Engine Work?
Now let’s quickly take a look at how search engine works when you type a search query.
- A user goes to the search engine
- User searches for something
- The search engine scans its database
- With the help of machine and human guidelines, it finds the best search results
- The user is presented with an accurate set of results to his/her search query
Let’s find out why all search engines are not the same.
First, we will take a look into the search engine market share.
The below market share data is as of Feb, 2021 (StatCounter)
- Google: 91.86%
- Bing: 2.71%
- Yahoo: 1.46%
- Baidu: 1.13%
- Yandex: 0.087%
- DuckDuckGo: 0.66%
From the above percentages, it’s clear that Google is the king of search engines.
It is because they have provided the best answers for all the problems search engines and users faced.
The Google search engine received 88.36 billion visits this year (Similarweb, 2019).
Google’s index or database contains hundreds of billions of web pages. (Google, Unknown)
Since Google is the market leader, it makes sense to concentrate most of our SEO efforts to it.
So, in this free SEO course I will primarily focus on Google SEO optimization.
However, as a secondary option, you can also optimize for Bing & Yahoo since they share databases and all the Microsoft devices, and the operating system’s default search engine is Bing.
How Do Search Engines Add Websites?
Let’s quickly see how search engine identify and add websites to its database.
The process includes three main stages:
In this process, search engines use automated apps that continually search for all webpages on the internet (World Wide Web).
These automated programs are called search crawlers or web-spiders.
There are several ways search engines gather information about web pages, but the most common way is by following links from web-pages already known to them.
In this stage, the search engine visits the pages it crawled before.
Content on every web page, including the text, images, video, PDF, and more, will be analyzed to determine the page’s topic and catalog it so these can be easily retrieved when needed.
All the information will be stored (cached) in their database (Search index).
Serving Search Results
It happens when a person searches for something on a search engine.
Search engines will display the most relevant & best set of search results from its database (Search Index).
The “Best” search results will be determined by several factors, including searcher location, language, device, search engine algorithms (search rules & guidelines), user intent, SEO competition, and more.
In this lesson, we learned what is search engine and understood the complexity behind them.
We also found out what are the top 3 search engines in the world.
Finally, we also looked into how search engine works and how it does add websites into it’s database.