How do search engines work?

Search engines are incredibly complex yet what they do is surprisingly simple.

They crawl the internet to find new or updated sites and pages (so that they can establish what’s out there).
They build an index that categorises each and every page (so that they can easily find them for you when you search).
They rank them all according to their relevance (so that they can establish which are the most useful to you).
They return the best results possible (i.e. the most relevant) on a search engine results page (SERP) for people typing in search queries.

Each search engine each uses its own software programs and algorithms, but the way they all work is the same.

CRAWLING
INDEXING
RANKING
RETURNING RESULTS

Let’s take a closer look at each of these.

Search engines and crawling

Search engines send computer programs through the internet to discover content, such as your web pages, images and videos. This is called ‘crawling’. And the programs that crawl are sometimes referred to as ‘bots’ (short for robots), ‘crawlers’ or ‘spiders’.

The bots follow links – just like you do when you are surfing the web. Except the bots never stop for a break!

They ceaselessly leap from page to page, visiting and revisiting them in their endless search for new content, new links and new pages.

The search engine’s insatiable desire to index more content is what drives these bots on.

Search engine indexing

The index is a gigantic virtual warehouse where all the web pages and content tracked down by the bots is stored. This warehouse is neatly ordered, so that the search engine can dip in and out of its content stores to return results in response to search queries.

The index, however, is rather picky: not everything found by the undiscerning bots will be deemed worthy of storing.

And one of the easiest ways to be ignored by the index is by having duplicate content.

Duplicate content is text that appears in multiple places across the web or on the same site. If you are using a manufacturer’s description word-for-word on your shopping site you may well not ever appear in the search results.

Which is to say content must be unique if it is to stand a fair chance of being indexed and then ranked.

Search engine ranking

When you type in a search query into a search engine it compares the phrase you use to its organised index.

I just typed ‘search engine rank’ into Google…

… and in just 0.66 seconds it told me that there were more than two million pages, documents or files (results) that it felt may be relevant to my query.

To save you time visiting each of these the search engine tries to rank which pages best match what you are after and will be the best source of information for you.

Now, the way search engines rank pages is a big secret. It is the online equivalent of Colonel Sanders’ secret KFC recipe – except Google’s recipe has most certainly never been leaked.

The method or formula used for ranking is often called the algorithm.

There are hundreds of signals or factors that search engines use to determine the rank of the content they find – and no two engines are the same.

They may include things like the words used on a page or the number of other websites that link to it.

Google claims (here) to use more than 200 ranking factors. And it continually jiggles and juggles its algorithm.

Brian Dean famously tried to list what these 200 ranking factors may be – take a look at what he found here.

Of course, these ranking factors are far from static.

Google’s algorithm changes as it tries to prevent people ‘gaming’ the system – you can follow some of its twists and turns here, courtesy of Moz.

By continually changing the way it ranks content Google believes it can return more accurate and relevant results for the user.

In a later post we’ll explore just how SEO has changed, but if you can’t wait you can get the lowdown on all things SEO in my Fast track SEO course – available now from Amazon.

Next up on our journey of SEO discovery is a look at the question that is on everyone’s lips: what does Google want?

For a list of all the posts in this series you can find them here.