Module 1: SEO Foundations

How Search Engines Work

Search engine optimisation (SEO) is the work of helping search engines find, understand and recommend your pages to people searching for what you offer. Before you can improve rankings, you need a clear picture of what a search engine actually does with a website. Most SEO mistakes come from misunderstanding this process, for example writing great content on a page that Google cannot crawl, or blocking a whole site by accident.

Stage 1: Crawling

Search engines use automated programs called crawlers (Google's is called Googlebot) to discover pages on the web. Crawlers find new pages mainly by following links from pages they already know, and by reading XML sitemaps that site owners submit. When Googlebot visits a page, it downloads the HTML and, for most sites, renders the page much as a modern browser would, including running JavaScript, so it can see the final content.

Site owners can give crawlers instructions in a file called robots.txt at the root of the domain (for example example.co.ke/robots.txt). It can tell crawlers not to crawl certain sections, such as an admin area. A single wrong line in this file can block the entire site, which is why you should always check it.

Stage 2: Indexing

After crawling, the search engine tries to understand the page: what it is about, its language, its main content, images, links and structured data. If the page is useful and allowed, it is stored in the index, a huge database of pages. Not every crawled page gets indexed. Common reasons a page is left out include:

• A noindex instruction in the page's meta robots tag or HTTP header.
• The page is a duplicate or near duplicate of another page, and the search engine chooses the other version as the main (canonical) one.
• The page is thin or low quality, such as an empty category or a page with only a few words.
• The page returns an error (such as 404) or redirects elsewhere.

If a page is not in the index, it cannot rank, no matter how good it is.

Stage 3: Ranking (serving results)

When someone searches, the search engine looks through its index for pages that match and orders them by what it judges most helpful. Google says its systems consider many factors, including the meaning of the query, the relevance of content, the quality of content, the usability of pages, and context such as the user's location, language and device. Nobody outside Google knows the exact weighting, and it changes as Google updates its systems. Anyone who claims to know "the algorithm" in detail is guessing.

Results also appear in many formats beyond the classic ten blue links: local map packs, images, videos, "People also ask" boxes, product listings and automatically generated overview summaries in some markets. Your SEO strategy should consider which formats appear for your important searches.

Worked example: a new bakery website

"Mama Wanjiru's Bakery" in Nakuru launches a website with a home page, a cakes page, a contact page and a blog post about choosing a wedding cake. Three weeks later, a search for the bakery's name shows nothing. Here is how the owner works through the three stages:

1. Crawling: she opens /robots.txt and finds Disallow: /, left over from when the site was being built. This tells all crawlers not to crawl anything. She removes it.
2. Discovery: she verifies the site in Google Search Console and submits the XML sitemap generated by her website platform.
3. Indexing: using the URL Inspection tool in Search Console, she checks the home page, sees that it can now be crawled, and requests indexing. She also checks that the pages do not carry a noindex tag, which some platforms add while a site is in draft mode.

Within days to a few weeks, the pages start appearing for the bakery's name. Ranking for broader searches like "wedding cakes Nakuru" will take longer and depends on content quality, local signals and competition.

Common mistakes to avoid

• Leaving development settings on after launch, such as robots.txt blocks or noindex tags.
• Hiding important content behind logins, or loading it in ways crawlers cannot access.
• Creating pages with no internal links pointing to them (orphan pages), which crawlers may never find.
• Expecting instant results. Crawling, indexing and ranking take time, and ranking is earned over months.