Module 1: Statistical Thinking

Populations, Samples and Bias

Almost every data science question is really a question about a group you cannot fully observe. You want to know what all your customers think, but you survey a few hundred. You want to know whether a new feature works for every user, but you test it on a sample. Statistics is the discipline of learning about the whole from a part, and it only works if you understand how that part was chosen.

Populations, samples, parameters and statistics

The population is the entire group you want to draw conclusions about: all registered M-Pesa agents in Kenya, every transaction on your platform this year, all smallholder tea farmers in Kericho. A sample is the subset you actually measure.

A number describing the population is a parameter, for example the true average monthly sales of all agents. Parameters are usually unknown. A number calculated from your sample is a statistic, for example the average sales of the 400 agents you surveyed. You use statistics to estimate parameters. By convention, Greek letters often denote parameters (μ for the population mean, σ for the population standard deviation) and Latin letters denote sample statistics (x̄ for the sample mean, s for the sample standard deviation).

The gap between a statistic and the parameter it estimates has two sources:

• Sampling error: random variation because you measured only some of the population. It shrinks as samples grow, and later lessons show how to quantify it.
• Bias: systematic error that pushes estimates in one direction. A bigger sample does not fix bias; it just gives you a more precise wrong answer.

How samples are chosen

The main types of bias

• Selection bias: the way units enter the sample makes it unrepresentative. Surveying shoppers in a Westlands mall about household spending will not represent Nairobi as a whole.
• Non-response bias: the people who answer differ from those who do not. Customers who reply to a satisfaction email may be the very happy and the very angry.
• Survivorship bias: you only see the cases that "survived". Studying only businesses still operating after five years hides the reasons others failed.
• Measurement (response) bias: the measuring process distorts answers, through leading questions, faulty sensors or people under-reporting income.
• Undercoverage: part of the population is missing from the sampling frame, such as households without smartphones in an app-based survey.

Worked example: drawing samples in Python

Suppose you have a table of 20,000 loan customers across four regions and want a 500-person survey that represents each region in proportion.

import pandas as pd

customers = pd.read_csv("customers.csv")   # columns: id, region, ...

# Simple random sample
srs = customers.sample(n=500, random_state=42)

# Proportional stratified sample by region
frac = 500 / len(customers)
strat = (customers
         .groupby("region", group_keys=False)
         .sample(frac=frac, random_state=42))

# Compare regional shares: population vs each sample
print(customers["region"].value_counts(normalize=True).round(3))
print(srs["region"].value_counts(normalize=True).round(3))
print(strat["region"].value_counts(normalize=True).round(3))

The stratified sample matches the population's regional shares almost exactly; the simple random sample will be close but can drift, especially for small regions. Setting random_state makes your sample reproducible, which matters when someone needs to check your work.

Now imagine only 180 of the 500 reply. Before analysing, compare responders with non-responders on what you already know (region, loan size, repayment history). If responders have noticeably better repayment records, any satisfaction figure is likely to be optimistic, and you should say so in your report.

Common mistakes to avoid

• Believing a huge sample guarantees accuracy. A biased sample of a million is still biased.
• Treating data you happened to collect (website visitors, app users) as if it were a random sample of everyone.
• Forgetting to define the population precisely before collecting data. "Customers" could mean active customers, all accounts ever opened or this month's buyers.
• Ignoring non-response. Always report the response rate and check how responders differ.