Python for data analysis: a working starting point

Python suits analysis where the cleaning is substantial, the data comes from an API or database, or the same pipeline will run again on new data. For a one-off t-test on a tidy spreadsheet, SPSS or even Excel will get you there faster.

The workflow below covers the large majority of practical analysis.

The libraries worth installing first

• pandas — tabular data. The centre of everything.
• numpy — numerical operations underneath pandas.
• matplotlib and seaborn — plotting.
• scipy.stats — classical statistical tests.
• statsmodels — regression with proper statistical output, including p-values and confidence intervals that scikit-learn does not give you.
• scikit-learn — machine learning and predictive modelling.
• pingouin — statistical tests with effect sizes reported by default.

A workflow that holds up

import pandas as pd
import numpy as np

# 1. Load, and look before you leap
df = pd.read_csv("survey.csv")
print(df.shape)
print(df.dtypes)
print(df.isna().sum())
print(df.describe(include="all"))

# 2. Clean explicitly — never silently
df.columns = df.columns.str.strip().str.lower().str.replace(" ", "_")
df["age"] = pd.to_numeric(df["age"], errors="coerce")

before = len(df)
df = df.drop_duplicates(subset="respondent_id")
print(f"Removed {before - len(df)} duplicate rows")

# 3. Reverse-score before computing scale means
reverse_items = ["q3", "q7"]
df[reverse_items] = 6 - df[reverse_items]      # 5-point scale
df["engagement"] = df[["q1", "q2", "q3", "q4"]].mean(axis=1)

# 4. Group and summarise
summary = (
    df.groupby("department")
      .agg(n=("engagement", "size"),
           mean=("engagement", "mean"),
           sd=("engagement", "std"))
      .round(2)
)
print(summary)

The mistakes that give wrong answers quietly

These are dangerous precisely because nothing raises an error. The code runs and the number is simply wrong.

• Chained assignment. df[df.age > 30]["score"] = 0 may modify a copy and silently do nothing. Use .loc: df.loc[df.age > 30, "score"] = 0.
• Merges that duplicate rows. A many-to-many join quietly multiplies your data. Always check len(df) before and after, or pass validate="one_to_one".
• Missing values excluded inconsistently. mean() skips NaN by default; numpy's does not. Decide your policy and apply it explicitly.
• Forgetting to reverse-score. A reversed item left unreversed will wreck a scale mean and depress your reliability coefficient.
• Mutating while iterating. Modifying a DataFrame inside a loop over it produces unpredictable results. Build a list and assign once.
• Index misalignment. Operations on Series align on index, not position. After filtering, reset_index(drop=True) when you need positional behaviour.

Statistical tests with usable output

scipy gives you a statistic and a p-value. For reporting you also need effect sizes and confidence intervals, which is where pingouin and statsmodels earn their place.

import pingouin as pg

# Independent t-test with Cohen's d and CI included
result = pg.ttest(group_a["score"], group_b["score"], correction="auto")
print(result[["T", "dof", "p-val", "cohen-d", "CI95%"]])

# One-way ANOVA with partial eta squared
aov = pg.anova(data=df, dv="score", between="condition", effsize="np2")
print(aov)

# Regression with full statistical output
import statsmodels.formula.api as smf
model = smf.ols("score ~ age + experience + C(department)", data=df).fit()
print(model.summary())

Work in notebooks, but write them for a reader

• Use markdown cells to explain what each block does and why.
• Restart the kernel and run all before you trust a result — out-of-order execution is the single biggest source of irreproducible notebooks.
• Keep raw data read-only. Never overwrite the source file.
• Number your notebooks (01-clean.ipynb, 02-analyse.ipynb) so the order is obvious.
• Export a requirements.txt. "It worked six months ago" is a version problem.

Questions this raises