Software Guides 11 min read

Python for data analysis: a working starting point

The pandas workflow that covers most real analysis, the mistakes that produce wrong answers silently, and when Python is the right choice over SPSS or Excel.

Navy laptop with connected teal modules beside an abstract technical drawing and brass connector.
On this page
  1. The libraries worth installing first
  2. A workflow that holds up
  3. The mistakes that give wrong answers quietly
  4. Statistical tests with usable output
  5. Work in notebooks, but write them for a reader
  6. Frequently asked questions

Python suits analysis where the cleaning is substantial, the data comes from an API or database, or the same pipeline will run again on new data. For a one-off t-test on a tidy spreadsheet, SPSS or even Excel will get you there faster.

The workflow below covers the large majority of practical analysis.

The libraries worth installing first

  • pandas — tabular data. The centre of everything.
  • numpy — numerical operations underneath pandas.
  • matplotlib and seaborn — plotting.
  • scipy.stats — classical statistical tests.
  • statsmodels — regression with proper statistical output, including p-values and confidence intervals that scikit-learn does not give you.
  • scikit-learn — machine learning and predictive modelling.
  • pingouin — statistical tests with effect sizes reported by default.

A workflow that holds up

import pandas as pd
import numpy as np

# 1. Load, and look before you leap
df = pd.read_csv("survey.csv")
print(df.shape)
print(df.dtypes)
print(df.isna().sum())
print(df.describe(include="all"))

# 2. Clean explicitly — never silently
df.columns = df.columns.str.strip().str.lower().str.replace(" ", "_")
df["age"] = pd.to_numeric(df["age"], errors="coerce")

before = len(df)
df = df.drop_duplicates(subset="respondent_id")
print(f"Removed {before - len(df)} duplicate rows")

# 3. Reverse-score before computing scale means
reverse_items = ["q3", "q7"]
df[reverse_items] = 6 - df[reverse_items]      # 5-point scale
df["engagement"] = df[["q1", "q2", "q3", "q4"]].mean(axis=1)

# 4. Group and summarise
summary = (
    df.groupby("department")
      .agg(n=("engagement", "size"),
           mean=("engagement", "mean"),
           sd=("engagement", "std"))
      .round(2)
)
print(summary)

The mistakes that give wrong answers quietly

These are dangerous precisely because nothing raises an error. The code runs and the number is simply wrong.

  • Chained assignment. df[df.age > 30]["score"] = 0 may modify a copy and silently do nothing. Use .loc: df.loc[df.age > 30, "score"] = 0.
  • Merges that duplicate rows. A many-to-many join quietly multiplies your data. Always check len(df) before and after, or pass validate="one_to_one".
  • Missing values excluded inconsistently. mean() skips NaN by default; numpy's does not. Decide your policy and apply it explicitly.
  • Forgetting to reverse-score. A reversed item left unreversed will wreck a scale mean and depress your reliability coefficient.
  • Mutating while iterating. Modifying a DataFrame inside a loop over it produces unpredictable results. Build a list and assign once.
  • Index misalignment. Operations on Series align on index, not position. After filtering, reset_index(drop=True) when you need positional behaviour.

Print the row count after every operation that could change it. It takes a second and catches most silent data-loss bugs immediately.

Statistical tests with usable output

scipy gives you a statistic and a p-value. For reporting you also need effect sizes and confidence intervals, which is where pingouin and statsmodels earn their place.

import pingouin as pg

# Independent t-test with Cohen's d and CI included
result = pg.ttest(group_a["score"], group_b["score"], correction="auto")
print(result[["T", "dof", "p-val", "cohen-d", "CI95%"]])

# One-way ANOVA with partial eta squared
aov = pg.anova(data=df, dv="score", between="condition", effsize="np2")
print(aov)

# Regression with full statistical output
import statsmodels.formula.api as smf
model = smf.ols("score ~ age + experience + C(department)", data=df).fit()
print(model.summary())

Work in notebooks, but write them for a reader

  • Use markdown cells to explain what each block does and why.
  • Restart the kernel and run all before you trust a result — out-of-order execution is the single biggest source of irreproducible notebooks.
  • Keep raw data read-only. Never overwrite the source file.
  • Number your notebooks (01-clean.ipynb, 02-analyse.ipynb) so the order is obvious.
  • Export a requirements.txt. "It worked six months ago" is a version problem.

Frequently asked questions

R has deeper coverage of statistical methods and better defaults for reporting. Python is stronger for data engineering, machine learning and integrating analysis into a wider system. Either is defensible; use what your field and your collaborators use.

Ask first. Many supervisors accept any properly reported analysis, but some assessment criteria specifically reference SPSS output formats.

Read in chunks with pd.read_csv(chunksize=...), use efficient dtypes such as category, or move to Polars or DuckDB, both of which handle larger-than-memory data with familiar syntax.

Python pandas data analysis Jupyter
Share LinkedIn X Facebook WhatsApp

Need expert help with this?

Send the brief and the deadline. You get a reference number straight away and a fixed quote, usually within one working day.

Get Expert Help Prefer to talk it through? Ask on WhatsApp

Keep reading

Related guides

Software Guides 8 min read

R vs RStudio: what the difference actually is

They are not competitors and you are not choosing between them. Here is what each one is, why you install both, and when R is worth the learning…

30 Aug 2026 Read
Get Expert Help Chat on WhatsApp Email us directly