Derick TW Data & Code

Python for field and survey data

If you know Excel, have never written code, and have a messy export open right now, this is the right place to start.

The track assumes nothing about programming and everything about the work: every lesson begins with a problem you have actually had, ships with a dataset you can download, and ends with something that runs.

1 Getting set up

The honest version of when Python is worth it, and how to install it without breaking your laptop.

  1. Why Python, and when Excel still wins 8 min
  2. Install Python without the headache 12 min

2 First clean

Take a real household survey export from the state it arrives in to a table you can analyse, deciding what each defect actually means rather than deleting it.

  1. Open a Kobo export without losing data 25 min
  2. Select, filter, rename 22 min
  3. Four kinds of missing in survey data 28 min
  4. The district spelled four ways 26 min
  5. Dates that arrive in five formats 30 min
  6. Export a table people can use 30 min

3 Shaping

Get the table into the shape the analysis needs: multi-answer questions, the roster join, validating it, and the reshape that repeat groups force.

  1. select_multiple columns 24 min
  2. Join the roster to the household 26 min
  3. Validating a merge 22 min
  4. Long and wide 25 min

4 Analysis

Produce numbers that survive a review, with the denominator stated and the definition written down.

  • Descriptives that survive a review
  • Survey weights: what they are, when you must use them
  • Disaggregation by district, sex and age band
  • Calculating an indicator, and documenting its definition
  • Data quality as code: outliers, impossible values, enumerator effects

Being written.

5 Visualisation

Charts that answer one question, and maps from the GPS points the form collected.

  • Charts that answer one question
  • GPS points to a map with geopandas
  • An automated chart pack for a monthly report

Being written.

6 Automation

Turn the clean into something that runs by itself, on a schedule, and tells you when it fails.

  • From notebook to a script you can rerun
  • Scheduling it, on Windows and on a server
  • Writing into an Excel template without destroying the formatting
  • Logging, so you know which run failed and why

Being written.

7 APIs

Stop exporting by hand. Pull from Kobo and DHIS2 directly, and push results back.

  • The KoboToolbox API: stop exporting by hand
  • The DHIS2 API: pulling, and pushing back
  • A small pipeline: Kobo to clean to Power BI

Being written.

8 AI, used honestly

Where a language model earns its place in a data workflow, and the line it must not cross.

  • Getting an LLM to write cleaning code, and checking it
  • Summarising open-ended responses, with the caveats
  • What never to paste into a chatbot

Being written.