Select, filter, rename

The problem

A Kobo export arrives with every field the form ever had, named the way the form named them. You need eight of the sixty columns, the households that consented, and names a colleague can read.

What you will have at the end

A narrower, clearer table, and a clear head about [] versus .loc[], which is the thing that confuses everyone for about a week.

The steps

Pick columns

A list inside the brackets gives you a smaller table:

import pandas as pd

df = pd.read_csv("household-survey-messy.csv", dtype=str, keep_default_na=False)

keep = ["_uuid", "district", "consent", "household_size", "monthly_income_ugx"]
small = df[keep]

The order of your list is the order of the columns. That alone is worth knowing — it is the easiest way to put a table into the shape a report wants.

Filter rows

A comparison gives you a column of True and False, and that column selects rows:

consented = df[df["consent"].str.strip().str.lower().isin(["yes", "y", "1", "true"])]
print(len(consented))

Read it inside out: build the test, then use it. Writing the test on its own line is usually clearer, and it is much easier to debug:

is_consented = df["consent"].str.strip().str.lower().isin(["yes", "y", "1", "true"])
consented = df[is_consented]

Combining tests needs & and |, not and and or, and every part needs its own brackets:

big_in_gulu = df[(df["district"].str.strip().str.lower() == "gulu")
                 & (df["household_size"] != "")]

Leaving the brackets out produces an error about truth values being ambiguous. That message means exactly this mistake, every time.

Both at once, with .loc

.loc[rows, columns] does the two steps together:

result = df.loc[is_consented, keep]

This is the form to prefer when you are about to change something. df[...] sometimes gives you a copy and sometimes a view, and assigning to a copy changes nothing while reporting no error. .loc is unambiguous:

df.loc[is_consented, "status"] = "interviewed"   # reliable

Rename

tidy = small.rename(columns={
    "_uuid": "household_id",
    "monthly_income_ugx": "income_ugx",
})

Only the columns you name change; the rest keep their names. To tidy every column at once:

df.columns = df.columns.str.strip().str.lower().str.replace(" ", "_")

Kobo's grouped fields arrive as group_demographics/age. Keep just the part after the slash:

df.columns = df.columns.str.split("/").str[-1]

Do this once, early, and every later line gets shorter.

Where it breaks

The copy warning

SettingWithCopyWarning: A value is trying to be set on a copy of a slice

You filtered, then assigned to the result. Either do it in one step with .loc, or make the copy explicit:

consented = df[is_consented].copy()
consented["income_ugx"] = ...

The warning is telling you your change may have gone nowhere. Do not silence it.

Comparing text that has spaces in it

df["district"] == "Gulu" misses "Gulu " and "gulu". Normalise before comparing — .str.strip().str.lower() — or you will quietly drop rows and your total will be slightly, unaccountably low. The whole of the district lesson is about this.

A column name you can see but cannot use

KeyError: 'district' when district is plainly there usually means trailing whitespace in the header, or the byte order mark from the previous lesson. Check with repr:

print([repr(c) for c in df.columns[:3]])

Your turn

Produce a table of consenting households in Gulu with household size and income, with columns named household_id, district, people and income_ugx, in that order.

Then count the rows twice: once comparing district exactly to "Gulu", once after stripping and lowercasing. The difference is the cost of not normalising.

Four kinds of missing — and why collapsing them quietly breaks an indicator.