Diagnose25 of 26 · about 10 min

Diagnose: the evaluation that was too good

A routing model scored 94% on the holdout set, up from 71%. Nobody could reproduce the improvement in the pilot. The model had not improved; the evaluation had broken.

Tickets were sampled into a development set and a holdout set — but the same ticket appears more than once in the export, and the near-duplicates went to different sides. A case the model was tuned against was also a case it was graded on.

Leakage is not usually dramatic. It is one join, one resample, one helpful person topping up the dev set from the same pool. The defence is a check that runs every time the sets are built, not care.

Example

overlap = set(dev_ids) & set(holdout_ids)
assert not overlap, f"{len(overlap)} tickets are in both sets"

One assertion, run on every build of the datasets. It is worth more than the rest of the harness.

Your task

Write leakage_report(dev, holdout) returning a tuple (shared_ids_sorted, adjusted_score) where the adjusted score is accuracy over the holdout rows that are NOT shared. Print the shared ids, the raw score and the adjusted score, each as a percentage with no decimals.

Stuck?

Explain it

The adjusted score is worse than the score you had before the model changed. What do you put in the release recommendation?

Not graded — this is for you.

Where this goes

D7 is the evaluation harness and D8 the release recommendation. Reproducibility is assessed directly: a result nobody else can reproduce does not count as evidence.

DEV = ["TCK-0001", "TCK-0002", "TCK-0003", "TCK-0004"]

HOLDOUT = [
    {"ticket_id": "TCK-0005", "correct": True},
    {"ticket_id": "TCK-0002", "correct": True},
    {"ticket_id": "TCK-0006", "correct": False},
    {"ticket_id": "TCK-0003", "correct": True},
    {"ticket_id": "TCK-0007", "correct": False},
    {"ticket_id": "TCK-0008", "correct": True},
]


def leakage_report(dev, holdout):
    return ([], 0)


shared, adjusted = leakage_report(DEV, HOLDOUT)
raw = round(100 * len([h for h in HOLDOUT if h["correct"]]) / len(HOLDOUT))
print("shared:", ",".join(shared) if shared else "none")
print(f"raw: {raw}%")
print(f"adjusted: {adjusted}%")

Ctrl+Enter checks

This is a teaching runtime for a subset of Python, running in your browser. Integers are exact and the errors are written in plain English, but the standard library is a small subset and anything missing is refused by name rather than approximated. Your project runs on real Python.

Press Run to see what your program does, or Check when you think it is right. Everything runs here in your browser.