Diagnose15 of 26 · about 12 min

Why did this assistant give the wrong answer?

A support assistant told a customer their warranty claim needed no site report. It did. The claim was rejected, the customer complained, and the account manager wants to know how it happened.

Nothing is broken in the model and nothing is broken in the retrieval. The assistant retrieved two versions of the same policy and picked the one that matched the question BETTER — which was the old one, because the new version is longer and mentions more things. Relevance and correctness are not the same quantity, and a retrieval score only measures the first.

The fix is not a better prompt. It is a rule the system did not have: **a policy applies to a claim by DATE, and a superseded version does not apply to anything raised after its successor took effect.** Selection by score, then filtering by date, gets this wrong. Filtering first, then ranking, gets it right.

Everything below is a recorded fixture — the retrieval trace as it happened. Nothing calls a model.

Example

# What the system did:
best = max(candidates, key=lambda c: c["score"])   # the most RELEVANT chunk

# What it needed to do:
applicable = [c for c in candidates if c["effective_from"] <= raised_on]
best = max(applicable, key=lambda c: c["effective_from"])   # the one IN FORCE

Rank within what is applicable. Never the other way round.

Your task

Fix policy_for so it returns the policy **in force on the day the claim was raised**, and None when no policy was in force yet. Then run it over all three claims. The printed line must name the document, the claim window in months, and whether a site report is required.

·Select by effective_from, not by retrieval score

Stuck?

Explain it

Write the two sentences you would send the account manager: what went wrong, and what changes so it cannot happen again. Do not blame the model.

Not graded — this is for you.

Where this goes

Project 5 builds the knowledge assistant this came from. The date rule, the citation that names a version, and the test that a superseded document cannot be cited are all in its acceptance criteria.

import json

data = json.loads(open("retrieval.json").read())
claims = json.loads(open("claims.json").read())
candidates = data["candidates"]


def policy_for(raised_on):
    """Return the policy in force on raised_on, or None."""
    # BUG: this picks the best-scoring chunk and ignores the date entirely.
    return max(candidates, key=lambda c: c["score"])


for claim in claims:
    chosen = policy_for(claim["raised_on"])
    if chosen is None:
        print(f'{claim["ticket_id"]} raised {claim["raised_on"]} -> no policy in force; escalate')
    else:
        evidence = "site report required" if chosen["requires_site_report"] else "site report not required"
        print(f'{claim["ticket_id"]} raised {claim["raised_on"]} -> {chosen["doc_id"]} (window {chosen["window_months"]} months, {evidence})')

Files you can open: retrieval.json, claims.json

Ctrl+Enter checks

This is a teaching runtime for a subset of Python, running in your browser. Integers are exact and the errors are written in plain English, but the standard library is a small subset and anything missing is refused by name rather than approximated. Your project runs on real Python.

Press Run to see what your program does, or Check when you think it is right. Everything runs here in your browser.