Write26 of 26 · about 9 min

Cost per useful answer, not cost per call

A cheap call that produces an answer an agent has to redo is not cheap. The number that matters to the customer is the cost per ticket the system actually handled, and it is always worse than the cost per call.

Latency works the same way. A p95 of 2.4 seconds is not a number on its own — it is a number against the budget in the charter, which for Northstar was cut to 2 seconds part way through the engagement.

A release recommendation is a comparison against predeclared criteria, not a table of impressive figures. If the criteria were written down first, the recommendation mostly writes itself.

Example

cost_per_success = total_cost_cents / successes
verdict = "go" if (accuracy >= MIN_ACCURACY and p95 <= MAX_P95) else "no-go"

Both conditions, decided in advance, written down before the run.

Your task

Load runs.json. Print, for the baseline and the candidate: accuracy as a percentage with no decimals, p95 latency, and cost per successful ticket in cents to two decimals (0.00 when nothing was spent). Then print the verdict against the criteria: accuracy at least 78%, p95 at most 2000 ms.

Stuck?

Explain it

The candidate is more accurate than the baseline and you are recommending no-go. How do you present that to the sponsor?

Not graded — this is for you.

Where this goes

D8 is the release recommendation. It must compare at least two configurations against the baseline, cite the failures, and trace the verdict to criteria written down before the run.

import json

MIN_ACCURACY = 78
MAX_P95_MS = 2000

runs = json.loads(open("runs.json").read())


def summarise(run):
    return (0, 0, 0.0)


for key in ["baseline", "candidate"]:
    run = runs[key]
    accuracy, p95, cost_each = summarise(run)
    print(f'{run["name"]}: accuracy {accuracy}%, p95 {p95}ms, {cost_each} cents per success')

# print the verdict for the candidate here

Files you can open: runs.json

Ctrl+Enter checks

This is a teaching runtime for a subset of Python, running in your browser. Integers are exact and the errors are written in plain English, but the standard library is a small subset and anything missing is refused by name rather than approximated. Your project runs on real Python.

Press Run to see what your program does, or Check when you think it is right. Everything runs here in your browser.