What to work on Monday
A week of customer complaints, sorted into problems, with the first move on each one. Here is what it decides, and the week it got badly wrong.
The only question it has to answer
A product manager gets a few hundred customer complaints a week and one afternoon. So they skim. Skimming feels fine, because you never find out what you missed. The complaint that mattered was in there and nobody counted it.
I built something that reads them instead. It reads reviews people posted publicly about an app I do not work on.
The first version handed back a ranked list. That is not an answer. A ranked list is a sorted queue. The deciding is still sitting inside it, untouched.
So I demoted the score to small print. Each problem now opens with a finding. Under it sits one first move, the person who owns it, and roughly what it costs. The page got longer and slower to scan. That is the trade I made.
Severity measures loudness, not cost
Every complaint gets an upset score. I ranked by it for weeks before noticing what it actually measures.
Upset is tone. It is how angry someone sounded while typing. It is not what the problem cost them. A customer who cannot find a product writes a furious review. A customer charged twice for one order often writes a short, tired one.
So ranking by upset buries the money. In the week above, 23 complaints cost somebody money. They sit across 7 different problems. Ranked on tone alone, most of them fall below problems that only irritated people.
The problem that came top last week shows the split. 44 people raised it. 13 described a package that was lost or delivered to the wrong address. 2 were charged for something that never arrived. Its upset score is 3.2, which is unremarkable. What it cost is not.
Every problem now carries what it cost as well as how it sounded. Four kinds: lost money, order never came, could not finish, just annoyed. That is two numbers to read instead of one. They disagree often enough to earn the second glance.
Then it told me 53 people were one person
One step grades how strong the evidence behind a problem is. I asked it to judge four things. Two of those I had already counted exactly. How many people raised the problem. How many app stores they came from.
It got both wrong. It described a problem raised by 53 customers as “only one person reported this issue”. That sentence rendered on the page, beside the number 53.
It was copying me. My instructions offered say “only one person reported this” as an example of plain language. It returned my example as its answer. Across that run, 41 of the gap descriptions were that sentence or its twin. 27 of them were contradicted by the counts printed next to them.
The cost was not cosmetic. Believing every problem was thin, it marked all 28 as weak evidence. Nothing came out ready to act on. The biggest problem of the week wore a badge reading “not enough to act on”. The product was confidently telling a PM to do nothing, and it shipped that way.
The fix gave the model more to do
I had asked it for the wrong thing. Counting is arithmetic, and I already had the number. So counting moved into code, where it cannot drift.
That freed the model up. It now reads a thousand badly written complaints and works out that four hundred are the same complaint. It writes the finding in a sentence a PM can repeat in a standup. It proposes the first move, names the team that owns it, and says roughly what it will take.
None of that existed before. The model does more of the work now, not less. It just does the part it is better at than I am, which is reading.
What it cannot do, and how I would know it works
60 of the 299 complaints it sorted last week did not fit any named problem. That is a fifth of them, sitting in a bucket called unclassified. The page says so rather than hiding it. It is the weakest part of the product and the honest number to lead with.
The metric I would judge it on is the decision rate. That is the share of surfaced problems a PM either acts on or deliberately defers. It tests the only claim the product makes. Someone who reads a problem and does neither was not given anything usable.
Deferring is recorded too, with the week it happened in. Every tool logs what got built. Almost none logs what was turned down and why. So “why did we skip returns in August” has an answer in November, instead of a shrug.
One number has to sit beside the decision rate. How often does the product call a problem ready when the evidence cannot carry it? The mistake above pinned that to the opposite extreme and everything looked unready. Watching one number without its opposite is how a team ships confident nonsense.
You can read this week yourself, or see how it is built.