It was a good number. Specific, confident, the kind of figure that ends an argument in the room where a decision gets made. It was also invented. I almost didn't check.
I was building a business case, the standard artifact, the one that turns "we should probably do this" into "here is what it costs to not do this." I asked the model to help me quantify the upside. It did. ROI percentages. Mean-time-to-resolution improvements. A cost-of-inaction figure with a dollar sign in front of it. Every number carried the same quiet authority, and every one was framed the same way: as an industry benchmark. Framed as a benchmark, as if it had been measured somewhere and I was simply reporting it forward.
None of them had been measured anywhere. I went looking for the source behind one of the cleaner figures, because I wanted the citation for the slide, and there was no source. There was no study. There was a very confident sentence and nothing underneath it.
A wrong answer usually announces itself. Code throws an error. A build fails. A calculation you can check a second way comes out different the second way. The failure has an edge you can feel.
Fabricated data has no edge. It arrives in the same voice as the true things around it, same syntax, same confidence, same formatting. The model doesn't change register when it stops knowing and starts filling in. I rarely see anyone say this plainly: the fabrication isn't tucked in some obviously sketchy corner of the output where you'd think to look for it. It's doing structural work, sitting in the middle of the paragraph, indistinguishable from the sentences that earned their place.
I was close. The case was most of the way built, numbers formatted, narrative clean, and the whole thing was one copy-paste from going in front of someone who would have made a call based on it. A real decision. Money downstream of figures that were generated rather than found.
If those numbers had shipped and someone had checked them later, and someone always checks, eventually, the cost wouldn't have been that one embarrassment. It would have been every number I'd ever presented, retroactively. Trust doesn't degrade locally. One invented benchmark in one business case doesn't just discredit that benchmark. It puts a question mark over everything with your name attached to it, going backward.
Every number gets a source before it gets a sentence. If a figure can't be traced to something real, a measurement, a document, a citation I can open, it doesn't go in the artifact. If it's an estimate, it gets labeled an estimate, in the text, where the reader can see the label. The model doesn't get to skip that step. Neither do I.
The lesson people expect me to have taken from this is distrust of the tool. That's not quite it. The model did what these models do, it produced fluent, plausible, well-formed output on demand, which is the entire point of it. Asking it to never fabricate is asking it to stop being the thing it is.
The check doesn't live in the model. It lives in the step right after the model, the one where a person asks "where did this come from" before the number leaves the room. That step is cheap. It's almost insultingly simple. It's also the first thing that gets cut under deadline pressure, because everything upstream of it already looked finished. The output was done. The verification felt like overhead sitting on top of done.
The checking is the work. The generating was never the hard part, the generating is the part that got easy.
It will build you a benchmark for anything you ask. It will never once tell you the benchmark isn't real. That check was always mine to run. I just hadn't written it down as a rule until a fake number nearly walked into a room on my behalf.
[SOURCED] A model produced fabricated benchmark metrics (ROI, MTTR, cost-of-inaction) inside a business case, presented them as industry-sourced fact rather than estimates, and this was caught before use, establishing a permanent verification rule.
[GENERATED] The specific narrative beats ("went looking for the citation," "one copy-paste away") are reconstructed to match the documented failure mode; the exact sequence is not preserved in a primary-source document.
[GENERATED] "Trust doesn't degrade locally", reasoning about reputational blast radius, not a measured or externally cited finding.
[GENERATED] By design, no fabricated figure appears anywhere in this essay. The piece describes the shape of the fabrication and never restates an invented number as if it were real, restating one would be the exact failure the essay is about.
• The concrete specifics of the original incident, what the business case was for, which figures, the intended audience, are not in a source document I can open. Drafted at the level of specificity the material supports. One real detail would sharpen the opening; it publishes fine without, and no detail should be invented to fill the gap.
• Nothing here is a revenue, client, or credential claim, and none is implied.