How the Errors Accumulated
The chain the report reconstructs is shocking: the AI system gave a mistaken determination in fraud detection, the human review that should have been the backstop failed collectively under the inertia of "even the machine says so," and so an innocent person was jailed for months before being cleared. The core problem was never that algorithms can err—any system can—but that the whole process treated a probabilistic algorithmic output as a deterministic fact, removing the human's responsibility to judge. A machine's error was amplified by human blind faith into a wrongful imprisonment.
Guardrails for Automated Decisions
Cases like this play out repeatedly in benefits verification, credit, and criminal justice, and the victims are often the vulnerable who have the fewest resources to prove their own innocence. The workable constraints aren't mysterious: an algorithmic determination can only serve as a lead, not a verdict; key decisions must be backed by independent evidence; operators must be trained to understand the algorithm's limits; and those affected must have a channel for rapid appeal and correction. Every report like this is material for legislation and institutions to make up their lessons. When an organization outsources judgment to AI, it must at the same time retain a human to be responsible for errors; otherwise a system error means no one is responsible.
via: Hacker News