Flock's AI Misread License Plates 71% of the Time. Officers Verified Anyway — Until the Alerts Piled Up.

Forrester principal analyst Alla Valente uses a documented failure of Flock Safety’s AI license-plate-reader system in Roseville, California as a case study in third-party AI governance risk: over two years, the system misread plates in 71% of alerts sent to police — incorrectly flagging vehicles as stolen or tied to felonies — despite Flock’s vendor claim of 96% accuracy under optimal conditions. Officers had a policy requiring human verification before acting on an alert, but abandoned it in practice as alert volume grew; one dispatcher is quoted saying “it’s easier at this point we have it memorized” rather than escalating discrepancies. Valente’s framing is blunt: “operating reality always wins over vendor promises.” This wasn’t a technology failure, it was a governance failure — a control that existed on paper but not in the daily workflow.

That gap between a control’s existence and its actual use shows up in B2B software too. Salesforce data scientists have documented an internal case where an AI agent confidently returned two different, both-wrong figures for the same metric — 50,000 and 75,000 leads — while the certified dashboard showed the correct number: 100,000. The root cause: the agent queried raw, uncertified data instead of the organization’s agreed-upon definitions. Their fix mirrors Valente’s recommendations almost exactly — route agents through certified datasets and machine-readable business logic, not whatever raw source happens to be fastest to query.

Anthropic’s own agentic-AI governance framework gets at the same failure mode from the deployment side. Deputy CISO Jason Clinton’s guide for security leaders asks whether agent actions can even be distinguished from user actions in the logs — a question Roseville’s police department couldn’t answer once verification became rubber-stamping. Clinton cites a Ponemon Institute figure that organizations average 67 days just to contain an insider incident; an unverified AI misread hiding in a “verified” workflow is a version of the same blind spot.

Valente’s six recommendations reduce to one discipline: validate a vendor’s accuracy claims against your own production data before scaling, and audit — genuinely audit — whether your team’s escalation policy is actually being followed. A control nobody checks isn’t a control.