The Question Nobody’s Asking About AI

Ford Motor Company spent the last few years cutting experienced engineers and leaning on artificial intelligence to catch quality problems before cars left the factory. Last week the company admitted what that actually produced: recalls, dependability ratings sliding, and a scramble to rehire the very engineers it had let go. Their own vice president of vehicle hardware engineering put it plainly — they’d assumed that introducing AI and feeding it the design requirements would produce a high-quality product. It didn’t. The tool worked fine. What was missing was the judgment of the people who used to catch what the tool couldn’t.

Ford isn’t alone. Commonwealth Bank of Australia replaced human customer service reps with an AI voice bot, then reversed course when complaints spiked. IBM’s AI-driven HR system handled 94% of routine requests fine — and had no answer for the 6% that involved a judgment call, so IBM announced it’s tripling entry-level hiring to rebuild the human layer it had cut. A recent industry survey found 29% of companies that laid off workers because of AI have already rehired for the same jobs. Nearly a third.

None of this means AI doesn’t work. It means something more specific, and more useful: every one of these failures happened at the exact same seam — the point where a machine’s job ends and a human being’s judgment is supposed to begin. And in every case, nobody had actually decided, in advance, where that seam was.

That’s the real question about AI, and almost nobody in public life is asking it. The debate we’re having is “is AI good or is AI dangerous” — a binary that generates a lot of heat and almost no light. The question that actually matters is: what are the rules, who’s inside the loop when it counts, and does the boundary hold when the pressure is on. The same tool, pointed in two directions

Here’s an example that shows exactly how much the answer depends on the boundary and not the technology. Palantir’s data-fusion platform, Foundry, is already deployed across the Social Security Administration, the IRS, Homeland Security, Health and Human Services, and the VA. In Nevada, a version of this kind of system is being used to catch financial fraud committed against people under court-appointed guardianship — cross-referencing bank records, flagging anomalies, alerting judges. That’s a real, if new, use of AI to protect some of the most vulnerable people in the country: elderly Americans who no longer have the capacity to manage their own affairs and, often, no family left to watch over them.

At the very same time, a Palantir-built tool called ELITE has reportedly been used by ICE to pull Medicaid data — health and benefits records — and generate deportation targets with a location “confidence score.” Same underlying technology. Same category of government data. One use protects the most defenseless people in the system. The other repurposes health records meant to keep people alive into a tool for finding and removing them.

Nothing about the software changed between those two uses. What changed is who was holding it, and what they were told the goal was. That’s not a technology problem. That’s a governance problem wearing a technology costume.

The fraud nobody’s catching

I’ve spent time this year researching how the country actually handles guardianship fraud —cases where someone gets legal authority over an incapacitated elderly person and uses it to steal from them, neglect them, or isolate them from anyone who might notice. The federal government’s own watchdog, the GAO, has been writing reports on this since 2004. Their finding, restated almost word for word in 2006, 2011, and 2017: state courts and federal agencies like Social Security still don’t systematically tell each other when they discover someone is being abused by their own guardian.

Congress has introduced bills to fix this — the Guardianship Accountability Act, more than once — and none of them have passed. Roughly 1.3 million adults are currently under guardianship in this country, controlling an estimated $50 billion in assets, and there is still no comprehensive way to track whether the people appointed to protect them are doing so.

When these cases do get caught, it’s almost never a system that catches them. It’s a journalist. A whistleblower. A dedicated county auditor, in the rare county that has one. One documented case in Washington State ran for eight years and cost victims close to $280,000 before Social Security’s Inspector General and local police finally closed in. New York City has 157 examiners covering more than 17,000 guardianship cases, with about a dozen judges to check their work. That’s not a system with gaps. That’s mostly gap.

This is a problem AI could genuinely help solve — the same kind of pattern-detection work Nevada is already piloting for financial fraud could, in principle, be extended to catch someone using a legitimate court appointment as cover to isolate and exploit an elderly person nobody else is watching. But it would have to be built by someone whose only goal is protecting that person, running on infrastructure that isn’t simultaneously being used for something that erodes trust in the systems the most vulnerable people depend on to keep them safe.

Where the rules actually have to live

I design medical technology for people the healthcare system routinely underserves — patients managing complex conditions with no family close by, no one to advocate for them, no safety net beyond whatever institution happens to notice they’re struggling. Every serious design decision I make comes down to the same question this whole piece is about: at what specific point does a machine stop and a person with real authority and real accountability have to look at what’s happening and decide.

That answer can’t be an afterthought bolted on after something goes wrong. Ford found that out expensively, and quality problems in a car are recoverable in a way that a missed medical emergency, or a stolen life savings, is not. The boundary has to be built in from the start, by people who’ve actually sat down and asked where a machine’s confidence should stop being trusted on its own — not because AI is a threat to be feared, and not because it’s an unqualified upgrade to be rushed into everything, but because it’s a tool. A powerful one. And every powerful tool in human history has needed someone to decide, deliberately, where its authority ends and a person’s begins.

We are not having that conversation in public right now. We’re having a fight about whether to be excited or afraid. Neither side of that fight is asking who’s holding the wheel, or what happens the day nobody checks.

That’s the conversation worth having instead.