The Machine That Never Says ‘I Don’t Know’ – and the Risk of Blackouts

India is about to put a brilliant, fluent and occasionally fabricating assistant next to the world’s largest synchronised grid. The question is who gets the final say.

On 30 July 2012, India’s northern grid collapsed, and the next day a larger collapse left hundreds of millions of people without power. Investigators blamed overdrawing, weak coordination and human fatigue, in a system where operators made fast calls under pressure. That is the room into which we are now inviting large language models (LLMs).

The pitch is seductive. A recent review (October 05, 2026) in Energy Nexus surveys how LLMs are entering power systems for forecasting, monitoring, diagnostics, predictive maintenance and decision support. Picture the exhausted night-shift operator who can ask a question in plain language and get a crisp answer in seconds. It is no wonder utilities are tempted.

But the same review lists the catch, and it should make any engineer sit up: hallucination, bias, explainability, cybersecurity, accountability and human oversight all become live concerns once these tools touch safety-critical infrastructure.

Note the items on that list. A relay that fails announces itself with a bang. A language model fails in a calm, well-formatted paragraph with confident citations. It has no instinct for saying “I don’t know,” and it is trained to sound sure. In a control room, sounding sure is the most dangerous trait a tool can have.

India’s special problem: we are digitising faster than we are securing

One might hope the underlying infrastructure is hardened enough to absorb the experiment. It is improving, but the picture is uneven.

Take smart meters, the nervous system of tomorrow’s distribution network. On October 19, 2026, India’s Central Electricity Authority (CEA) released a unified framework, a five-year roadmap for discoms to adopt smart meters, GIS and AI-based systems, targeting to cut losses, improve reliability and strengthen cyber security. Industry analysts warn that many deployed meters rely on unencrypted RF (radio frequency) or Zigbee communication, leaving data open to interception or manipulation, and that buying from multiple vendors makes it hard to guarantee hardware and firmware integrity. Now imagine an LLM tasked with summarising meter anomalies or flagging theft. If its inputs can be tampered with, its fluency turns a corrupted data stream into an authoritative-sounding report. Garbage in, eloquence out.

Add a threat that barely existed a few years ago, prompt injection. An assistant that reads logs, tickets or emails can be steered by instructions hidden in the text it reads. This is my own extrapolation rather than a finding from the review, but it follows directly from how these systems work: an assistant wired into operational data inherits every weakness of the pipes feeding it.

The regulatory patchwork has a hole in the middle

To its credit, India has moved. The Ministry of Power created CSIRT-Power in April 2023 as an extension of CERT-In, and the CEA has now notified Cyber Security in Power Sector Regulations, 2026, replacing voluntary compliance with a statutory system covering audits, incident reporting, exercises and response protocols. Reports describe the strictest controls for operational technology, including isolation, encryption and India-based storage. The data requirement extends to third-party cloud service providers.

Notice what that implies. Many of the most capable LLMs live in somebody else’s cloud, often abroad. A tool that is fine for summarising a tariff order may be a compliance breach the moment it ingests SCADA data. Utilities that rush to plug a commercial chatbot into operations may discover they have bought an audit finding.

Then there is the other half of the regulatory picture. India has chosen not to write a standalone AI law. The guidelines the Ministry of Electronics and Information Technology (MeitY) released on 5 November 2025 lean on existing law and enforcement by sector regulators, not a new AI statute. Their principles are sensible: human-centred design and oversight, accountability across the AI value chain, and explanations that users and regulators can understand. Deployers are told to check fitness for purpose and add human-oversight checks where required.

“Where required” is carrying a lot of weight in that sentence. Required by whom? In a control room, who counts as the “human in control”? The national framework hands that question to sector regulators, which means the power sector’s regulators must now answer it, and soon.

Make physics the referee

So what should a sensible utility do? The Energy Nexus review proposes a layered framework combining data and model governance, deterministic engineering verification, cybersecurity safeguards, human authorization, auditability and regulatory oversight.

If I could underline only one phrase, it would be deterministic engineering verification. Put simply: the model may draft, but a conventional, testable engine must check before anything touches the grid. Power-flow solvers and protection-coordination studies do not have moods. They do not hallucinate. Let the LLM propose a switching sequence, and let validated physics veto it.

Equally crucial is the second human safeguard, which people often get wrong. A person clicking “approve” on a screen full of polished AI output is not oversight, it is a rubber stamp. Real authorisation means the operator can see what the suggestion rested on, and refusing it is easy and carries no penalty. We know from aviation and medicine that humans routinely over-trust confident automation. Designing against that tendency is an engineering task, not an afterthought.

A ladder, not a ban

I am not arguing for keeping LLMs out. That would waste a real opportunity, and the review itself is balanced about the potential. My proposal, which is my own synthesis and not the paper’s, is a three-rung ladder:

Green rung: drafting regulatory filings, searching standards, summarising orders, helping young engineers navigate manuals. Mistakes are cheap and caught by ordinary review. Deploy freely, with no operational data.

Amber rung: forecast interpretation, maintenance prioritisation, anomaly triage. Valuable but consequential, so require logging, cross-checks against conventional models and clear escalation.

Red rung: switching, dispatch commands, protection settings (also see, the NLDC Operating Procedure 2026). No autonomous LLM role, ever. At most it may offer options that independent deterministic checks and qualified humans validate.

This lets India harvest the efficiency where errors are survivable and refuse to gamble where they are not.

The quieter harm: who gets answered well?

Not every risk is a blackout. Consider the consumer-facing chatbot that will inevitably appear on discom websites. India’s customers speak dozens of languages. A bot that resolves complaints smoothly in English but stumbles in Odia, Tamil or Bhojpuri will quietly deliver better service to the already advantaged. The review’s concern with bias is not abstract here: fairness means equal quality of resolution across languages, incomes and the urban-rural divide. A utility should measure that before launch rather than discover it through complaints.

And when something goes wrong, who is accountable: the vendor, the integrator, the discom or the operator? If procurement contracts do not answer that now, litigation will answer it later.

The test of a mature grid

The fluent intern is coming to the control room whether we plan for it or not. The Indian power sector has a short window, as AI adoption accelerates, to decide what kind of colleague it will be: a tool that is verified, logged and answerable, or an oracle nobody dares question.

Engineers have a very old rule: trust, but verify. For a technology engineered to sound trustworthy, the second half of that rule is the one that keeps the lights on.