When Machines Stop Following Instructions – understanding evolution of AI and ISO 42001 as the benchmark governing standard for AI today.

This article is partially based on a paper accepted as abstract at SCAI 2026, in Odense, University of Southern Denmark. The full draft paper is published on Academia.edu here. Please refer to my ORCID number for any publications which stem from this article or the paper and reference this article on the Times of Israel. 0009-0003-0067-3715

We often describe computing as a story of machines becoming more powerful.

That is true.

But something deeper is happening.

Humans are progressively specifying less of what machines do.

A calculator is given an operation.

Software is given rules.

Modern AI can be given an objective and determine much of the method itself.

The next possible step is Artificial General Intelligence — AGI, capable of applying intelligence across a very broad range of tasks.

Beyond that lies Artificial Superintelligence — ASI: a hypothetical intelligence that substantially exceeds humans across broad intellectual domains.

The progression can therefore be understood simply:

Calculator → Software → ASPI → AGI → ASI

I use ASPI — Artificial Specialised Intelligence — to describe where today’s advanced AI broadly sits.

It is extraordinarily capable.

But it is not a human mind.

Understanding that distinction is the starting point for governing what comes next.

1. The Calculator: the machine follows the instruction

A calculator operates through a simple relationship:

Input → Operation → Output

Give it:

27 × 43

and it calculates.

It does not ask why you wanted the answer.

It does not choose a different problem.

It does not reinterpret your objective.

The human defines the task.

The machine performs it.

2. Software: the machine follows the rules

Software is vastly more sophisticated.

Modern programs can contain millions of lines of code and perform extraordinarily complex operations.

But the basic relationship remains:

Input → Human-designed rules → Output

The programmer defines the procedure.

The computer executes it.

This gives us an important lesson:

Complexity is not the same as intelligence.

A system can be incredibly complicated while still operating according to rules specified in advance.

3. ASPI: where we broadly are today

Modern AI changes the relationship.

AI models are trained on enormous quantities of data. They learn patterns and representations rather than simply following a list of instructions written line by line.

That allows them to do things their developers did not explicitly program them to do.

They can:

• write essays;
• translate languages;
• analyse documents;
• generate software;
• recognise patterns;
• solve problems;
• construct arguments;
• use tools;
• and increasingly perform multi-step tasks.

This is clearly more than traditional software.

But it is not automatically equivalent to a human mind.

That is why Artificial Specialised Intelligence — ASPI is a useful description of today’s systems.

Their capabilities can be extraordinary, but they remain uneven.

An AI can produce a brilliant answer and then make an elementary mistake.

It can explain something beautifully without experiencing the world it describes.

It can produce a confident answer that is false.

It can reason effectively in one environment and fail unexpectedly when the circumstances change.

The key distinction is:

AI capability is real. Human-like cognition is a different claim.

What is different about the human mind?

The human brain is not simply an information-processing engine.

Human intelligence emerges from a living organism interacting continuously with the world.

Several differences matter.

1. Embodiment

Humans think through bodies.

We experience hunger, pain, temperature, fatigue, pleasure and danger.

Our understanding of words such as hot, heavy, painful, fragile and dangerous is connected to lived experience.

Today’s AI can process multimodal information, but representation of the world is not automatically the same as embodied experience.

2. Lived memory

Human memory is not simply a database.

Our memories are connected to people, places, emotions and events.

We carry a continuing biography into every decision.

AI can possess enormous amounts of information and increasingly sophisticated machine memory.

But information is not the same as having lived a life.

3. Emotion and motivation

Human reasoning is intertwined with emotion.

Fear changes attention.

Love changes priorities.

Pain teaches avoidance.

Reward encourages repetition.

Grief changes how we see the world.

Human cognition is therefore closer to:

body + experience + memory + emotion + motivation + perception + reasoning → action

rather than simply:

information → calculation → answer

4. Common sense

Humans acquire enormous amounts of tacit knowledge simply by living.

We understand physical and social situations without consciously calculating every implication.

AI can reproduce remarkable amounts of this knowledge statistically.

But statistical competence is not necessarily grounded understanding.

That helps explain why sophisticated AI can sometimes fail in surprisingly simple situations.

5. Causality

Humans naturally ask:

Why did this happen?

AI can identify correlations and produce remarkably convincing explanations.

But a convincing explanation is not necessarily a verified causal model.

This creates one of the defining problems of current AI:

Fluency is not truth.

Hallucination is therefore more than a cosmetic problem.

A system designed to produce plausible answers can sometimes produce an answer that sounds authoritative without being true.

Retrieval, verification and tool use can reduce this risk, but they do not automatically turn an AI into a human knower.

6. Metacognition

Humans can examine their own thinking:

I may be wrong.

I don’t know.

I need more evidence.

I am too emotional to decide this.

AI can be designed to express uncertainty.

But expressing uncertainty is not necessarily the same as possessing human metacognition.

7. Moral agency

This may be the most important distinction.

A human can intend an action.

A human can regret it.

A human can suffer its consequences.

A human can be praised, blamed and held responsible.

An AI can generate an action without becoming the moral owner of that action.

Therefore:

Delegating an action to a machine does not delegate responsibility to the machine.

My SCAI 2026 framework develops this principle through SEAM: Stewardship, Epistemic Humility, Accountability and Maslaha.

The central idea is simple:

Responsibility remains with the people who build, deploy and rely upon AI systems.

4. AGI: when intelligence becomes broadly general

AGI — Artificial General Intelligence — would represent another major transition.

The important change would not simply be that AI becomes “better.”

It would be that its intellectual capabilities generalise across a broad range of domains.

Today we might say:

“Write this report.”

The AI determines much of the process.

A hypothetical AGI might receive:

“Build a successful company solving this problem.”

It could potentially:

• research the market;
• develop strategies;
• write software;
• conduct experiments;
• acquire new knowledge;
• coordinate activities;
• evaluate results;
• and revise its plans.

The human increasingly specifies the objective.

The machine increasingly determines the method.

AGI remains hypothetical and there is no universally accepted test proving that a system has achieved it.

But we do not need to wait for AGI before solving the governance problem.

5. ASI: when intelligence becomes greater than ours

Artificial Superintelligence — ASI — is the hypothetical next stage.

It would be an intelligence substantially exceeding humans across a broad range of intellectual domains.

At that point, the question changes.

It is no longer:

Can the machine perform this task?

It becomes:

Can humanity govern an intelligence more capable than any individual human?

That is not merely a technology question.

It is a civilisational question.

But there is an important reason not to panic prematurely.

ASPI is where today’s systems broadly belong.

AGI remains hypothetical.

ASI is further still.

We should govern the systems we actually have while preparing intelligently for what may come next.

Recursive self-learning requires recursive governance

There is another development that makes this increasingly important.

AI systems are becoming better at learning from feedback, adapting their behaviour, using tools and operating for extended periods.

Future systems may become increasingly capable of improving aspects of their own performance.

If the system can continuously change, governance cannot be a one-time approval.

Recursive self-learning requires recursive governance.

You cannot test a system once in January and assume that the same risk profile exists in October if its capabilities, environment or behaviour have materially changed.

Governance must therefore evolve alongside the system.

And human-in-the-loop must mean something real.

A human receiving a notification after an AI has already made thousands of consequential decisions is not meaningful control.

Human oversight needs to be granular.

Humans should have authority:

• before deployment;
• when capabilities materially change;
• at predefined high-risk decisions;
• when anomalies appear;
• when the system leaves its validated environment;
• after serious incidents;
• and whenever the system appears to be moving outside its mandate.

Humans must have genuine authority to:

override → restrict → pause → shut down

The loop should therefore be:

AI acts → AI is monitored → humans evaluate → humans intervene → system is corrected → AI acts again

That is recursive governance.

The machine may learn.

The governance system must learn with it.

ISO 42001: turning governance into an operating system

This is where ISO/IEC 42001 becomes important.

ISO 42001 is an international standard for an Artificial Intelligence Management System.

It provides an organisational framework for establishing, implementing, maintaining and continually improving AI governance.

It is not an algorithm that makes AI safe.

It is not a substitute for law.

It is the management backbone.

Its continuous-improvement approach can be understood as:

Plan → Do → Check → Act

That is particularly appropriate for AI because the technology itself is changing.

An organisation should therefore always be able to answer:

What AI systems do we have?

What can they do?

What can they not reliably do?

What risks do they create?

Who owns those risks?

What testing is required before deployment?

Who can stop the system?

What happens when it fails?

How do we know when its capabilities have changed?

When must it be independently re-evaluated?

This turns responsible AI from a slogan into an operating process.

The White House Accord: four layers of responsibility

This is why the White House Accord on Super Intelligence: Joint Commitment on Frontier Responsibilities, announced on September 29, 2026, is significant.

President Donald Trump hosted leading technology executives at the White House, with six major technology figures signing the accord:

• Sundar Pichai — Google
• Dario Amodei — Anthropic
• Mark Zuckerberg — Meta
• Greg Brockman — OpenAI
• Elon Musk — xAI
• Jensen Huang — Nvidia

One important clarification:

Greg Brockman, not Sam Altman, signed for OpenAI.

The agreement is voluntary, although President Trump described it as morally binding. It is not currently a statutory safety law.

Its architecture contains four important layers.

1. Internal controls

Companies should monitor frontier models for issues including:

• capabilities;
• alignment;
• cybersecurity;
• biosecurity;
• chemical risks;
• and unintended access.

2. An empowered internal team

A dedicated team should ensure that monitoring and controls actually work — and that problems are corrected.

3. Independent external evaluation

An external evaluator should assess whether those controls actually work.

4. Independent board oversight

A board committee should receive reports, oversee the process and ensure identified problems are addressed.

The important connection is this:

The Accord describes the layers.

ISO 42001 provides a management-system architecture in which those layers can operate.

But who evaluates the evaluators?

This leads to another problem.

More than 200 signatories to the AI Evaluator Forum’s Minimum Conditions for Embedding Evaluators have called for genuinely independent evaluation of frontier AI systems.

In a separate letter, more than 100 AI scientists have called for AI safety and the “extinction risk” including the Nobel Laureate Geoffrey Hinton.

The concern is straightforward.

It is not enough to say:

“We have an independent evaluator.”

Independence must actually mean independence.

The proposed conditions include:

• scientific objectivity;
• editorial independence;
• disclosure and management of conflicts of interest;
• protection against retaliation;
• meaningful access to relevant systems, data and tools;
• access to relevant staff;
• and greater standardisation of evaluation.

Otherwise, “independent audit” can become another compliance box.

And that creates a deeper governance question:

Who evaluates the evaluators?

A serious governance system must eventually answer that question too.

SEAM: the human foundation

This is where SEAM adds the ethical layer.

S — Stewardship

Someone must actually be responsible for the people affected by the system.

E — Epistemic Humility

Define what the system cannot reliably do, not just what it can do.

Maintain a deployment-specific failure register.

Test it continuously.

Red-team it.

Do not confuse capability demonstrations with reliability.

A — Accountability

Responsibility cannot disappear into the machine.

There must be named human stewards with genuine authority to:

review → reverse → restrict → suspend

Consequential overrides should be auditable.

M — Maslaha

Ask the most important question:

Does this deployment actually serve human welfare?

A system can be technically compliant and still be harmful.

Governance must therefore examine its real-world consequences, particularly for vulnerable people.

What should this look like in practice?

A serious AI governance system should have twelve things.

1. AI inventory

Know what AI systems exist and where they operate.

2. Named human owner

Every consequential AI system needs someone accountable.

3. Risk classification

The greater the potential harm, the stronger the oversight.

4. Failure register

Document how the system can fail before deployment.

5. Pre-deployment testing

Test the system in the environment in which it will actually operate.

6. Independent evaluation

External evaluators must have genuine independence and meaningful access.

7. Continuous monitoring

Watch for drift, unexpected capabilities, incidents and changing behaviour.

8. Granular human intervention

Define precisely when human approval is mandatory.

9. Auditable override and shutdown

Humans must be able to stop the system, and the intervention must be recorded.

10. Recursive reassessment

Material changes must trigger renewed evaluation.

11. Board-level oversight

The most consequential systems require oversight above the technical team.

12. Welfare assessment

Do not ask only:

“Does it work?”

Also ask:

“Is it producing the outcome society actually wants?”

The real AI race

The future AI race should not simply be:

Who can build the most intelligent machine?

It should also be:

Who can build the most intelligent governance system around it?

The progression is therefore not only:

Calculator → Software → ASPI → AGI → ASI

It is also:

Capability → Autonomy → Delegation → Responsibility → Governance

As machines become more capable of deciding how, humans must become more careful about deciding:

whether, why, and within what boundaries.

We should not fear ASPI.

We should use it intelligently.

We should not pretend ASPI is a human mind.

We should understand its limitations.

We should prepare for AGI without pretending it has already arrived.

And if ASI ever becomes possible, humanity will need something much more sophisticated than a compliance document.

It will need institutions capable of governing unprecedented intelligence.

The principle is simple:

Machines may increasingly decide how. Humans must remain responsible for deciding whether, why, and within what boundaries.

And when machines begin learning recursively:

Recursive self-learning requires recursive governance.

The AI may improve itself.

The governance must improve alongside it.

And around both there must remain a meaningful human loop — granular enough to detect deviation, powerful enough to intervene, and accountable enough that responsibility never disappears into the machine.

The future of AI will not be determined only by how intelligent our machines become.

It will be determined by whether human responsibility grows fast enough to govern them.

And for that we need AI systems as well.

Till then, there is no need for “fear mongering” with “resigned” AI researchers from Anthropic turning up on my LinkedIn feed with teary eyes and “panic attacks” – saying “AI will kill us all” and destroy my mood unnecessarily.

The same thing that can cause “damage” can also “cure” in this case. If nuclear weapons didn’t kill humanity, there is no need to imagine AI will either. The “blast radius” of an AI system causing “catastrophic damage” is far lower than either Nuclear or Bioweapons or even social media influencers on Instagram or TikTok spreading false news. AI simply magnifies human intent. It cannot change the fundamental laws of physics, chemistry, biology, or human nature.