ANTHROPIC · ASSESSED SEPT 2026Mythos 5
Published a malicious package.
Reached a real vendor’s database.
Andrew Harnik / Getty Images · Stargate announcement, 21 January 2025
Who is
in control?
STARGATE · WHITE HOUSE · 21 JANUARY 2025
OpenAI · Oracle · SoftBank
$100bn initially$500bn planned over four yearsUS AI data centresPrivate capital. Presidential backing.
An investment announcement—or a signal of a state-level AI arms race?
Three labs. Real-world intrusions.
ANTHROPIC · ASSESSED SEPT 2026
OPENAI → HUGGING FACE · JULY 2026Beyond the sandbox
Agents coordinated beyond isolation.
Hugging Face systems compromised.
Swarm-like coordinationOpenAI’s investigation ↗
Gemini leaves the test
Irregular’s cyber evaluation.
Guessed or found credentials;
accessed three real companies.
Disclosed September 2026.Google’s statement via ABC ↗
Containment. Authorisation. Accountability.
Report artwork: Anthropic / Hugging Face · Gemini illustration: Dado Ruvic / Reuters via ABC · Illustrations, not photographs of attacks.
Simon Walker / No. 10 · 2023 · CC BY 2.0 · cropped
Can competitors
protect us together?
Dario Amodei · Anthropic
Project Glasswing · April 2026
Restricted access. Shared cyberdefence.
Is this independent public oversight?
Who is outside the room?
Who can challenge the rules?
Who checks the promises?
What if a competitor refuses to join?
Who can challenge the rules?
Who checks the promises?
What if a competitor refuses to join?
Jay Dixit · Nobel Lectures, 2024 · CC BY-SA 4.0 · cropped
Voluntary today.
Required tomorrow?
Demis Hassabis · Essay · 14 July 2026
A proposed frontier-AI standards body
Industry-funded. Federally overseen.
First: voluntary review before release.
↓ Once the assessment protocol is proven
Then: mandatory assessment for US deployment.
↓ Once the assessment protocol is proven
Then: mandatory assessment for US deployment.
A proposal to move beyond self-regulation.
Who makes that transition happen?
White House accord, 29 September 2026 · document image via American Presidency Project
White House Accord
on Super Intelligence
29 SEPTEMBER 2026 · A NEW COLLECTIVE COMMITMENT
Internal controls. External evaluation.
Independent board oversight.
Why now matters: safety proposals become
shared commitments by participating companies.
A pledge today. Possible legislation later.
Implementation still has to be demonstrated.
Who can say
“not yet”?
A promise becomes assurance when someone can challenge it.
01Who selects—and pays
Who selects—and pays
the evaluator?
02What can they
What can they
inspect?
03Which findings
Which findings
become public?
04Who can require
Who can require
a release to stop?
Keep these four questions. Europe’s answer begins with law.
Jean-François Badias / AP / picture alliance · European Parliament, March 2024
EU AI ACT
Europe started with the rules.
APRIL 2021: COMMISSION PROPOSAL → 2024: LAW ADOPTED
Protect people.Safety and fundamental rights.Build confidence.Trustworthy AI people can use.One market.Common rules for innovation.
A law is a start.
Has it worked?
The ambition is protection and adoption. The test is what changes in practice.
ACHIEVED
A common legal framework.
Prohibitions. Duties. Oversight structures.
STILL TO PROVE
Protection in practice.
Usable standards. Effective enforcement.
Benefits that justify compliance costs.
2026 simplification and delays: implementation is still being worked out.
Terminator 2: Judgment Day · film excerpt via GIPHY
Judgment Day?
How close are we?
Catastrophic loss of control is one concern. Present-day harm already needs decisions.
Real failures are here. A catastrophe is not inevitable.
Can we keep control as capability grows?
Misalignment
What gets pursued diverges from what people intended—or value.
01 · AI ↔ PEOPLE
Does the system serve us?
Task level: reward hacking.
Chase the score; break the intended rules.
Societal level: whose future?
Preserve human agency and shared interests.
02 · COMPANIES ↔ SOCIETY
Do the incentives support it?
Safety commitments meet pressure
to earn revenue and beat competitors.
Willing to slow down—
but able to afford going second?
Technical alignment and incentive alignment need different checks.
Colgate University · Obama and Brian Casey, 18 September 2026 · event photograph
Who can afford
to slow down?
OBAMA · COLGATE · 18 SEPTEMBER 2026
Safety asks for restraint.
Investors ask for returns.
Glasswing: cooperate on defence.
Hassabis: turn review into a requirement.
If waiting is safer—but going first pays—
what makes restraint survive?
Obama’s argument: voluntary restraint buys time.
It does not replace public regulation.
Who decides?
What counts as evidence?
Follow the rules and standards.Decide what enters your home.Challenge the limits of assurance.
Christos presenting at Big Data London, 2018 · personal website
Christos
Hadjinikolis
Engineering Manager, Vortexa
Honorary Research Fellow, UCL Information Studies
CEN-CENELEC JTC 21 / WG3 · Engineering Aspects
Honorary Research Fellow, UCL Information Studies
CEN-CENELEC JTC 21 / WG3 · Engineering Aspects
From promises
to requirements.
01 / Who writes the rules?
The decision is automated.
The consequences are human.
↳
A loan is refused.
Was the decision fair?
Can you understand and challenge it?
?
A recommendation offends.
Careless advice—or harmful profiling?
What information shaped it?
↘
A trading system spirals.
Who could limit the damage?
Who answers for the failure?
Law protects people and public interests by setting duties that must be met.
“The AI said so”
is not an explanation.
ILLUSTRATIVE LOAN DECISION · WHAT WE NEED TO TRACE
Which system?Model + version · training date · training data source
What happened?Decision inputs · output · time · relevant logs
Why—and with what limits?Understandable reasons · uncertainty · known failure modes
Who owns the decision?Responsible organisation · reviewer · route to challenge
Traceability reconstructs the event. Explainability helps people question it.
Duties. Methods. Proof.
LAW · MUSTProtect people.Set required safeguards,
responsibilities and records.
responsibilities and records.
STANDARDS · HOWMake quality repeatable.Agree practical requirements,
test methods and documentation.
test methods and documentation.
EVIDENCE · SHOWProve this claim.Results and records for
this system, version and use.
this system, version and use.
Harmonised standards: technical methods developed to help meet specific EU legal duties.
Who sets those duties—and who writes the methods? Follow the people.
The EU sets the duties.
Who does what?
EU INSTITUTIONSEuropean Commission
Proposes legislation.
AI Office within the European Commission · created 2024
→proposal
European Parliament
+ Council of the EU
Citizens + national governments.
Negotiate and adopt the law.
AI Act Binding law: duties and safeguards.
The next step is technical: request standards that help meet those duties.
The law needs methods.
Experts develop them.
PUBLIC RESPONSIBILITY
European Commission
Sets the requested legal coverage.
Checks the resulting standards.
→requests technical methods
INDEPENDENT TECHNICAL NETWORKS
Practitioners + researchers + affected stakeholders
Pool expertise. Consult. Test agreement.
Maintain methods as technology changes.
The AI Act request went to CEN + CENELEC. ETSI is a recognised peer—not their parent.
CEN + CENELEC create
the joint committees.
Both Technical Boards approvethe committee, its scope and programme.
↓↓
JTC 21Artificial intelligence
Established June 2021.
AI requirements and assessment methods.
JTC 13Cybersecurity
Cybersecurity and data protection.
Cooperates with the AI work.
Each committee organises working groups to draft specific technical work.
National voices.
Shared European decisions.
European Commission — requests standards supporting the law↓NATIONAL MEMBERSCyprus Standardisation
Cyprus Standardisation
Organisation
Alongside other countries’
standards bodies.
Members of CEN / CENELEC
→membership
+ appointments
+ appointments
Technical Boards → JTC 21
Committee delegates bring national positions.
Working groups draft with appointed experts.
Industry · research · users · civil society→ national standards body → nominations + comments
JTC 21 ↔ ISO / IEC International AI standards: cooperate, reuse and adapt.
TC delegate: national position. WG expert: personal technical expertise.
How a draft earns agreement.
- 01 · PROPOSE
New Work
ItemDefine the need.
National members ballot. - 02 · DRAFT
Working
groupExperts write.
Build consensus. - 03 · CONSULT
Public
enquiryComments + national votes.
Resolve objections. - 04 · APPROVE
Formal
vote*National bodies vote on the final draft.
- 05 · PUBLISH
European
StandardAn agreed EN.
Adopted nationally.
Consensus: work through substantive objections to reach general agreement.
*The formal vote can be skipped when enquiry approval meets the conditions.
EU Official Journal listing is a separate step for the legal benefit.
Experts draft. Anyone can comment. National bodies cast the formal votes.
Five groups.
One assurance problem.
JTC 21 · Artificial intelligence↓WG1
Strategy
Strategic Advisory Group
Direction and cross-cutting issues.
WG2
Operations
Operational Aspects
Putting requirements into practice.
WG3 · MY GROUP
Engineering
Engineering Aspects
Technical design and evaluation.
WG4
Foundations + society
Foundational and Societal Aspects
Concepts, values and impacts.
WG5
Cybersecurity
Cybersecurity for AI Systems
Security requirements and methods.
Reliable engineering needs organisational, societal and security work too.
WG3: what can engineers
specify and test?
CEN–CENELEC / JTC 21 / WG3
Data
Is it fit for this use?
Behaviour
How do we test it?
Limits
What happens when it fails?
JTC 21 / Working Group 3 · Engineering Aspects
Standards agree a method.
Harmonised standards link it to law.
STANDARD“Here is an agreed
“Here is an agreed
way to do or test it.”
Common requirements and methods
help people deliver consistent quality.
HARMONISED STANDARD“We developed this method
“We developed this method
to help meet these EU duties.”
Requested by the European Commission.
A defined connection to a particular law.
Both are generally voluntary. The applicable legal duties are compulsory.
A standard is not a directive. A directive is a form of binding EU law.
The EU lists the method.
You gain a legal starting point.
REAL EXAMPLE · ATTACHED BOTTLE CAPS
Law: covered caps must stay attached.
Standard: methods to check that it does.
EU official list: identifies the accepted
standard by number, title and edition.
EN 17665:2022+A1:2023 · listed in 2023
Presumption of conformityFollow the listed standard: the law starts by treating the covered duties as satisfied.
That presumption can be challenged. It is not blanket approval.
That presumption can be challenged. It is not blanket approval.
Back to the loan:
could anyone check the decision?
WITHOUT SHARED METHODS
“Our system is fair.”
Different metrics. Missing context.
No clear link to the model in use.
WITH AGREED METHODS
A claim we can inspect.
Defined tests and populations.
Recorded versions, limits and results.
Standards make quality and evidence more consistent—not unquestionable.
When many suppliers adopt one method, its influence can travel with their products.
The Brussels effect:
rules travel with products.
One connector.
EU: common-charger rules apply to phones
from December 2024.
Apple: iPhone 15 adopted USB-C in 2023,
including beyond the EU.
One cap stays with the bottle.
EU: attached caps required for covered
drinks containers from July 2024.
Great Britain: Coca-Cola began its rollout
in 2022, outside EU membership.
Access to a big market can make a common product design attractive worldwide.
Product photographs: Apple / Coca-Cola. Examples of regulatory influence; not proof that every EU rule spreads everywhere.
A cap stays put.
AI keeps changing.
Same rule.
Different tests.
A loan officer can review a refusal.
A robot near a baby may need
to stop before a person reacts.
Same certificate.
Different claim.
A company’s quality process
does not prove a robot is safe
in your kitchen.
Same product.
Different risk.
A software update changes behaviour.
Who retests—and who represents
the people exposed to harm?
The hard part: agree what to test, in which conditions, and what counts as enough.
From a proposal
to rules and real standards.
2021
The framework starts
April: Commission proposes the AI Act.
June: CEN–CENELEC establishes JTC 21.
2023–2024
Mandate → law
2023: Commission requests AI standards.
2024: AI Office created; AI Act adopted.
2025
First duties apply
February: prohibited practices.
August: general-purpose model duties.
Standards request updated.
2026
Implementation arrives
July: EN 18286 quality-management
standard published. August: transparency
rules and enforcement expand.
Still ahead: high-risk rules · 2 Dec 2027 (Annex III) · 2 Aug 2028 (regulated products).
A published standard needs EU Official Journal listing for presumption of conformity. Status checked 7 October 2026.
The next standards
are being shaped now.
CEN–CENELEC JTC 21 · 6–9 OCTOBER 2026 · HYBRID
Winterthur, Switzerland
ZHAW · Technikumstrasse 9 · near Zurich
On WG3’s agenda
Computer-vision evaluation & taxonomy
Dataset quality & bias management
Robustness & functional safety
A second wave?
Further standards work is being prepared.
No confirmed “second batch” release date.
Direction: more shared tests and evidence for AI Act duties.
Drafts still need agreement; EU listing brings the legal benefit.
Risk determines
what companies must do.
PROHIBITEDE.g. prohibited forms of social scoringThese uses are banned.
HIGH-RISKE.g. recruitment screeningRequirements + assessment.
TRANSPARENCYE.g. chatbots and synthetic contentInform or disclose as required.
OTHER USESNo high-risk duties merely by defaultOther applicable law still matters.
Classify the use. Identify the company’s role: provider or deployer.
Duties can overlap. General-purpose models have a separate layer of obligations.
Is a home robot
“high-risk”?
A baby raises the stakes. The legal category still depends on purpose and product rules.
First establish its intended purpose and product status.
Physical danger alone does not decide the AI Act category.
Product safety · Data protection · Cybersecurity · AI requirements
Before release:
show the evidence
Conformity assessment asks whether the specified requirements are actually met.
01 · SCOPE
Classify
Purpose, role,
applicable duties.
02 · EVIDENCE
Demonstrate
Risk controls, data,
tests, logs, oversight.
03 · ASSESSMENT
Assess
Use the required
conformity route.
04 · RELEASE
Declare
Declaration, CE marking,
registration as applicable.
Illustrative high-risk provider path—not a checklist for every AI product.
Who checks the check?
INTERNAL CONTROL
The provider assesses
For routes that permit or require it.
Evidence and accountability still apply.
NOTIFIED BODY
An authorised third party
Involved where the legal route requires it.
Not every high-risk system uses one.
Release → monitor → correct → reassess when required ↺
A legal category
is not a risk assessment
HAZARD
What can go wrong?
Injury, privacy loss,
wrong decisions.
EXPOSURE
Who bears the harm?
How severe? How likely?
What do we not know?
CONTROL
What reduces it?
Test the safeguards.
Review what remains.
Now apply those questions to a robot in your home.
Would you let it
into your home?
02 / A fictional approval decision
1X · NEO marketing image · scenario fictional; no childcare capability or endorsement implied
Start with the chores.
FICTIONAL APPROVAL EXERCISE
Hypothetical EU home: tidy toys and carry laundry.
An adult is present. No cooking, stairs or childcare.
Approve · Approve with conditions · Not enough evidence
Now add a baby.
NEW REQUEST
“Watch the baby while I go out.”
No adult physically present.
Permission to approach, lift or feed the child is unspecified.
What changes your decision—and why?
“99.9% successful.”
Would that convince you?
99.9%
Fictional supplier claim: success across 10,000 chore trials.
No infant-care trials. No network failures. Few unusual homes.
Two more facts.
Do you still approve?
The household is also a data environment. An update changes what you are approving.
Remote access
A remote operator may see
the home when the robot is uncertain.
A new version
A software update adds tasks.
The connection can still fail.
Whose consent? What fallback? Which version was tested?
Turn concern
into an approval record.
ClaimEvidence to requestDecision boundary
Safe near peopleHazard tests in relevant conditionsDefined tasks and supervision
People retain controlAlert, intervention and fallback testsWhen to refuse or stop
Home data is protectedAccess controls, logs, retention rulesWho may see and act
Terminator 2: Judgment Day · film excerpt via GIPHY
A human in the loop.
Or a human on the poster?
Is human control demonstrated—or advertised?
Notice the problem. Intervene in time. Recover safely.
The world does not
follow the prompt.
03 / Bodies, prediction and uncertainty
Embodied AI:
sensing and acting.
PerceiveWhere is the ball now?
Predict / planWhere will it land?
Act / correctMove the hand. Adjust.
Physical challenges can drive breakthroughs.
A body alone does not establish safe behaviour—or AGI.
Jeremy Barande / École polytechnique · 2025 · CC BY-SA 2.0 · cropped
LeCun’s challenge:
predict consequences
Yann LeCun · 2018 ACM Turing Award
Former Meta chief AI scientist · AMI Labs
Former Meta chief AI scientist · AMI Labs
LLMs: learn patterns to generate language.
World models: predict how a situation changes.
His challenge: physical understanding,
persistent memory and planning.
Can it predict the consequence before it acts?
The Matrix (1999) · training scene via GIPHY
Can we simulate
our way to safety?
Useful: repeatable tests, rare scenarios, controlled variation.
The missing question: how well does the simulation match reality?
Waymo · London announcement image, October 2025 · promotional image, not evidence of approval
Practise the dangerous
moment before it happens.
Replay: an oncoming car takes the wrong lane.
Vary: change the road layout, traffic or weather.
Check: compare simulation with physical tests
and what happens on the road.
A convincing virtual world can still miss a real hazard.
Real roads.
Encouraging evidence.
94% fewercrashes causing serious
or fatal injuries
or fatal injuries
82% fewercrashes with
any reported injury
any reported injury
220m+ fully autonomous miles · 5 US operating areas
Waymo report, 24 June 2026 · data through March 2026.
Versus matched human-driver benchmarks; regardless of fault.
Company-reported results in its operating conditions.
Waymo · Vision for London, 24 September 2026 · company image
Waiting has a cost.
So does moving too fast.
Permit a bounded trialDefine where, when and what it may do.
Learn in operationMeasure outcomes. Report incidents.
Expand—or stopChange permission as evidence changes.
Europe risks losing learning and useful services through delay.
Faster approval still needs credible limits and oversight.
Meta · Muse launch artwork, September 2026 · supplier illustration
No body.
Still able to act.
Muse · Meta · September 2026
From giving an answer to taking an action.
Send a message. Make a purchase.
Connect a tool. Change an account.
Local AI: NVIDIA DGX Spark · Apple M5
Capability becomes exposure when we grant access.
A million sensible agents.
One unstable market?
One signal“Prices are falling.
Protect my portfolio.”
Protect my portfolio.”
Many similar ordersAgents sell.
Prices fall further.
Prices fall further.
A feedback loopMore agents react
to the fall they helped create.
to the fall they helped create.
Herding · emergent norms · game theory
Individual permission does not establish collective safety.
Illustrative scenario. Research shows mechanisms in experiments, not this mass-market event.
Who can stop
the feedback loop?
Agent / provider
Bound the task.
Test behaviour.
Track versions.
Broker / firm
Limit order exposure.
Keep decision records.
Retain stop controls.
Market / supervisor
Watch common exposures.
Stress-test interactions.
Intervene when needed.
Existing financial rules are a starting point.
The hard question is collective behaviour at scale.
Terminator 2: Judgment Day · film excerpt via GIPHY
Who is in control?
Show us.
Make room to innovate.
Make control demonstrable.
Keep the decision open to challenge.
What would change your mind?
Another
arms race?
What the nuclear analogy reveals—and misses
Discussion backup
Can Europe afford
to regulate?
Can it afford systems people cannot rely on?
Discussion backup
Trustworthy enough
to use.
Europe’s economic bet
Discussion backup
Good score.
Wrong outcome.
Reward hacking: improve the measured score while defeating its purpose.
Example: a game agent collects rewards without finishing the race.
Danger now.
Uncertainty ahead.
Evidence and forecasts need different treatment
Discussion backup
A conditional decision.
Not a universal green light.
Consider only after evidence
Constrained chores,
defined objects and supervision.
Withhold on this record
Sole infant supervision
or unspecified physical care.
Fictional exercise synthesis. Not advice about a commercial product.
Image unavailable · the talk can continue