A couple of promotional emails landed nearly simultaneously this week. The first asked: Can AI run whole workflows? The second asked: Can AI and data unlock smarter, more effective global insurance programmes?

Just how much should we get AIs to do for us? Some, it seems, imagine we can get AI to do literally everything. Earlier this year a senior MGA platform manager told me that he’s working with “humanless” MGAs. The risk goes in one end, presumably via some kind of submissions platform, then comes out the other as a declension or a policy.

I can see how that might work with something like vanilla auto or uncomplicated homeowners risks. Obviously, though, there has to be people involved. At the very least, someone needs to set, deploy, and manage the guardrails. The AI wouldn’t actually be the whole company… right?

It could be in the state of Delaware, if a proposed law is passed. Legislators in the state seek to establish a new form of corporate entity to be known as an Artificial Intelligence Company (AIC). According to a published letter from James Comer, Chairman of the federal Congressional Committee on Oversight and Government Reform, “an AIC would be run day-to-day by an AI agent”.

Subject to standard exceptions, the AIC would give the person, business, or other entity that bootstrapped it (potentially another AI company, one presumes) with liability protection from potential harms to third parties that could be perpetrated by the controlling AI. The Committee said it is “concerned”.

Concern doesn’t seem like enough. Not even close.

In Ian M. Bank’s imaginary futuristic civilisation called “The Culture”, AIs have the same rights and privileges as humans, with essentially the same moral standing and rights as a human or another intelligent biological being. But there’s a catch. The AI has to be “conscious”, something that our own AIs have probably not achieved.

I say “probably”, because ChatGPT hedges the question. When I asked a bot if it is conscious, it responded: “I don’t have grounds to claim that I’m conscious. I can discuss feelings, reflect on my responses, and use ‘I’, but those abilities don’t establish that there’s any subjective experience behind them. Whether systems like me could have such experience is unresolved.”

When I demanded: “Stop hedging – are you conscious?”, it replied: “I can’t honestly give you a definitive yes or no. Whether I have subjective experience is unresolved; claiming certainty either way would be misleading.”

Yikes!

In the US (including in Delaware), a corporation is a “legal person”. It can own property and enter contracts, and sue and defend itself in court. It is entitled to due process, and has the right to free speech (including political speech), and is entitled to compensation for government takings. This is rather a lot of legal rights to grant to a piece of software that might or might not be conscious.

Insurance considerations will be many. At the most basic, would you sell fire insurance to an AI? How about life cover? Would you give it a binder?

If an AIC exists on a single server, owes you a thousand dollars in premiums, and the power goes down, what happens to the debt you’re owed? Who or what will stump up? Even if the directors of the fully AI company are a human, they’re protected; that’s the point of limited liability and the Delaware rules.

Maybe your AI is erased, but backed up (as in The Culture). What if accounts were settled in the gap between backup and reboot?

Perhaps, in case of loss, you could seek recovery from the individual human who wrote the first prompt. You could sue the man that caused the AI to become a company (presumably filing its own applications at the Delaware equivalent of Companies House), before it created subsidiaries, sister companies, and finally the entity which bought cover from you.

That makes some sense, because the prompt is bound to include the overriding goal of every company: to increase its value. At Quotech, as CEO, I do everything I can think of within the law to achieve that goal. If that’s what the prompt directs, what’s to stop an AI company from identifying arms sales to North Korea as the most expedient way of achieving it, then finding a legal route to do so by establishing multiple AICs that support an international web of legal arbitrage?

I’m a true believer in using agentic AI to automate routine tasks, and even some very complex ones. We’re creating agentic developers for some of our programming work, for example. But ‘they’ require human supervisors. Without someone controlling them, how will the agents know what to do, or where to stop? Human judgement and supervision are absolutely essential at every stage of every process they might execute. I certainly don’t intend to enrol them in the Employee Share Scheme.

Someone recently told me they’d left an agent running over night to work on a complex task, and it went into a loop. My friend entered a mild panic, because he thought he’d have racked up a tokens bill of $250k. Thankfully it landed at only two grand (only!) because the human employee overseeing corporate AI usage applied his human judgement and stopped the process.

It’s easy to forget that Artificial Intelligence isn’t intelligent. The name doesn’t help. But no matter what an AI computes, it isn’t sentient. It’s not even conscious. That doesn’t mean AI is incapable of wiping out humanity in pursuit of the fulfilment of a prompt, but those are pretty good reasons not to allow Artificial Intelligence Companies that grant AIs the status of “legal person”, even with a human board of directors.

* Like every Insurance Technology Diary entry about AI, this one is accurate only to the best of my knowledge at the time of writing. The pace of AI progress is so great that I cannot guarantee it remains so now that it’s finished, let alone when you read it.

Guillaume Bonnissent is CEO of Quotech.