This is possibly not the gotcha people want it to be.
LLMs have no morality. That's not a value judgment. That's pointing out not to personify a non-person. Like Bryan Cantrill's lawnmower[1], writ large. They're amoral in the most literal sense. They find and repeat patterns, and their outputs are as good or evil as the patterns in their training data.
Or maybe it is.
This means that if someone puts an LLM on the critical path of a system that can affect the world at large, in some critical way, without supervision, the LLM is going to follow some pattern that it got from somewhere. And if that pattern means that the lawnmower cuts off someone's hand, it's not going to be aware of that, and indeed it's not going to be capable of being aware of that.
But we already have human-powered orphan crushing machines[2] today. What's one more LLM powered one?
[1]: Originally said of Larry Ellison, but applicable more broadly as a metaphor for someone or something highly amoral. Note that immoral and amoral are different words here. https://simonwillison.net/2024/Sep/17/bryan-cantrill/ [2]: Internet slang for an obviously harmful yet somehow unquestioned system. https://en.wiktionary.org/wiki/orphan-crushing_machine
A billion people use LLMs which have been trained to refuse unethical demands. LLMs show higher than human average ethical thinking. So it's not like a lawnmower by any stretch.
It is easy to persuade any modern LLM to demonstrate unethical behavior. Those trained “safety boundaries” are basically no more than baby corner guards on a construction site.
> the LLM is going to follow some pattern that it got from somewhere.
Good thing we also didn't write scripts on AI taking over the world and exterminating humans, numerous times