>people keep trying to apply engineering principles to an inherently unstable and unpredictable system in order to get a feeling of control.
What's the alternative?
Using predictable systems.
If your C compiler invents a new function call for a non-existent function while generating code, that's usually a bug.
If an LLM does, that's... Normal. And a non-event.
If we have to use predictable systems, how could we use humans in the first place?
And?
What other engineering domain operates on a fundamentally predictable substrate? Even computer science at any appreciable scale or complexity becomes unpredictable.
Every engineering domain operates within "known bounds". That makes it dependable.
An engineer doesn't just shrug and pick up slag because it contains the same materials as the original bauxite.
Of course not, but how did we get to that point with materials science and chemistry?
We’re basically in the stone ages of understanding how to interact with synthetic intelligence.
Through research and experimentation, yes.
But attempts to integrate little understood things in daily life gave us radium toothpaste and lead poisoning. Let's not repeat stone age mistakes. Research first, integrate later.
Pretending that the world is stable predictable and feeling in control while making fun of other people. Obviously.
Math and physics are pretty stable. So is computer science. Avoid voodoo.
LLMs are just math.
It’s reasonable to scope one’s interest down to easily predictable, simple systems.
But most of the value in math and computer science is at the scale where there is unpredictability arising from complexity.
It's reasonable to perceive most of the value in math and computer science being "at the scale" where there is unpredictability arising from complexity, though scale may not really be the reason for the unpredictability.
But a lot of the trouble in these domains that I have observed comes from unmodeled effects, that must be modeled and reasoned about. GPZ work shows the same thing shown by the researcher here, which is that it requires a lot of tinkering and a lot of context in order to produce semi-usable results. SNR appears quite low for now. In security specifically, there is much value in sanitizing input data and ensuring correct parsing. Do you think LLMs are in a position to do so?
I see LLMs as tools, so, sure I think they’re in a position to do so the same way pen testing tools or spreadsheets are.
In the hands of an expert, I believe they can help. In the hands of someone clueless, they will just confuse everyone, much like any other tool the clueless person uses.