>with the technology plateau-ing
People were claiming that since year 2022. Where's the plateau?
The pre-training plateau is real. Nearly all the improvements since then have been around fine tuning and reinforcement learning, which can only get you so far. Without continued scaling in the base models, the hope of AGI is dead. You cannot reach AGI without making the pre-training model itself a whole lot better, with more or better data, both of which are in short supply.
While I tend to agree, I wonder if synthetic data might be reaching a new high with concepts like Google's AlphaEvolve. It doesn't cover everything, but at least in verifiable concepts, I could see it produce more valuable training data. It's a little unclear to me where AGI will come from (LLMs? EBMs - @LeCun)? Something completely different?)
> with more or better data, both of which are in short supply
Hmmm. It's almost as if a company without a user data stream like OpenAI would be driven to release an end-user device for the sole purpose of capturing more training data...
There's frequent discussions about how sonnet-3.5 is in the same ballpark or even outperforms sonnet-3.7 and 4.0, for example.
Could it be that at least for the "lowest" fruits, most amazing things that can one can hope to obtain from scraping the whole web and throw it at some computation training was already achieved? Maybe AGI simply can not be obtained without some relevant additional probes sent in the wild to feed its learning loops?
If you can't see it you're blind.
LLMs haven't improved much. What's improved is the chat apps: switching between language model, vision, image and video generation and being able to search the internet is what has made them seem 100x more useful.
Run a single LLM without any tools... They're still pretty dumb.