Isn’t that just a Turing test?

I’m perfectly willing to bet that there are LLMs that can pass a Turing test, even against a mind like Chomsky.

I think that misses the point entirely. Even if you constructed some system the output of which could not be distinguished from human-produced language but that either (1) clearly operated according to principles other than those that govern human language or (2) operated according to principles that its creators could not adequately explain, it would not be of that much interest to him.

He wants to understand how human language works. If I get him right — and I'm absolutely sure that I don't in important ways — then LLMs are not that interesting because both (1) and (2) above are true of them.