The idea with calculators was that as a tool there are higher level questions that calculators would help you answer. A simple example is that calculators don't solve word problems, but you can use them to do the intermediate computations.

What are the higher level questions that LLMs will help with, but for which humans are absolutely necessary? The concern I have is that this line doesn't exist -- and at the very best it is very fuzzy.

Ironically, this higher level task for humans might be ensuring that the AIs aren't trying to get us (whatever that means, genocide, slavery, etc...).