Yes they can. If the token you give the LLM isn't permitted to access private repos you can lie all you want, it still can't access private repos.
Of course you shouldn't give an app/action/whatever a token with too lax permissions. Especially not a user facing one. That's not in any way unique to tools based on LLMs.
I thing you are just arguing about words, not about meanings. I’d call what you are referring to “secure llm infrastructure ”, not “secure llm”.
But the thing is that we both agree about what’s going on, just with different words