This has happened to me. Can't find the exact combination of scopes required for the job to be done so you end up with the "f this" scenario you mentioned. And it is a constant source of background worry.
Don't forget the also fun classic "what you want to do is not possible with scoped tokens so enjoy your PAT". I think we're now at year 3 of PATs being technically deprecated but still absolutely required in some use cases.