Was the M1 using its GPU cores? It should have more power than a 1080.
Does macwhisper utilize metal GPUs by default or is it something you have to tick?
I'm not sure, sorry, been a while since I used it.