Not necessarily. For example, the Xbox 360 runs every game in a hypervisor, so technically, everything is running in a VM.

It's all a question of using the right/performant hardware interfaces, e.g. IOMMU-based direct hardware access rather than going through software emulation for performance-critical devices.