A combination of university classes on cybersecurity/computer science and having experimented/messed with virtualization for years. The theory is quite simple: is there something that stands out among most virtual machines? Often, they all use the exact same network cards, the same sound cards, the same USB hubs, and they have some backdoors for sharing files that can be detected. From there on out you can extrapolate.

Malware tries to detect virtual machines security researchers use to analyze them, which is where I first started reading about this stuff for. An afternoon reading into hypervisor tech a few years ago taught me a lot as well; just keep digging deeper until you’re close to the hardware. Then there was a vain attempt at running a game or two in a Windows VM and digging into why it didn’t work.

If you’re interested into this stuff, I recommend following a tutorial about writing your own hypervisor (if you know any native programming language, it’s actually surprisingly easy with modern CPUs! the real difficulty is writing all the device implementations and their drivers…) and experimenting with detecting virtual machines yourself.