AtmosphericIgnition would think that the risk of a bad actor flashing my key with bad firmware could be effectively prevented by only allowing loading of firmware signed by the manufacturer's keys.
This is naive! Those HSM manufacturers are relatively small companies. I.e. Nitrokey GmbH has a capital stock of 25000€. Companies can be sold any time. The buyer gets the power of updating the keys if they are updateable.
AtmosphericIgnition Does anyone here have some general advice on what I should be looking for, or avoiding when buying a security key/HSM?
Think about the format first. Do you prefer a key with USB or a smartcard? Contact based keys (they can be cards as well) are in risk of you forgetting them plugged into the device. Flat cards fit better in your mobile cover or purse. You can even put them in a finger ring.
Then, think about the security level. The strongest standard is cc eal6. But that applies only if your whole process is certified. FIDO2 Level 1 devices on the other hand do not have certified hardware at all, they are designed for over the internet attacks only.
Finger rings are inherently safer because you wear them all the time. Not easy for an attacker to tamper them while you are asleep or showering.
That said I think the perfect solution does not yet exist.