Are there any guides or documentation on how to set up such keys? For example, is there a recommended setup?
Maybe try the official yubico docs? https://www.yubico.com/setup/yubikey-5-series/. The most important thing is to always have a backup key.
do I need google play services?
Yes, with storage scopes enabled.
which browser(s) are compatible?
I don't have an exhaustive list but most chromium based browsers are. The recommended browser here would be Vanadium.
do I need the YubiKey app?
No, the app is not necessary for a pure FIDO2 use case. If you want to use other protocols with your key, like TOTP, then it is.
Also, is it possible to use the built-in hardware security key present in the phone (Pixel 6a)?
I'm not sure on this one, someone else will have to weigh in.
I know there is an existing issue to support FIDO without play services that mentions that the future GrapheneOS solution will use on-device keys: https://github.com/GrapheneOS/Vanadium/issues/61. I'm not sure outside of that.