Using your steps and all the apps you mentioned, I was unable to register a passkey with my Google account on GrapheneOS. I get to the point when it's about to prompt me for fingerprint authentication, and then the Google account passkey registration page on my computer simply says "something went wrong".
This is on Windows 11, stock Chrome.
Oh, and by the way, you need to be logged in to your Google account on GrapheneOS (I logged in through Play Store) to even begin the authentication process. I don't understand what you mean by the "Google app". I am not seeing any app called simply "Google" on my device.
The Yubico demo page worked fine. There was no need to be signed in to a Google account (thankfully). Chrome still needs to be installed and enabled, as it seems to provide the registration and authentication pages. There's no need for any third-party QR scanning app.