PrivacyEndGame or a high level threat holding your device in an unpatched state long enough until an inevitable exploit to unlock it is found
This is only a concern if your encryption is weak.
In BFU, they need to be able to decrypt before they can do anything. An inevitable exploit can't just magically find the decryption key. It would have to be logged somehow, or bruteforced.
It's not going to be logged, because you're not there to enter it. So it would need to be bruteforced. The encryption uses a combination of a hardware-backed key + your unlock method, so even if you had a 1-digit pin, they can't just dump the data from the chip and bruteforce it on a computer. It needs to be done on-device, so the key from secure element is accessible.
If you were using a very short PIN (2-3 digits), that can be bruteforced in a reasonable amount of time with no exploits. A PIN of 6 digits in length is revommended because the throttling means you can't bruteforce 6-digits in a reasonable amount of time.
But there can be vulnerabilities that would allow bypassing the throttling, as you know. So use a long passphrase instead of a PIN. Then it doesn't matter what exploits ever come out, unless the attacker was able to log your passphrase, they're not going to be getting in. A sufficiently long passphrase would take millions of years to bruteforce even if the throttling were to be bypassed.
But a sufficiently long passphrase isn't convenient, that 1) takes a long time every time you need to unlock, and 2) every time the password has to be entered, there is a risk that prying eyes could read as you are entering it (especially in public places, or other private spaces that aren't your own home.) Fortunately, you can set fingerprint unlock to be used as a secondary unlock method. So you would have a strong passphrase keeping storage secure, while being able to easily unlock with one tap.
But then you'd be vulnerable to law enforcement requiring you to unlock, because it's a fingerprint and not a PIN. This is part of the reason why GOS is planning on adding the option for a second factor PIN to go with fingerprint unlock.
So the idea is that you have a strong passphrass to keep your storage securely encrypted, that you only need to enter once while at home after a restart, and every time you need to unlock in AFU, you touch with your fingerprint and then enter a short PIN. Autohreboot nearly guarantees you will be put back into BFU if the device is seized, the storage can't be reliably decrypted even with throttle bypass, you don't lose usability in daily use, and you can't be required to unlock it because of the PIN.
https://github.com/GrapheneOS/os-issue-tracker/issues/28
It's not implemented yet and I don't know how far along progress has come for it, but somebody very recently opened pull requests for it. With any luck, their code meets GrapheneOS quality standards, and it can be available in an update soon.