Here's what it boils down to:
If you want to rely on the secure element's throttling capabilities, a 6 digit PIN is sufficiently secure.
If you don't (you want to rule out a secure element bypass exploit being used), you'll need a password/passphrase that provides more entropy. Around 90 bits of entropy is what you want.
90 bits would be a 18 character password comprised of lowercase letters and numbers, or a 7 word diceware passphrase, which cannot really be bruteforced.
Beyond 90 bits of entropy, you're entering extreme overkill territory, and you don't realistically need that.
I would recommend reading this reply by the project account on the matter:
https://discuss.grapheneos.org/d/4049-security-from-bruteforce/66