If you want to protect against brute force, you need at least a random 6 digit PIN to provide a high level of security based on the secure element throttling. If you want to entirely avoid dependence on hardware-based security, then you need a strong random passphrase such as around 7 random diceware words. That's very impractical for anything but a secondary user for particularly sensitive data unless you're using biometric secondary unlock.
For everything in between a random 6 digit PIN and a truly strong random passphrase not depending on hardware security to outright prevent brute forcing, it's more complex. There are multiple software and hardware-based features designed to stretch the benefits of a weaker passphrase: scrypt-based key derivation in the OS, the secure element throttling (Weaver) and finally hardware accelerated, hardware-bound key derivation by the TEE using the outputs from the OS and secure element. The secure element throttling is very aggressive so it works with a random 6 digit PIN. The scrypt usage by the OS and the TEE hardware accelerated, hardware-bound key derivation are only really useful with a passphrase. Even if the secure element does get compromised by an attacker, they still have to brute force, and the TEE feature prevents them brute forcing faster than the rate supported by the TEE unless they can bypass that feature by extracting the key from hardware if it's implemented as intended (physical extraction or some kind of side channel flaw are required if it's done as intended where the key derivation is keyed based on a key in hardware that's not available to TEE firmware). If they can bypass that feature too, then they're limited only by their available server farm infrastructure, and therefore if you want to entirely avoid reliance on hardware security features, you need a strong random passphrase that's secure simply based on having enough entropy. Around 90 bit is still secure, and there's scrypt + TEE key derivation making that better than simply a 90 bit key. You don't actually need to go overboard with an 128-bit entropy random passphrase but you could.