I'd hate to answer with a simple yes or no.
The big difference you're looking at here is how easy is it for an adversary to guess passwords to hack you.
If hashed passwords from some website are leaked, for example, there's no limit to how fast an adversary can guess passwords. They can guess hundreds or thousands of passwords a second.
However, on our phones, there's password guess throttling. An adversary has a very limited number of guesses before the phone rejects more guesses.
So, four words are probably more than sufficient to protect your phone from hacking, but, of course, it's your phone so it's completely up to you.
The guess throttling stuff is covered in this section on the website: https://grapheneos.org/faq#security-and-privacy. It's kind of long, but if you scroll down a bit to the bulleted list it's there.