Yes and no.
Anyone can access the recovery mode and do a factory reset, thus making you lose any data you did not back up elsewhere. That’s part of the data security definition.
Otherwise, as long as the firmware and software have not been tampered with (check with Auditor), the data should be safe and private due to the encryption.
I guess that in some extreme cases, you should not enter the passphrase again on the device.
Obviously, if someone already knows your passphrase or is threatening you, it’s another story.
Not sure it answers your question. And not sure everything is accurate.