I tend to agree that PIN scrambling is overkill for the majority of standard threat models, it's a good feature and it's good that it's available, but the majority of users probably don't need to add this complexity every time they want to use their PIN.
Also, if you trust SE, a random 6-digit PIN code is correct, Google Pixel generations 6 to 8 use the Titan M 2 which is a really robust security chip.