I'm one of these "crazy" people. I think that Google apps not having elevated privileges is already a huge privacy improvement. They aren't able to spy on you in the background anymore, only when you use one of their apps. They can't bypass the microphone, location and camera indicators anymore either.
I might be biased but I feel like I get a lot less ads that are related to what my phone could have "heard" after I installed GrapheneOS.