I may be wrong here so these are just my musings but I think one reason is that it draws attention if it's apparent in any way that you're using it (for example, something scans and sees the auditor app). People who care about (or are "technical" enough to know how to care about) their digital privacy/security are a really tiny minority, so we stick out like a sore thumb in a sea of all-in laid bare Samsung and Apple users and will be automatically assumed as "having something (illegal) to hide". I'd prefer if if they saw me for what I am, which is basically a protester sick and tired of government overreach (UK surveillance state in particular) and surveillance capitalism (Google), but the downside is ending up with probably 100x the suspicion that you're doing something bad and being watched all the more closely.
This wouldn't be as bad if it was easy to at least follow through and... not be leaking everything everywhere. This is not GOS's fault, it's just how reliant everyone is on Google. A big part of my reason for using it is to stop Google hoovering up every bit and byte I ever interact with but so far, unless I leave it as a "slightly smarter dumbphone with a great camera" (i.e. stock GOS) I'm coming across Google's claws at every turn. Install a few things you need and inevitably you end up needing to get Google Play Services and whilst it's sandboxed you end up needing to poke holes in its permissions and then effectively, Google is in and tracking everything just as if it was a Google ROM in the first place (except now with added suspicion as far as any "spooks" are concerned)
I guess at least once the IPC controls are in place it'll mitigate some of that Play Services leakage, but there are other issues like push notifications - seemingly only a small handful of apps support UnifiedPush and good luck getting developers to tailor their apps to support a tiny niche, so you end up needing FCM. The notification service famous for popular apps like Discord leaking your DM contents in plain text, keeping a record of all push notifications perhaps indefinitely and then just casually handing them over to law enforcement (apparently Apple at least needs a warrant now, not seen the same of Google). Which of course isn't a practice that would necessarily stop with terrorists or child abusers - or those definitions necessarily staying where they are, considering how many people have gone back to throwing the P word at LGBTs these days. So everything ends up out there on the Googlenet after all except now you're on a watch list as well, imo.
It feels a bit hopeless to be honest. On top of that, I'm reading how you can be identified just from your typing style on posts like this, which I can believe - sometimes I come across one of my old posts somewhere long forgotten and think "that's exactly how I'd say it" and then notice it was me, heh
On the flip side, it's also a reason to use and encourage it. If we can finally get more people to care about their privacy and security then we can get more developers to care about looking after it, get more people off data-hoarding and data-leaking services like Discord, and make it something that normal people do rather than something that only a few tinfoil hatters do.