Title says it, how do I fix it? It's really annoying
This has been an issue for a long time, and is almost certainly an upstream issue. You can see it's already been reported in the GrapheneOS issue tracker a few times. Here's one from August 2022:
Here's a forum post that I responded to a few months ago. Apparently I searched Android's issue tracker as well but didn't find anything.