I have a Graphene OS phone with installed both free software from Android and proprietary stuff that I trust less from Google Play. Some proprietary applications are just for convenience, some because life without them would be noticeably more difficult (apps for things like banking, paying invoices etc.) I assign only minimum necessary for application to work from permissions when I install new software.
My current workflow is Debian with only free software from main repo + non-free firmware and more pragmatic setup on my Google Pixel. I treat my Pixel as a second computer whose main purpose is to allow most important things quickly, so that all job needing more predictable and secure setup is done on computer with Debian
Is it a good idea to install from F-Droid and configure app like ConnectBot to manage my servers through SSH from my phone with Graphene OS? The reason I consider doing this is I don't always bring my laptop when I travel to places far away from where is my home. For SSH authentication I always use keys, not passwords and SSH server is behind WireGuard. I consider doing this SSH setup with separate key for phone of course
Regarding to threat model, I'm regular Linux enthusiast and person that is interested in free software, system administration, not high value target, activist etc.