Ledger had an embarrassing moment earlier this year, when they announced users will be able to backup their seed phrases to Ledger's servers. Supposedly an opt-in feature, but you still have to trust their firmware updates.
https://decrypt.co/140364/is-there-a-backdoor-in-ledger-hardware-wallets
If you're dealing with relatively small amounts, you could just Trust Wallet, it's open source and they provide the signed APK on their website for download.
Paper wallets are another cold storage method that you could look into.