Apps can access data you let them access. If you give an app access to your contacts, it can access them.
One possible thing you could do is use a contacts app that allows you to separately store contacts within its internal storage where other apps can't access them.
One such app is Connect You: https://github.com/Bnyro/ConnectYou
GrapheneOS will eventually have a Contact Spaces feature that allows you to decide which contacts apps can access, but using apps like the ones mentioned above is a stopgap measure.
Of course, by using an app like that, any contacts you put into its internal storage will stop being accessible to all other apps, not just whatever Dialer you use.