The threat model for Contact Scopes is to give specific apps access to only specific contacts saved on your device.
Contact scopes don't help with your particular threat model.
If somebody has your phone number saved on their device, and the Signal app installed and setup, there's no way that I'm aware of that would prevent them from being able to see that you have Signal.
The only thing that might work is blocking the user in the Signal app, but I'm not sure if that just stops them from being able to message or call you, or actually stops them seeing that you Signal installed altogether.
For further support with this, I'd suggest contacting Signal directly.