The browser will let you escape having to install the app which can communicate with other apps. But the website is still pinging google servers with identifying info like cookies and IP address .
At the very least, if your motive is to avoid tying your phone to this Google account, I'd use it with a VPN, in a separate browser, and block as many app permissions and site permissions as possible. If you want to go full paranoid, use it in a browser in a separate user profile with a VPN.
Obviously google still has access to all data you give it in Google voice such as messages, call history, regardless of how you choose to access it.