For anyone who encounter the issue it's needed to install the "google" app for the setup. Just installed without network permission and added nearby devices rights.
You have to allow shortly network access at the Google assist point to proceed and can revoke it direct afterwards.
After the setup I removed nearby again and deactivated the app.
All works fine so far.
Not happy about it but looks like a must.
Hope it helps anyone who has issues and search for it here.