Rapunzli
The first is AF-Wall. But it is only operable on rooted devices. I would like to have a replacement for this, because I want to have detailed control, whether an app may use WiFi or mobile net or nothing. There are many apps, which do not need any network connection.
This is a badly designed app which massively reduces OS security. You should read our FAQ sections on these topics where we mention apps like RethinkDNS able to do local filtering which also using a VPN. We don't specifically recommend RethinkDNS but unfortunately there aren't other examples without bigger issues.
So I have tried NetGuard Firewall, which will insert a virtual VPN within the device. But it seems to be incompatible with GrapheneOS, because after installation and activation, no app could connect to network services.
It's not incompatible with GrapheneOS. It's incompatible with the standard Android leak blocking toggle because of implementation flaws. It shouldn't be used.
Second I would lie to use AdAway. This app will function without root access, but the will insert a virtual VPN similar to NetGuard Firewall and with the same result: No other app can connect to network services any more. So my question is, whether there is no need for an app like AdAway on GrapheneOS, because it would suppress unwanted ads on system level. Or does someone know of an app, which would be compatible with GrapheneOS?
You're trying to use the Android leak blocking toggle with a VPN app not providing traffic routing. The leak blocking toggle is working as designed and preventing all the non-DNS traffic from leaking. This is a configuration error on your part. There's nothing about this specifically incompatible with GrapheneOS. It's a bad implementation of this though and it encourages massively reducing OS security. Recommend avoiding this app too.
NetGuard is meant to actually filter traffic and SHOULD be compatible with the leak blocking toggle like RethinkDNS. None of this is specific to GrapheneOS. We improve the leak blocking but it does not break any of these apps. They would break with it on the stock Pixel OS too, it's just not enabled by default in the setup screen there.