phil-p you should be fine in terms of operation. In my experience, it's safe to remove all permissions from GSF if that brings you any peace of mind. It's the only Google component I install as well, for the same reasons.
If you remove GSF and the applications do indeed rely on it, they will break and remain broken until you install GSF again and then uninstall/reinstall the affected apps (in that order).