Deleting the data might slow things down until the cache builds up again, but it shouldn't cause issues.
Privacy wise, I'm pretty sure that the storage data is a black box. Deleting the data from your phone won't delete it from Google's servers. If you never gave GPS network access then deleting the data would prevent Google from accessing it - however, other apps which can IPC with GPS may have already exfiltrated the data.