I did full testing on AA when released, and agree that getting the contacts to work on assistant was one of the biggest hurdles.
I unfortunately did not write everything I did down, but from memory believe google play services requires contacts permission as well.
Good luck !