I downloaded the Fossify contacts app and it is the same issue. Each time I add a new contact to the app, a random color background is assigned to each contact and there isn't an option in the pencil (edit) icon to change the color.
If all the colors were on the darker side, I wouldn't mind as much. But the lighter colors like pink, red, orange, are hard to look at, and those colors I want to change to a darker color like blue, cyan, green, so it is easier to look at and easier on my eyes.