Yes, I solved it by deregistering the number with google.
I guess since I'm on a google pixel it was trying to send rcs to the other android phones or something, which wasn't working with GrapheneOS.
I'm not sure how it got registered with google in the first place as I come from iPhone.
On the other hand I'm still stuck using the default messaging app or it makes a new chat and messages myself in group chats.
I'm guessing this has something to do with dual sim but I haven't found a solution yet.