I'd suggest you read the Audio Adapter Accessory Mode section of the USB Type-C Cable and Connector Specification, which may help you better understand how USB-C audio works.
As others have mentioned it is possible for there to be an inline chip performing nefarious activities, but it is not something I would suspect occurs very often outside of a physical "phone bugging" style attack - this of course depends on what/who your perceived threats are. Skimming other parts of the document, it would seem any sort of inline 'chip' is outside the realm of the specification.
There are also the USB preferences inside GrapheneOS, e.g. no data transfer. I don't have a USB-C to 3.5mm adapter available, but I would suggest testing with one you have some level of trust towards and seeing how that interacts with the USB preferences. My understanding would say it is just a dummy adapter and you would not see any data options.