Thanks for the clarification.
Any mention of signal or other end to end encrypted apps are completely off topic here.
You are absolutely correct to be concerned. SMS is built in to every phone and mobile carrier. It's vulnerabilities will exist regardless of whether we have encrypted messaging apps for friends and family.
I don't think there is a way to prevent auto retrieve. I believe the way the SMS are handled, means each message is processed in whole. I don't think the header information is retrieved separately first. Which means logic cannot be applied to determine if the payload will be retrieved or not. MMS I think can prevent downloading media. But the SMS exploits you're talking about, would still be retrieved.