Good question. I had a little look into this.
There is the menu in Vanadium > Kebab menu (3 dots) > Settings > Downloads. There in theory you should be able to select a different directory to Downloads. However, just like when you download a file, it is not possible. Not sure why that is the case.
I tried adding a different directory to storage scopes for Vanadium. That didn't help.
Other than this I am out of ideas. If this feature is important to you, I would suggest opening a new issue on GitHub, as a feature request: https://github.com/GrapheneOS/Vanadium/issues
As you say, it would in any case be interesting to know why Vanadium doesn't allow you to change the Downloads folder, and also to know how this works upstream.
I had a search of the repository on GitHub. The only things I have come across are a couple of patches committed all the way back in 2019 to Vanadium. But I am not convinced necessarily that they make a difference to this functionality. You could mention this in an issue, if you create one.
https://github.com/search?q=repo%3AGrapheneOS%2FVanadium+download&type=commits