Two other things. First, I've had issues deleting stuff from the "home page" downloads folder too, but I could delete the files by going there directly. It might not work, but it's worth attempting if you haven't already.
If you don't know what I mean, you can hit the three lines in the upper left hand corner > tap your phone's name > Downloads. Try to delete from there?
Also, I found a thread on Google's support forum. A few people had luck using Google's proprietary Files app to delete their "ghost" files.