I used to have that problem you're describing, but that was some time (months) ago now. It used to happen after a GBoard and/or Speech Recognition and Synthesis from Google (SRSG) update. What I then had to do was temporarily enable network (i.e. internet access) for GBoard and SRSG. I found that I had to speak one word with the internet available in the GBoard app, then I disabled network access from there on in. I had to do this regularly, esp. shortly after a GBoard update. The GBoard offline speech-to-text was kind of working, but it was incredibly slow and not as accurate.
But I remember toying around with some settings, as it was getting a little annoying like what you describe. I cleared the GBoard and SRSG cache, made sure in GBoard 'Use voice typing' and 'Faster voice typing' were chosen, re-downloaded my preferred offline languages files, spoke the one word thing.
At the time I was using a couple of other keyboards like HeliBoard, but now I just have for the on-screen keyboard Keyoard, GBoard, Google Voice Typing, and Whisper Voice Input --> all in the main user profile, I don't now use another profile or Private Space.
And under the Speech settings I only have for Voice input: Whisper Voice Input, not SRSG. From memory, I used to have SRSG as the main Voice input.
These things above I did, and it seems to have made a difference, as I've been enjoying offline speech-to-text with GBoard for some time without having to go through the processes as described above for some time now.