Holy moly, I think @MetropleX just gave me the perfect hint here. I usually have animations disabled in Accessibility.
Now I have enabled it again and playing with it for a few minutes, also changing the setting back and forth a few times confirms this: The problem is gone!
@DeletedUser499 Do you also have animations disabled in Accessibility?
Maybe this was also the reason for this problem I had. This seems to be fixed, so the mode without any animations seems to have gotten more stable.
Now I have to decide: A wobbly UI or this bug. I chose the bug, cannot stand these animations!