I agree. On my Pixel Fold it is actually the opposite issue, in that it is always too dim. I wish it had an options screen with fine tuning adjustments.
The only downside with it off, is if I dim the screen all the way at night, and wake up the next morning, the screen is so dim relative to the ambient light, that I can't even find my way to the brightness slider.
I remember in the old days, there were 3rd party apps to take better control of auto brightness. Maybe they still exist.
I have to say, coming from Samsung's One UI, the vanilla Google AOSP or what ever this is, sucks comparatively. I feel like Samsung actually tested the software before releasing it, and cared about how it functions and feels. Google is pulling the old Apple B.S. where they decide what's 'good' for you, because "YOU are not important and YOU are stupid; we will tell you what you want."