Leo_51 I reported this problem on the Grapheneos GitHub repository, but they closed it.
https://github.com/GrapheneOS/os-issue-tracker/issues/6709
If there is nothing actionable for the developer team, closing the issue may be the right thing.
Unfortunately it seems that at present there is a disagreement between you and the secure element, and, by design, the secure element has the upper hand.
I think it is best to write the PIN down, wait for the timer to expire, and then very carefully enter the PIN (and it may actually be better to ask somebody else to do it).