The way I managed to make it faster is by creating a shortcut to the "Select user" activity using "Activity launcher" app. I made a shortcut on the desktop, that once pressed immediately opens a list of users. When the user is selected, I still have to extra-tap the "Switch to user" button. This was a tad faster than double swiping from top, pressing the "switch user" icon and picking a username.
What's most annoying currently is that fingerprint stops working for switched profiles, but that's unrelated to the question. Just to express disappointment. This problem hasn't been fixed for like half a year.