Oh My Zsh adds bloat but sets sensible default settings compared to vanilla zsh. I tried using zsh without OMZ and I realized how many things are in OMZ which I took for granted, especially keybindings.

~150ms with minimal setup is good enough for me, although I would prefer <50ms.