It's been a bug for a while and the only way to avoid it is to not select that server and you should be fine. I'm sure it would be an easy fix if someone looked into it.
I've had the exact same problem and it was especially annoying when I had a habit of using the arrow keys and as I'm sure it can be fixed just avoiding it by not using arrow keys is the simplest solution.