You shouldn't have router problems at all to be honest, unless you're doing a lot of downloading/uploading in the background.

K, so it's a Linksys... So, there's a good chance that it's got a config for 'quality of service'. You could add port 53 & set it to high priority, and if your router allows you to add a range set ports 20000 - 20300 to high priority too. That would ensure that your router is isn't causing the problem.