There's something I didn't consider.  I have OpenNIC in the DNS for both server or client, so it's possible if its going over the OpenNIC rootservers it may be slowing things somewhat.

I'll run some tests when I have my head about me some more.