The problem causing the slow load times was reverse DNS look-ups throughout the server.  I had set /etc/resolv.conf to use Cablevisions DNS and they were not responding properly so I changed it to use OpenDNS’s 4.2.2.x addresses.  Everything seems happier now.