You could try home networking, of if you can, plug the hard drive into your new computer and copy them over that way.
Yes, a USB stick/jumpdrive etc. would be sufficient to transfer things over quickly depending on the size. However, depending on the operating system, you may still need to put in the drivers to identify it (i.e. Windows 95, 98).
If you plan to keep the old computer, I would suggest setting up a wireless LAN if they are distant, or connecting directly through a ethernet cable if close by (assuming both have the port).