I download without VPN's from time to time, never had a problem. I don't do it very often, though, besides porn.
Though from what I've heard about my ISP, they don't really care about pirating like other ISP's do, they basically just ignore the warnings that content creators send them because they're lazy.