Is downloading torrents legal?

I wasn't 100% sure if they were legal or not. I thought I heard somewhere that there was a loophole where technically it is legal. I don't want to get in the habit of doing it, if I could be potentially penalized for it.