That is the difference really ... the cause of this argument. Some call recasting stealing. Personally, I don't think it is. I wouldn't consider recasting any more morally reprehensible than using proxies. Either way, you're not using the "official" tinker-toy.
I also think the very notion of "IP" is simply a ploy devised by lawyers to make lawyers rich. The original concept of copyright and patent was never intended for the shennanigans it's being put to today. The original intent was solely to allow an inventor or writer to recoup his development costs, and nothing more. It was not intended to become a proverbial ball and chain to shackle the public to.