I decided to try recreating the part from the beginning. I chose a generic IC and changed all the svg's and Metadata and connections in a single go and only saved it once. This produced a usable part. I then went and copied all of the new working parts svg's and renamed them to match the old part. This did update all the svg's but it did not fix the issue. That only leaves the Fritzing part metadata file. I tried copying the working one and renaming it to match the old one but Fritzing complained so I reverted it. I then opened both files with a text editor and switched their moduleId= 's while leaving their file names intact.
What do you know once I reopened my sketch I could switch sides and everything worked. I hope this helps someone in the future.
I believe the cause of this is editing the part multiple times with the Fritzing part editor. It created so many groups (30 deep or more) and transformed and translated the paths multiple times. When I recreated the part I removed all the extra groups and translations etc and used very clean SVG's that Fritzing hadn't mangled yet.
