Sorry to know that. I can remove my USB drivers (more than one, with more than one partition) without trouble.
Any other software could be interfering?