If they are local network IPs then any external source still has to go through your firewall and your security program/s to be able to do anything.

If they are external IPs then I would say they are less of an issue.

But for sure no one can say without information, chicken and egg.