The short answer to the question in your subject (Topic Title) is yes.
Longer answer, whilst routers often have firewalls, they only act in one direction, inbound, so you need something to protect against unauthorised outbound protection (unless your routers firewall provides this, unlikely).
That is what you are seeing in the third party firewalls as they ‘learn’ about what is on your system and what they do, this phase takes a little time and if as has been suggested you reduce the sensitivity level that to will reduce the pop-ups.
I went back from Comodo to ZoneAlarm on my both machines (XP and Vista)… I am unbelievable happy with how it works and to tell you the truth, not thinking of going back to Comodo.
ZoneAlarm can be a bit of a nag when you first set it up however if you use the “Back up and Restore Security Setting” feature once you have it up and running ZoneAlarm is very non invasive. I don’t use the Anti-Spyware feature silly addition imo.
Indeed, I find this review totally biased to payed firewalls.
I’ve bought a router with a firewall. I’m not so sure I’ll keep my software firewall running… why, just to see if a non-authorized program is connecting the Internet? I hope the antivirus/antitrojan protection do it before (avoiding installation). Some people wants outbound protection to avoid pirated software to connect, update and then losing that programs due they’re non-legit. For a while, Comodo is there on system tray… I’ll see if it will worth…
Actually all I need from it is Outbound filtering and ZA does pretty good job in that field. I am behind two routers (linked) right now, so no fear when it comes to inbound protection. I feel like I live in a fortress, hehe… ;D