Good for you.

IRS will never contact you by any other means than snail mail. What that means you will never get:

  • a visit from an agent at your door without prior notice via snail mail
  • telephone calls such as the one you’ve described
  • contact you via email [SPAM] electronic mail

Additionally, you could’ve reported the number used to call you to the authorities:
https://www.irs.gov/privacy-disclosure/report-phishing
https://www.treasury.gov/tigta/contact_report_scam.shtml
https://www.irs.gov/newsroom/scam-phone-calls-continue-irs-identifies-five-easy-ways-to-spot-suspicious-calls

The threat of financial damage is real and is of a very real consequence to every victim that is taken in by this scam, so reporting all instances is a must, imo.

It’s the best way to help.