Well, don't feel bad. Address the issue appropriately,

Firstly, don't scold him, children are very innocent, you've got to spend time with him. Censor what he watches, even cartoons (not all cartoons pass the right message) who he plays with, get to know his friends & what kind of family they come from.

Have a talk with him & let him know it is inappropriate to say such to anyone. However busy you're make sure you squeeze time for your family.