The only thing you can do is tell him directly what happened and don't get derailed by some stupid conversation about you violating his privacy. Apologize for that and get it out of the way and then tell him how you feel about it and ask him what's up. *I* would tell him I promise not to get all crazy if he will just be honest...in other words, don't tell me whatever bs you think I want to hear, tell me the truth and we can work on it.
At the VERY least, it sounds like he is engaged in some pretty heavy flirting and IMO, that energy is better spent on your WIFE. Other couples can make their own rules, but in my relationship that would be crossing a line.
Good luck and it sucks.