When I first met my wife over 40 years ago, I was playing in bands at the time. My wife (GF at the time) thought I was trying to shag every waitress I encountered. She was right. What can I say?
Maybe you've let your imagination run here. Next phone call just say "hey man, this might sound crazy but it crossed my mind, you don't think I'm seeing your wife do you?"
