Definitely suspicious Erik, glad it didn't turn out to be more serious. I made a similar mistake several years ago. My wife, daughter, and I were all asleep in the apartment. A knock on the door and I opened it, my pistol laying on the coffee table, and found a HUGE black man in his late teens/early 20's holding magazines. I had no shirt on, and was a big time weight lifter then so I had muscles everywhere that I have extra cushion now. That may have been the only reason he only made me tell him "No thanks" twice (2nd time not so nice) and walked on. Maybe 30 minutes later, cops are everywhere in the circle drive and seems this big piece of $&#*@! decided that an elderly man several buildings down was an easier target and shoved his way in the house and beat him. I'm assuming to rob and run, but seems the old guy fought back so he didn't get anything. I was really hoping I'd find that slime walking down a dark road in the following weeks, I'd loved to have gave him a fair fight!
I've always been suspicious of uninvited visitors, but let me tell you, that really pissed me off and set me at a much higher level of alertness. Now I'm like you were, pistol always on my side, in my hand at the door, and watching everything.