Two Men In A Lift

Had to deliver some documents to another office. I got in the lift with 2 men (one in his 30s and the other one in his 50s) in some soldier/police/bush ranger type of uniform. I pressed the button to the top level.

Younger Man: Going all the way to the top?
me: yeah
Younger Man: You must be special
me: I am
Younger Man: See you later

After I delivered the documents, I got back in the lift and guess who comes in a few floors below?

Younger Man: Are you following us?
me: *fake laugh*
Younger Man: what do you do here?
me: bringing some stuff
Younger Man: which company do you work for?
me: I don't work here
Younger Man: so you're not that special then?
me: no, I'm not

I don't know why but those sort of men creep me out. They way they gave each other smirking looks, like I couldn't see. If some old lady said the same things to me, I'd just think she was chatty and wouldn't find it weird at all.