Why do people give me a hard time about my accent?

I have a thick Ricardo Montalban accent, and a lot of people mostly girls ask me in a very stern way where I was born. I sometimes ask girls that have accents where they are from but it's in a curious way. I thought accents were endearing to girls but the way they question me about my childhood seems otherwise.