This reminds me of when my first roommate in college seriously asked me why black people sing all the time.
You should just tell them, you're just saying hello. If anyone is over-thinking anything, THEY are. Who cares if you greet other black people. What's the big deal?