Don't know if this helps clarify your experience, but sounds to me like they probably CAN tell you are "black and white"
Many or most Latino, Cuban and Dominican people are a mix of African and Caucasian ... and often more. If they think you're Puerto Rican or Brazilian, same thing. Heck, same if they say you look like an Arab - all those groups are very racially mixed. Many North Africans (leaving out most Sudanese, Somalians, Eritreans and Ethiopians) can blend right in with Saudi Arabians.
People guess I'm Greek, Spanish or Italian (or just think I'm straight up "Jewish"/Israeli). ... But in no way do I take that to mean they think I "just look white", even when they ignorantly say they think that. No. I trust they're seeing someone "mixed" with black/white because the Greeks, Italians and Spanish are located just above North Africa and many of them have Moorish ancestry and you can look at them and see they aren't just white. And there are Jewish and Israeli people that look "mixed" and there are many tribes/ethnicities it comprises (though I happen to be an Ashkenazi Jew).
So yeah, I think people usually DO know what time it is with many of us mixed folks. I think they unconsciously know we're mixed when they "mistake" us for the endless list of ethnic groups that just happen to be as mixed as we actually are
This also tells me we're probably not as "special"/rare as we're portrayed to be. I mean, sure, some of our experiences are special in that we often have to bridge more than one culture, but there are whole countries whose people reflect the genetic diversity we call "mixed".