I'm guessing this isn't a general impression Brits have of how Americans sound, but I had an English guy tell me that 'men sound like cowboys and women sound like porn stars' here. He was in California so that may have had something to do with it. Cracked me up.
Don't take statements like that to heart. If you put something on the internet like a voice clip, you'll get all kinds of conflicting feedback. Sometimes people will give you a hard time just to give you a hard time. Your voice sounded perfectly fine to me.
I'm thinking by "neutral" they mean an American accent that is more prone to be seen in the middle of the US. You know, not a Georgian, Texan accent (just an example, there are plenty more regional accents, those were just the ones I could think up at these wee hours). Everybody has an accent, it just depends on where you're from and where other people are from.
I've been told by Brits, French, German, Russian, Polish and all sorts of other nationalities that my accent is strange, probably twangy. I suppose it probably is. I very much have an accent from the place I grew up, but still live there, so it isn't noticed here. My vowels probably sound pretty hard to someone who isn't used to my accent.