I wear my unique jackets with pencil skirts or dress pants. Because it's a more casual jacket, I wouldn't put a collared blouse under it (though I could see it looking great with a white mens' style dress shirt) but would wear it with a turtleneck or long-sleeve shell-type top. I'd wear navy with cream, red or brown. I'm not sure about grey - a deeper charcoal grey perhaps. Plum might be an interesting contrast. Black is possible.
I also wear jackets over simple dresses, like a shift dress, sheath dress, sweater dress, wrap dress. For that one, you could use the contrast of what sounds like a pretty "rugged" type jacket with a floral or patterned dress, or you could just go for a very simple solid dress in one of the above colours. I agree that boots would be a good match with the jacket - I like the idea of a dress and boots. Either knee-high boots or little booties.
Get used to me. Black, confident, cocky; my name, not yours; my religion, not yours; my goals, my own; get used to me. -Muhammad Ali