I automatically think of tans and browns. I also love gray with navy. How about khakis?
I have a navy military style blazer and I haven't worn it yet, but some dark gray or brown skinny pants are just screaming to be worn with it...and boots. I gotta have my boots