Stress makes me break out around my chin and neck also, so could that be a factor for you?
Don't give up, but you might try using different oils. I know jojoba oil is supposed to be great, and most people rave about it, but for some reason my skin hates it! It took a bit of trial and error to figure out which combo works best for me. I personally use 50% castor, 40% grapeseed, and 10% camellia.
Hang in there and good luck!