The olive oil, and the oils in the milk will pull the color out. And that will happen pretty much anytime through the life of the color.
Truthfully, the molecules of the oil are too large for your hair to absorb anyway, so it really wouldn't do much but sit on the outside layer or your hair and make if feel soft.
or Too Shea
is good to use while coloring.