Do you have hard water? Sometimes the only thing that will remove hard water buildup is a clarifying shampoo (unless you can find a shower filter that works well, which can be a crapshoot).
Given that you use it occasionally, and that your curls seem springy and healthy afterwards, I wouldn't worry about it too much.
2c, fine, extremely dense, color-treated, high porosity.
CG since 10/12