ITA with telling your ex that they are coming home too late and need to just come home after swimming.
She can make other arragements to see them and she doesn't need to take care of their basic hygiene needs. That just sounds crazy, IMO. she needs to be in control of SOMETHING from what it sounds...
Unfortunately, this means that you need to say something to your ex, take a stand, and make it so she doesn't have that control/choice.