Calling people "illegals" is actually pretty offensive as you thought when typing that but without a different term to use. "No, you're not legal" sounds a bit like "you should have never been born." Their immigration status may not be legally validated, but their existence itself isn't illegal.
Ah, it all makes sense now. Goldy is the puppet master!