Yes, babies can def get stressed. And it's worse for them bc they can't communicate their feelings and you can't verbally console them. So all they can do is cry and then internalize the rest. IDK that they can "sense" emotions but they can def be startled by shouting and door slamming and being put down/picked up less gently than they should be.
Poor little guy. I'm sure he'll appreciate the break he gets when he's w/ you.