Maybe not get up and leave but consider that if someone just didn't come back or stopped coming because of it but would not feel comfortable saying so, you might never know the reason.
