“The family is happy that they can put this behind them,” a source tells PEOPLE.

“It got a little out of hand,” the source admits. “Phil has never been disrespectful to anyone and treats everyone with kindness. He made some comments off the cuff that he shouldn’t have made, and he knows that.”

The source says that the headlines didn’t cause any drama among the family, who all support their dad.

“The family knows his heart and knows that he wasn’t trying to be hurtful,” says the source. “They just want to finish celebrating the holidays and are looking forward to getting back to work.”