President Trump was originally slated to spend the entire weekend at the retreat and come back to the White House on Sunday.