Jemima Goldsmith, 52, said she had to call the police to have Brand removed from her house in 2014 during a "violent incident"