A 79-year-old grandad who lived off-grid in Swanscombe for 35 years has lost a court bid to return after his home was bulldozed.