Residents have expressed public safety concerns after it was announced that a Kent prison will hold only convicted male sex offenders.