Health bosses say they have no significant concerns for people living and working near a massive lead refinery after fears were raised.