The number of visitors to a council-run museum has dropped dramatically after the authority introduced a budget-balancing charge.