A council is having to hand back thousands of pounds after wrongly dishing out fines for offences like dog fouling.