A watchdog has raised concerns after too many primary school children were found to be falling behind in reading, writing and maths.