Foreign Secretary Ed Miliband said all options for a closer relationship with the EU would have their drawbacks.