Banning the wearing of the Islamic full veil in public would be "un-British", Immigration Minister Damian Green says.