A women’s college which admits only female students is not discriminating unlawfully against men.
If the college admits a small number of men to make up the numbers on a particular course of study, it is still regarded as a single-sex college. It is not discriminating unlawfully by refusing to admit men to other courses.
A women’s college which admits men to certain courses but refuses to let them use the student cafeteria would be discriminating unlawfully against them.