Iyengars are essentially people who are called 'Sri Vaishnavas'. The word Sri Vaishnava means "the one who belongs to Sri Vishnu". So, the question essentially means to ask "why do not those married ladies live with other men rather than with their own husbands?" The answer is simple: It is because they are meant to live only with/for their husbands (and vice versa). While they do not hold any grudge or hatred towards other males, they do not also indulge in unnecessary talks/movements with them either.
Very similarly, Sri Vaishnavas are considered as wedded to Sriman Narayana on the day they have surrendered to Him (through panchasamskaram). Having been wedded to Him, how would they even think of others (called 'demigods') even in the wildest of their dreams, if they were to be 'chaste'? It does not mean they hate the others or belittle them in any way.
Rather, Sri Vaishnavas 'neither praise the others,
nor curse them'. They are just indifferent to
all others, except their own Lord or His servitors. For that matter, not just
Ganesha but Iyengars do not care about any demigod including Brahma, Shiva, Indra,
et al.
