I think it has to do with, for me, that I desire the men (being straight) while place myself in the woman's shoes.

I think you hit it right on the money. We tend to forgive the object of our desire his mistakes and quirks, but criticize the person we project ourselves onto for every mis-step.