Yes men have it harder, because men are more harshly selected in evolution. It is simply nature. It shouldn't be otherwise, but let's not pretend women have it harder than men.
The problem is that you are looking at both men and women from the perspective that nature has accorded to the woman's role.
In nature, men and women serve completely different purposes. Each one has completely different conditions for success and failure.
The female is historically a fixture of nature. Most female animals do not choose their sexual partners. Males fight, someone wins, and the female simply goes into estros and submits. The female carries the baby, cares for it for as long as it takes, a baby she did not choose (but nevertheless, typically, doesn't have any problem at all with). Meanwhile the male goes and does what he wants with other females, enjoying all the fruits of his capabilities. He owes little to her in return for what she gave to him.
Is that a good deal? As a matter of fact, I think this simplified and primitive scenario plays out more than one might think in the lives of women, even in this day and age. As difficult as it is for many people to accept, a woman only fractionally controls her own desire. It is contrary to her nature to approach men, and the question of her choice is only barely conscious or rational. She is far easier to dominate emotionally than a man, and she operates on very strong instincts that mimic many of the most prehistoric aspects of courtship. In relationships, she is far more submissive and he is far more dominant, and especially when a baby is involved, she is way more vulnerable as well.
Already, if we are talking about 'equality', we can see that a woman has to date up merely to balance out her relative lack of power (divorce courts and feminism notwithstanding). Also, a man can marry a woman who is unintelligent and weak, and they can both prosper, but the same is not true when it works the other way around. This makes the quality of both their lives far more dependent on how good a male he is, compared to how good a female she is. So it's not even fair to compare men and women at the same level of sexual value.
A man, on the other hand, can conquer other men, even countries. He can fight and compete to multiply his wealth and power many times over, and decide to whom he wants to give it. And, he can be a complete failure, wiped off the face of the earth by his own self-destruction or in competition with better men than himself. What happens is in his hands - he can wander the depths of hell or the heights of heaven, and choose who he brings, for better or worse. And when a woman marries him, none of that capability transfers to her - she only gets some of the rewards of it, at his discretion. And she pays the price equally, or more, when things go south.
So on one hand you are right, men have it 'harder', if you think of both men and women as being just 'people' with the same stuff inside them. But in truth, men and women are so varied that you can't really compare them. Is it better to be a leader or a follower? Dominant or submissive? For men and women, the answer to this question alone is often worth more than all the sexual selection in the world - and the answers are different, coming from different natures, different perspectives, and different natural roles.