By Tomas Chamorro-Premuzic* 

The real gender issue isn’t a lack of qualified women, but a surplus of unqualified men.

There are three popular explanations for the clear under-representation of women in management, namely:

(1) they are not capable

(2) they are not interested

(3) they are both interested and capable but unable to break the glass-ceiling

an invisible career barrier, based on prejudiced stereotypes, that prevents women from accessing the ranks of power. Conservatives and chauvinists tend to endorse the first; liberals and feminists prefer the third; and those somewhere in the middle are usually drawn to the second. But what if they all missed the big picture?

In my view, the main reason for the uneven management sex ratio is our inability to discern between confidence and competence. That is, because we (people in general) commonly misinterpret displays of confidence as a sign of competence, we are fooled into believing that men are better leaders than women. In other words, when it comes to leadership, the only advantage that men have over women (e.g., from Argentina to Norway and the USA to Japan) is the fact that manifestations of hubris — often masked as charisma or charm — are commonly mistaken for leadership potential, and that these occur much more frequently in men than in women.

This is consistent with the finding that leaderless groups have a natural tendency to elect self-centered, overconfident and narcissistic individuals as leaders, and that these personality characteristics are not equally common in men and women. In line, Freud argued that the psychological process of leadership occurs because a group of people — the followers — have replaced their own narcissistic tendencies with those of the leader, such that their love for the leader is a disguised form of self-love, or a substitute for their inability to love themselves. “Another person’s narcissism”, he said, “has a great attraction for those who have renounced part of their own… as if we envied them for maintaining a blissful state of mind.”

The truth of the matter is that pretty much anywhere in the world men tend to think that they that are much smarter than women. Yet arrogance and overconfidence are inversely related to leadership talent — the ability to build and maintain high-performing teams, and to inspire followers to set aside their selfish agendas in order to work for the common interest of the group. Indeed, whether in sports, politics or business, the best leaders are usually humble — and whether through nature or nurture, humility is a much more common feature in women than men. For example, women outperform men on emotional intelligence, which is a strong driver of modest behaviors. Furthermore, a quantitative review of gender differences in personality involving more than 23,000 participants in 26 cultures indicated that women are more sensitive, considerate, and humble than men, which is arguably one of the least counter-intuitive findings in the social sciences. An even clearer picture emerges when one examines the dark side of personality: for instance, our normative data, which includes thousands of managers from across all industry sectors and 40 countries, shows that men are consistently more arrogant, manipulative and risk-prone than women.

The paradoxical implication is that the same psychological characteristics that enable male managers to rise to the top of the corporate or political ladder are actually responsible for their downfall. In other words, what it takes to get the job is not just different from, but also the reverse of, what it takes to do the job well. As a result, too many incompetent people are promoted to management jobs, and promoted over more competent people.

Unsurprisingly, the mythical image of a “leader” embodies many of the characteristics commonly found in personality disorders, such as narcissism (Steve Jobs or Vladimir Putin), psychopathy (fill in the name of your favorite despot here), histrionic (Richard Branson or Steve Ballmer) or Machiavellian (nearly any federal-level politician) personalities. The sad thing is not that these mythical figures are unrepresentative of the average manager, but that the average manager will fail precisely for having these characteristics.

In fact, most leaders — whether in politics or business — fail. That has always been the case: the majority of nations, companies, societies and organizations are poorly managed, as indicated by their longevity, revenues, and approval ratings, or by the effects they have on their citizens, employees, subordinates or members. Good leadership has always been the exception, not the norm.

So it struck me as a little odd that so much of the recent debate over getting women to “lean in” has focused on getting them to adopt more of these dysfunctional leadership traits. Yes, these are the people we often choose as our leaders — but should they be?

Most of the character traits that are truly advantageous for effective leadership are predominantly found in those who fail to impress others about their talent for management. This is especially true for women. There is now compelling scientific evidence for the notion that women are more likely to adopt more effective leadership strategies than do men. Most notably, in a comprehensive review of studies, Alice Eagly and colleagues showed that female managers are more likely to elicit respect and pride from their followers, communicate their vision effectively, empower and mentor subordinates, and approach problem-solving in a more flexible and creative way (all characteristics of “transformational leadership”), as well as fairly reward direct reports. In contrast, male managers are statistically less likely to bond or connect with their subordinates, and they are relatively more inept at rewarding them for their actual performance. Although these findings may reflect a sampling bias that requires women to be more qualified and competent than men in order to be chosen as leaders, there is no way of really knowing until this bias is eliminated.

In sum, there is no denying that women’s path to leadership positions is paved with many barriers including a very thick glass ceiling. But a much bigger problem is the lack of career obstacles for incompetent men, and the fact that we tend to equate leadership with the very psychological features that make the average man a more inept leader than the average woman. The result is a pathological system that rewards men for their incompetence while punishing women for their competence, to everybody’s detriment.

* Tomas Chamorro-Premuzic is the CEO of Hogan Assessment Systems, a Professor of Business Psychology at University College London, and a faculty member at Columbia University.