This explains a lot! http://www.wired.com/wiredscience/2011/07/neanderthal-human-mating/

This is an excerpt from this article:

“In a paper published in the July issue of the journal Molecular Biology and Evolution, the geneticists write: “We provide evidence of a notable presence (nine percent overall) of a Neanderthal-derived X chromosome segment among all contemporary human populations outside Africa”. The team analyzed 6,092 X-chromosomes “from all inhabited continents”. Their discovery confirms earlier hypotheses that early modern man and Neanderthals mixed and mated.”

Go to the link to read this entire article