I was wandering through the Elder Scrolls Wiki and I stumbled upon this gem.
http://elderscrolls.wikia.com/wiki/Snow_Elf
When the Atmorans arrived from Atmora, the Snow Elves struck up a somewhat comfortable relationship with the humans.[1] While at first peaceful, the Snow Elves soon discovered that the humans, though they lived shorter lives, grew in population rapidly. Seeing that the humans were rapidly beginning to outnumber the Elves in the land, the Snow Elves began to cull down the population en masse.[2]
As they regarded the humans as a lesser species, the elves had little qualms about the slaughter they intended to commit. This genocidal spree would be known as the Night of Tears.[2] However, this decision would cost them greatly, almost their entire society, as Ysgramor would survive and return to the original Nord homeland of Atmora to raise an army. Ysgramor and his army, known as the Five Hundred Companions, would return to Skyrim to take revenge on the Snow Elves. The series of events that would soon follow would be known as the Return, and with this return, a new war would begin.[3]
As time passed, the humans once again returned to Tamriel. The Nords of Atmoran lineage that had immigrated to Skyrim from Atmora had returned under the leadership of Ysgramor and his sons. An army of 500 men and women, known as the Five Hundred Companions, marched across Skyrim under Ysgramor's leadership to avenge the fallen. The elves would wage war with the humans for years, slowly losing to the very "creatures" they had culled in years prior.
The Snow Elves were broken and scattered following numerous losses to Ysgramor and the Companions and retreated to the island of Solstheim for safe harbor. Led by an elf only known as the Snow Prince, the Snow Elves took a final stand against the Nordic peoples, and, inspired by their leader, it seemed as though they'd come out victorious.
However, at the Battle of the Moesring, which was to be the last battle against the Nords, the Snow Prince was slain by a child by the name of Finna, who in her grief after the death of her mother, had stabbed him. Due to the death of the Snow Prince, the elves' hope was shattered, and a great deal of them attempted to flee the battle. It's not known what the exact date was, but it was sometime towards the end of the Merethic Era.[3][4]
With the elves finally broken once and for all, the Nords then mounted a genocidal campaign and killed Snow Elves by the thousands. The Snow Elves retreated underground where they sought refuge with the Dwemer.
The Dwemer agreed to grant them refuge, but at a high price. The Snow Elves were forced to eat toxic plants in order to survive underground. The toxins gradually destroyed their sight, and this loss was passed down to their offspring. The Dwemer then forced the blind Snow Elves into slavery.
Not all of the Snow Elves agreed to such a price. Splinter groups had formed to search for alternate solutions. These elves sought out other alliances, but, ultimately, most were slaughtered, vanished, or gave in and accepted the Dwemer's offers.
Over time, the Snow Elves began to devolve into twisted, blinded creatures, becoming the degenerate race now known as the Falmer. They acted as slaves to the Dwemer, and, over time, their history was lost as the race became more and more primal. Because of this, it is commonly believed that the Snow Elves were actually tricked into blindness and slavery, rather than having agreed to the situation to survive. This is debunked, however, upon speaking to Knight-Paladin Gelebor. He states that most Falmer agreed to the Dwemer's terms to avoid being eradicated by the Atmorans. Some pockets of resistance existed, but were eventually eliminated. Despite this, he believes there may be other isolated Snow Elf survivors scattered about Tamriel.
According to The Falmer: A Study, it was thought that the Dwemer betrayed the Snow Elves by force-feeding them a toxic fungus that inflicted blindness after they sought safe harbor in their lands. The poisoning effects of the fungi caused the Snow Elves to devolve into the twisted beings they have become.
However, according to Gelebor, this is not the case. Faced with near extinction, the Snow Elves and Dwemer struck up a deal: in exchange for safe harbor, the elves would have to ingest a toxin (likely from the mushrooms), which would blind them. With little option left, the elves accepted and became the Dwemer's slaves. Gelebor states that the toxin was not enough to de-evolve the Snow Elves as they are now, and as such it must have happened gradually as time passed.
(Following is an excerpt)
Frontier, Conquest
and Accommodation:
A Social History of Cyrodiil
Ysgramor was certainly not the first human settler in Tamriel. In fact, in "fleeing civil war in Atmora", as the Song of Return states, Ysgramor was following a long tradition of migration from Atmora; Tamriel had served as a "safety valve" for Atmora for centuries before Ysgramor's arrival. Malcontents, dissidents, rebels, landless younger sons, all made the difficult crossing from Atmora to the "New World" of Tamriel. New archeological excavations date the earliest human settlements in Hammerfell, High Rock, and Cyrodiil at ME800-1000, centuries earlier than Ysgramor, even assuming that the twelve Nord "kings" prior to Harald were actual historical figures.