Whether Altmer themselves were the original inhabitants of Skyrim is not really important, because neither were the Nords, however Elves, in other words Mer, were the 1st inhabitants of Skyrim, if it was the dwemer for example they are elves and definitely more closely related to the Altmer than the Nords.
Again whether 1 race of the elves like the Altmer have the right to rule is debatable but I believe that collectively all Mer whether it be dwemer, famler, altmer, bosmer, dunmer etc have a greater claim to rule skyrim than all the races of men or either of the beast races.
The reason it is so often the Altmer is because they are the most powerful of the existing elven races in Skyrim no other race of elves could conquer tamriel, this is why the dominion exists to unite all mer and prove superiority over men, obviously there are differences in beliefs between the mer just like there are between the redguards and imperials and imperials and some nords but ultimately mer>>>Men
What makes the altmer the stronger race? What do you have that the dunmer for example don't?
What does an altmer posses that makes him superior over man, in any way? I mean sure, they have the power to talk to the divines directly, they are masters of magic from the day they are born, they have silent farts and blah blah blah, but that dosen't make them all-knowing, that dosen't make them invincible, as proved before. If they were invincible, Tiber Septim wouldn't have conquered them.
The argonians, for example, ar far superior to the altmer, they way I see it. I mean you can talk to a god that can tell you "jaja, keep worshiping me" , but the argonians can talk to trees who will tell them the fluffing future. That is what I call all-knowing. Also Black Marsh is the only part of Tamriel which was not completely invaded by Tiber Septim, which makes them superior not only to the elves, but to the men also.