Yeah, that's the thing about this game that makes it interesting. You go around thinking you're top of the food chain, then suddenly you come upon enemies that totally kick your tail! But these experiences forces you to refine your character's strategies.
Another group to stay away from when you're at low levels are the Mammoth Camps. Although its kinda cool if you can get dragons to attack a mammoth camp. Sorry to hear about the wandering Khajitas though. There are some you need alive for certain sidequests. Oddly in my first play-through, dragons seemed to target blacksmiths of whatever town they attacked. That kinda sucked.