How come the Starks are so few if they have been ruling for thousands ...

How come the Starks are so few if they have been ruling for thousands ...