I would say the British and the Norwegians are tied.
Wikipedia dates the Norwegians to about 872, with the unification of Norway and the rule of Harald Fairhair.
The English monarchy became clearly English around the same time with the reign of Alfred the Great and, more importantly, his son Edward the Elder - so between about 880 and 925. Their family had ruled Wessex since (according to the Anglo-Saxon Chronicle) the reign of Cedric in 519.
The first Scottish monarch, however, was Kenneth MacAlpin whose reign began around 858. Given that the current British monarchy is the result of a union between the English and the Scottish kingdoms (and the English kingdom is in many ways the result of the Wessex conquest of the rest of England...) it's kind of hard to say just when the British monarchy began. And that's without getting into the Dukes of Normandy, from whom the Queen's claim to the throne descends. William the Conqueror didn't arrive on the scene until 1066, but his house had ruled Normandy since 911.