I agree that it is is probably to do with James' age, and from the images from the Horse Show he can't even sit through that! What a dude. Last year when Sophie was attending an engagement a fan of the family said that soon James would be attending Trooping the Colour and Sophie, and the man behind her, both winced! I reckon James is perhaps a bit of a trouble maker/typical little boy who loves to run around and does not enjoy sitting still. Little boys are a lot different than little girls, I have to say. Louise attended Peter's wedding when she was 4 1/2, she was obviously able to sit through the ceremony, but she is a little girl who was probably fascinated by the lights and pretty dresses, and the story of "We got married her too". I often wondered why Louise attended that wedding at the age of 4, yet James does not even attend Easter, but after thinking about it, he is a boy who would want to run around.
I DO have hopes that James will make an appearance on the balcony this year, as Louise did that when she was 5 1/2. It's not every year your grandmother celebrates 60 years as a Monarch, and these celebrations will likely not happen again for a long time. I hope we do see the four of them on the balcony, it would be so cute.