The Imperial Household Agency exists to support the Emperor and the Imperial Family. In most cases it seems to do a pretty good job: the Emperor gets to where he has to go, his dealings with the government are cordial, official visits take place smoothly, the Imperial residences are maintained, the various behind-the-scenes Shinto rituals are performed, and the Empress's silk worms are fed. It is, no doubt, staffed by men and women deeply loyal to the Imperial Family.
There are 18 adult members of the Imperial Family. Of these, one or two seem to have hinted at dissatisfaction with the Imperial Household Agency. But it seems a bit of an over reaction to take this as proof of cabal of manipulative men, scheming away in the background, determined to regulate the Imperial Family’s every move. Who cares about minor details like facts and verifiable evidence? Let’s focus on gossip and rumours instead. Throw in the old cliché of faceless men (in grey suits of course) and we have well and truly passed into the realm of fiction.
If the Crown Princess is coming to terms with the Imperial Household Agency, maybe it's time for those beyond the Imperial Palace walls to try as well.