Pippa probably does see George and Charlotte. However we don't see James or Pippa taking them anywhere, nor do we know whether they changed the babies' nappies.
Plus, when asked about them by Lauer she is hardly likely to say "Oh, yes, I see them now and again" is she? If both the younger Middletons see George and Charlotte every weekend, which I doubt, it's odd that George has only ever been seen on outings with Carole and Carole and Kate, not any other Middleton, not even grandad, however close they all are.
Pippa and James are on holiday at St Barts now, doing their own thing, just as Harry does his own thing with his friends. He didn't say he never saw his niece and nephew, just that he'd never babysat them, and we don't know that the Cambridges have ever asked him to. Perhaps Kate prefers her mother to do any babysitting required.
My adult son in his 30's has never babysat his baby twin niece and nephew who are 14 months old. That doesn't mean he isn't a devoted uncle, because he is. He goes over to his sister's house and plays with them. At Anmer and KP there is a nanny, housekeeper who does child-care duties and two parents and yet Harry is selfish and old-fashioned because he hasn't baby-sat? Oh, come on!!