Wills, 31, is believed to have left wife Kate Middleton and their son at her parents' house in Bucklebury, Berkshire, before travelling to nearby Ascot.
While at the event, the new dad received a miniature polo mallet for George and, moments before he was handed his gift, he jokingly asked if it was nappies.
Prince William added: "I've got to pick some up on my way home!"
When speaking about the match he had just played alongside his brother Harry, Wills also admitted: "I was in baby mode out there, thinking about nappies, I wasn't really in the zone."
Speaking in more detail about his new baby, William also said: "He likes to move around a lot, he wriggles a lot, he keeps on our toes."
Proud parents William and Kate welcomed George into the world on Monday 22 July.
After posing for their first family photograph on the steps of St Mary's Hospital in London, the couple briefly returned to Kensington Palace before heading to Kate's parents' house.
Chatting to guests yesterday at the Audi Polo Challenge, William said: I've got to make sure I'm not late back."
It has since been revealed that William is to return to his search and rescue job with the RAF following his two-week paternity leave.
The Prince received a statutory paternity pay of £137 a week while he was off work and it is thought he will rejoin the Household Calvary Blues and Royals Regiment at its Windsor or Knightsbridge barracks.
Prince William, Kate Middleton and Prince George leave hospital in pictures: