Doctor Who: 10 Things We Actually Know About The Doctor's Family
10. The Doctor Didn't Grow Up With Money
The shabby old barn from The Day of the Doctor, Listen, and Hell Bent heavily implies that the Doctor didn't grow up with Time Lord status, or any other lordly status for that matter. After all, it's located pretty far from the Citadel, and it's not exactly glamorous.
In Listen we see a young Doctor sleeping in this barn, scared and alone and crying himself to sleep. This suggests that the Doctor's family wasn't wealthy and couldn't afford to send him to some nicer digs, or, at the very least, that the Doctor didn't grow up surrounded by wealth.
A man and a woman then enter the barn, and have a brief exchange. The woman invites the Doctor back into the house "with the other boys", while the man is pushing for the Doctor to join the army.
The man and the woman definitely aren't the Doctor's parents. The reference to "other boys" rather than "brothers" indicates that, like the young Rupert Pink, the Doctor was orphaned and then grew up in some kind of children's home.
It also sounds like the adults in the children's home were raising kids for service with Gallifrey's Chancellery Guard, setting up the Doctor's complicated relationship with the military and seeding his disdain for guns and soldiers.