The Doctor meets a hell of a lot of people while travelling the universe, some good, some bad, and some... a bit of both.
Because of this, the Doctor has to be incredibly selective when choosing individuals to become full-time TARDIS crew members, and while the correct decision isn't always made (cough Adam Mitchell cough), the Doctor has, for the most part, been sharing the TARDIS with some absolute gems, from 1963 to the present day.
But due to the constant amount of danger involved, journeying through time and space isn't for everyone, and over the years, quite a few characters have rejected the Doctor after being offered the chance to board the TARDIS.
We're not saying that these characters told the Doctor to 'eff off or anything, but for their own individual reasons, they all turned the Time Lord down, reluctant to live the life of an intergalactic adventurer. Whether it's a companion who doesn't want to carry on any longer, or a side character who refuses to transition to a more permanent position, not everyone is eager to travel the stars.