The relationship between the Fourth Doctor and Sarah Jane could be likened to that of a father/daughter in many aspects. He cared for her and protected her and when he had to part ways with her upon a summons back to Gallifrey, it visibly left him distraught. He had now not only parted with his granddaughter but a woman he treated like his own daughter as well. Thankfully, they were able to bring closure to this departure some 30 years (and half a dozen regenerations) later. It's never really the end, after all.