Doctor Who: Every Doctor's Darkest Moment

The Fourth Doctor era is one of the darkest in all of Doctor Who, garnering multiple complaints from the puritanical Mary Whitehouse. And yet, Tom Baker's beloved incarnation of the Doctor largely keeps his nose clean.

Harrison Chase gets ground into compost but that wasn't the Fourth Doctor's fault, and he did tell Sarah to shut the machine off. Vynda-K gets a bomb placed in his coat, but the Doctor was just playing on the villain's greed, really.

Isn't the Fourth Doctor's darkest moment his decision to not destroy the Daleks at their inception? For all the Doctor's moralising about how the universe could become a better place by uniting against the Daleks, isn't it also true that he doomed countless lives in the process?

As Sarah tells the Doctor in Genesis of the Daleks, he can't doubt that the universe would be better without the Daleks. For all his talk about how the universe needs a common threat to stand against, wouldn't it be nicer if there was no threat at all?