The Big Bad of Season 4 (a.k.a. Arrow's worst season), Damien Darhk was thinly-written, full of corny dialogue and simply far too powerful for a show like Arrow. On top of that, his motivations were poor and predictable (he literally tried to blow up the world) and was responsible for one of the biggest mistakes the show had ever made (Black Canary's death).
In spite of an excellent charismatic performance from Neal McDonagh, it's hard to see the character as anything other than a major loss in the villains department. His magic and mysticism was everything Arrow was not, and resulted in the show losing the last remnants of that charm it once possessed - which is why it was so nice to see McDonagh thrive on Legends of Tomorrow when the character was moved there.