Doctor Strange: 31 Easter Eggs & References You Must See
15. The Avengers' Role In The World
The second mention of the Avengers comes as a brief but clever explanation of why the Sorcerors wouldn't have turned up to help with the Incident or any other major attack on the real world. They don't intervene simply because it isn't their remit.
Wong tells Doctor Strange that while the Avengers "protect the world from physical dangers. We safeguard it against more mystical threats."
It's only now that more magical threats like Loki and Thanos have appeared that the sorcerors have broken cover. (Except both of them have been around for loads of films already).