There's a scene at the start of the Age of Ultron trailer that shows some civilians looking decidedly p***ed off. Even after Ultron is defeated, we can hardly see the Avengers being the toast of the world's population, given that it was Tony Stark who created the robot villain in the first place. As a result, the public are still going to be extremely angry with the Avengers and the superhuman community in general. It will likely be the trigger to Civil War in the first place and we can certainly envisage that Captain America and Iron Man et al will be receiving a hard time from the general population come the time for Civil War.
