Claudio Ranieri is known as the Tinker Man, and since he won the league with Leicester City, this ideology clearly has merits in the world of football.
In FM18, it's important to master at least two, ideally three, formations. Not only does this prevent you from becoming predictable in the eyes of the opposition, it will give you the chance to switch things up mid-match if the game isn't going your way.
If you lose playing 5-3-2 after switching from 4-4-2, don't be discouraged. It will take your players time to adapt to each system, but it pays to invest in this in the long run.
Tweaking your lineup, instructions and player roles from the touchline during games is important if it's a mess out there on the pitch, although keep in mind that bombarding your squad with too many instructions only makes things worse.