Here we have another instance of The Beatles doing a song that had potential, but just couldn't get the job done. All I've Got To Do was the second track off of the With The Beatles album, slowing things down after the fast and raucous It Won't Be Long. Let's just say thank the rock gods for the rest of the record, otherwise fans may have fallen asleep after this track.
Now, Lennon was influenced heavily by Smokey Robinson & The Miracles' soulful style, and it's clear he was indeed trying to channel that feeling in this song. The problem is that unlike Robinson, Lennon simply cannot channel that bluesy, jazzy feeling naturally. He may have been able to in other tracks, but All I've Got To Do is not one of them.
Needless to say, With The Beatles easily would have been just as strong an LP without this song.