In Chupacabra (S2 E05), Rick and Daryl enlist the help of one of the farm hands as well as one of Hershels horses to search for Sophia. When they return, Hershel tells them through steely eyes to respect his property, demanding they ask his permission to use any of it in the future. To punctuate his point, he tells them to leave his farm as soon as they get on their feet. It may seem like Hershel is just being a studious landlord, but when you consider his position, he is actually much more than that. After Otis dies, Hershel is the only remaining grown man on the farm. Hes got a young son and a farm hand, but, for the most part, its just Hershel and his daughters living on the farm. If the group wanted to overrun the place, they could easily do so. But no one is willing to challenge him face-to-face hes like the tired old leader in a wolf pack; its his presence, not his strength that keeps an enemy at bay. Hes got the aura of a badass. In season 2, the group was in Hershels territory, and he wasn't afraid to let them know it. He was a Machiavellian Prince on the farm, both loved and feared by the group. They were grateful for his protection, but they were always walking on egg shells around him.