Ollys family and the rest of his village were murdered by wildlings. They had no real motive as it was a peaceful village. Tormund Giantsbane led the raid, and Olly doesnt understand how Jon could possibly put trust in such a man, or how he could even comprehend risking lives to save wildlings. Sam, who is now perhaps the wisest man left at Castle Black, tries to reason with him, but its to no avail. You'd listen, Olly, if you could see what Jon Snow was facing - youd bloody well trust him. He cant though, and that mistrust seeded within Olly and Alliser Thorne and various other members of the Nights Watch continues to grow. Jons on his way back to The Wall, and hes going to have some rather more human problems to face up to.