Poor James: he was basically used by everyone in the show for different reasons and it's no shock that Norman wasn't his biggest fan when he found out that he had been sleeping with his mother. On top of Norman's obsessive and borderline sexual thoughts about his mother, his natural instinct to attack James after learning about their intimate encounters was enough for James to run away. But that wasn't the end of it. James' relationship with Norma basically cost him his health: targeted by Bob Paris and his thugs, James is tortured for information about Norma and Norman. The poor psychology professor got quite a beating, enough to make him leave town and it's not even like his character did anything to deserve this - but maybe that's the point.
