Whilst it's clear that Norman is mentally unstable (what with all of the blackouts and murders), the cross dressing becomes so much more than him being confused about his sexuality. He isn't just trying on Norma's clothes for the thrills of because he feels transgender, but because he legitimately believes that he and Norma are the same person. The scene in which he starts making breakfast for Dylan and talks about himself as his mother is simply disturbing. He becomes the version of Norma that he believes her to be, so when you add that on top of the hallucinations and Norman being convinced that his mother is actually the true killer, serious alarm bells start ringing.
