It's tricky to pinpoint quite when the psychological thriller came into being. In cinematic terms, it might be dated back to 1960, with Michael Powell's Peeping Tom and Alfred Hitchcock's Psycho. That said, some of the down-and-dirty film noirs of the 1940s might also qualify.
Its literary origins might be traced back even further. In many respects, the writings of Edgar Allen Poe - Gothic shockers in which the object of horror is almost always the human mind - could be considered the psychological thriller's starting point.
In the 1990s, after the success of The Silence of the Lambs, the term 'psychological thriller' tended to be applied to just about any serial killer film that was just that bit grounded enough to sidestep the apparently undesirable label of horror; although there are clearly instances where the two greatly overlap, The Silence of the Lambs very much included.
However, as might be self-evident from the use of the word 'psychological,' the primary focus of the subgenre is the mental well-being - or rather, lack thereof - of the protagonists. Oftentimes, this reaches such a fever pitch that we're left uncertain as to what is or is not really happening.
In other words, the best psychological thrillers tend to be the movies which not only get under your skin, they also get into your head and start messing around in the best/worst way.