What caused it: The books, for no apparent reason.
In their second years and beyond, the students who survived a boat trip across a dangerous lake filled with mermaids and other monsters with only one chaperone get to travel up to the castle in magically led carriages. Except, as we learn in Order Of The Phoenix, they're actually pulled by Thestrals, bony, horselike things that apparently it's easier to keep all year round for an hour of action than, say, use magic.
Anyway, what makes these creatures particularly creepy is that you can only see them if you've seen someone die.
Which doesn't explain at all why Harry saw them as invisible before the fifth book; he's most well known for having seen someone die. We know Harry remembers Lily's death thanks to the recurring green flash he's had as a child. Do you have to know it was a death or have been affected by it? And even if you do, Harry still kinda fits into both categories.
The plot point was clearly introduced to hammer home how much Cedric's death had affected Harry, but when youre creating a massive hole to make emotion it isn't worth it.