What do the words “literary horror” mean to you?
Well, I don't really have strict definitions of either of those words - if you try and define anything you end up missing things out, in my opinion. I guess in very general terms 'literary' might mean a story where formal, thematic, or stylistic concerns are of equal weight as the actual plot. And equally generally 'horror' might mean any story with an intention to scare, unnerve, freak out, or disturb the reader in some way. If you drew a Venn diagram of those two groups, then the overlap is 'literary horror.' But given that such a diagram is impossible to actually draw, you might not consider that I've actually answered the question...
Look, it's Kafka, alright? It's Ligotti and Du Maurier and William Hope Hodgson. That kind of people.