A towering influence
My starting point for Lovecraft was Vintage's anthology - The call of Cthulhu and other weird tales. Besides Call of Cthulhu, the volume also includes The rats in the walls (one of the few Lovecraft tales that I had read previously, though I had forgotten quite how horrifying it was), The Dunwich horror, The nameless city, The shadow over Innsmouth, The whisperer in darkness, The festival, At the mountains of madness, as well as The picture in the house, The thing on the doorstep and The haunter of the dark.
The first group (with the exception of Rats in the walls) are a mix of sci-fi and horror as alien life forms come down to earth are either inadvertently brought back to life, or, suddenly decide to take over life on earth. This could be a recipe for extreme daftness (see, for example, my review of the heavily Lovecraft influenced Kraken by China Mieville); that it isn't, that it manages both to be scary and thoroughly convincing is down to the sheer skill of Lovecraft's writing. Part of this is the way in which his stories are narrated. Virtually all of them are told in the first-person, an outsider, who has somehow become involved in extraordinary events. He is as likely as you the reader to be suspicious of the events that he has become embroiled in, and will often have the same reservations. There was an extraordinary moment in one of the tales where I started to wonder why the narrator was behaving the way he was, only for the narrator to explain himself in the next sentence - it's quite rare to feel that you, the reader, are having an intercourse with the central character; but Lovecraft does this brilliantly.
Despite the very firm suspension of disbelief necessary for these tales, Lovecraft unerringly draws his reader in. The shadow of academe that wraps itself around most of his tales, and (like Jules Verne) his love for science, albeit of a rather darker variety, make these tales oddly believable. He truly is the father of just about every sci-fi horror film from Quatermass and Doctor Who to The Thing and Alien, Invasion of the Body Snatchers and Doomwatch. My favourite of the "thing from Outer Space"-type stories was the wonderful At the mountains of madness, a fabulous tale of Antarctic exploration gone wrong, and of thwarted contact with aliens.
Of the more "usual" (as usual as you ever get with Lovecraft) horror stories, The rats in the walls is the scariest and most blood curdling, but the cleverest and my particular favourite was The thing on the doorstep.
Lovecraft is a truly brilliant, unsettling writer. Yes, he's a writer of pot-boilers, but what a writer. Oddly enough I wasn't particularly scared by his writing, it didn't affect me in the way that M.R. James or Edgar Allen Poe would do, but I found his dark, eerie world completely captivating and compelling. Much of Lovecraft's work is now available online for free. An exceptionally good source is The H.P. Lovecraft Archive.