A couple of suggestions for reading Lovecraft for the first time...
Some of his earlier work is horrid. "The Street" and "Facts Concerning the Late Arthur Jermyn and His Family" come to mind. He was a very
well read man but never finished high school so his early work lacks discipline. Don't be put off by these stories. Furthermore, some of these early stories(and some of his later ones) have strong racial overtones. His opinions on other races become pretty clear if you read all his work (and crystal clear if you read his letters).
Furthermore, a lot of his stories can't even be called works of horror (he himself referred to all his work as "weird fiction"). He was strongly influenced by Lord Dunsany so there is a very dream-like quality to some of his stories. These are the stories you'll often hear referred to as being a part of his "dream cycle". Stories like "The White Ship" and "The Quest of Iranon" are strong examples of this. These may not be to your liking, so again, don't be put off by them.
In short - Lovecraft wrote some bad stories. He also created some of the greatest works of horror in American literature. You just have to know what to read.
I'd like to add some other reading suggestions in addition to those already mentioned.
"The Music of Eric Zann" is one of the best of his early stories. It suffers from his still decadent prose but that prose also helps to create a fantastic
mood and setting. Other good early stories include "The Temple" and "The Statement of Randolph Carter". The latter one of those is sometimes referred to as his first truly "Lovecraftian" tale.
I'm surprised that no one has mentioned "The Rats in the Walls". He wrote it around the middle of his career, and is probably the best story he wrote before he moved to New York. It is absolutely horrific and has one of my favorite endings. Also a great introduction. The whole story is fantastic. Read it.
"The Colour out of Space" is, in my own opinion, his greatest work. It's not my favourite but I think it is the best example of the cosmic horror he tried convey. He himself thought it was his best work. The horrific parts of the story are very different from what we see in horror these days. You might say that for all of Lovecraft's work, but like I said it's the best example of this.
As for other recommendations, the H.P. Lovecraft Literary Podcast
is a favourite of mine. They are currently working their way through every one of his stories, including his collaborations. A lot of fun and very informative.
The H.P. Lovecraft Historical Society
(which has close ties with the aforementioned podcast) has a lot of great Lovecraft material as well. Their production of The Call of Cthulhu
is the best I've seen, and they have a production of The Whisperer in Darkness
that they will release on DVD soon.