In "The Listeners" who is the traveller expecting to meet in the house?

The difficult thing about this poem is that it does not provide a lot of clear information, instead leaving most of it up to the reader to guess at what might have been going on.  While this enigmatic trait makes the poem difficult, it also lends it quite a bit of appeal, making it more intriguing, interesting and thought-provoking.  If the author had just given us all of the details, and filled in all of the gaps, then nothing would be left up to the imagination.  Instead, we are left with very little information, which injects our imaginations with all sorts of possibilities.  It makes it a much more reader-involved poem.

If we look at the text, the only information we get about who the traveller might be expecting is the fact that he went to a very particular house--it was pretty obvious that he knew that was the house he was supposed to go to.  So either he was given very specific instructions on where to go, or, he had been to the house before, and knew who owned it.  However, when he gets to the house, he doesn't go in.  He is perplexed that no one answers, but, he doesn't go in to double-check.  This implies that he was expecting someone to be there, and to answer.  If he hadn't expected and answer, he wouldn't have been bothered that no one did.  He knocks several times, expecting a response.  Then, it gets to him:

"He felt in his heart their strangeness,
Their stillness answering his cry,...
For he suddenly smote on the door, even
Louder, and lifted his head:--
'Tell them I came, and no one answered,
That I kept my word,' he said."

Here, we get more clues to his expectations.  First of all, he senses that someone is there, listening.  He then shouts his message, referring to "They."  So, he was expecting not one person, but several, and these people were important enough to inform that he had kept his word to them.

So, we don't know exactly who he expected.  We can infer that it was a group of people that he had promised something to, and to whom it was very important to deliver the message that he had remained true.  For the rest--what the promise was, what he was on a journey doing, who these mysterious people were, and who the "phantom listeners" were--is left to our imagination.  I hope that helped; good luck!