Streaming with New .NET HttpClient and HttpCompletionOption.ResponseHeadersRead

How to consume a streaming endpoint with new .NET System.Net.Http.HttpClient and the role of HttpCompletionOption.ResponseHeadersRead
2012-08-01 09:27
Tugberk Ugurlu


For a while now, I have been playing with a new application I created for fun: TweetMapR and I am not sure where it is going :) The application itself consumes Twitter Streaming API and broadcasts the retrieved data to the connected clients through SignalR. I used new HttpClient to connect to Twitter Streaming API and I used OAuth as authentication protocol. In fact, I created my own OAuth Twitter client to connect to Twitter which is so raw right now but works well.

But how to send a request to Twitter Streaming API and keep it open infinitely was my main question for the whole time. So, I brought up the ILSpy and decompiled the System.Net.Http.dll to see what is really going on and I realized that it was going to be so easy. First of all, if your request is a GET request, you are covered by the framework. The GetStreamAsync method of the HttpClient sends a GET request and returns back the stream as soon as it completes reading the response headers. The following example code shows the usage.

using (HttpClient httpClient = new HttpClient()) {

    httpClient.Timeout = TimeSpan.FromMilliseconds(Timeout.Infinite);
    var requestUri = "http://localhost:6797";
    var stream = httpClient.GetStreamAsync(requestUri).Result;

    using (var reader = new StreamReader(stream)) {

        while (!reader.EndOfStream) { 

            //We are ready to read the stream
            var currentLine = reader.ReadLine();
        }
    }
}

First of all, don’t use Result on Task :) It blocks but I used it here for the sake of simplicity to stick with the main point of this post :) Assuming that the localhost:6797 is a streaming endpoint here, the above code should work perfectly fine and shouldn’t be timed out as we set the Timeout to System.Threading.Timeout.Infinite. But what if the request we need to send is a POST request? In that case, we have a little more work to do here.

If we just try to send a POST request with the PostAsync method or any of its variants, the response will never come back unless the server ends the request because it will try to read the response till the end. In order to omit this problem, we can pass a HttpCompletionOption enumuration value to specify the completion option. The HttpCompletionOption enumeration type has two members and one of them is ResponseHeadersRead which tells the HttpClient to only read the headers and then return back the result immediately. The following code shows a sample example where we need to send a form-urlencoded POST request to a streaming endpoint.

using (HttpClient httpClient = new HttpClient()) {

    httpClient.Timeout = TimeSpan.FromMilliseconds(Timeout.Infinite);
    var requestUri = "http://localhost:6797";

    var formUrlEncodedContent = new FormUrlEncodedContent(
        new List<KeyValuePair<string, string>>() { 
            new KeyValuePair<string, string>("userId", "1000") });

    formUrlEncodedContent.Headers.ContentType = 
        new MediaTypeHeaderValue("application/x-www-form-urlencoded");

    var request = new HttpRequestMessage(HttpMethod.Post, requestUri);
    request.Content = formUrlEncodedContent;

    var response = httpClient.SendAsync(
        request, HttpCompletionOption.ResponseHeadersRead).Result;
    var stream = response.Content.ReadAsStreamAsync().Result;

    using (var reader = new StreamReader(stream)) {

        while (!reader.EndOfStream) { 

            //We are ready to read the stream
            var currentLine = reader.ReadLine();
        }
    }
}

Again, don’t use Result as I do here :) We have obviously more noise this time but if this is something that you will use often, there is nothing stopping you to write an extension method.



Comments

Ebrahim Byagowi
by Ebrahim Byagowi on Wednesday, Aug 01 2012 12:42:06 +03:00

Hi. Thanks for your great post :)

I think you could read/link from/to Mono GitHub sources instead of MSDN documentations (for GetAsyncStream https://github.com/mono/mono/blob/master/mcs/class/System.Net.Http/System.Net.Http/HttpClient.cs#L313 )

Thanks again! :)

Tugberk
by Tugberk on Wednesday, Aug 01 2012 21:04:32 +03:00

@Ebrahim that's certainly not the same implementation as the actual dll. I will stick with decompilers. thanks, though.

Salman Farsi
by Salman Farsi on Monday, Aug 06 2012 18:40:51 +03:00

Hi, thanks for great post, can you please upload the source code.

Tugberk
by Tugberk on Tuesday, Aug 07 2012 07:25:07 +03:00

@Salman

the source code of the TweetMapR project is up on GitHub:

https://github.com/tugberkugurlu/tweetmapr

Jason
by Jason on Wednesday, Aug 28 2013 04:16:35 +03:00

Small typo in your POST example:  "HttpMethod.Get" should be "HttpMethod.Post".

Tugberk
by Tugberk on Wednesday, Aug 28 2013 13:02:02 +03:00

@Jason

fixed it, thx!

Ashok Bhardawaj
by Ashok Bhardawaj on Friday, Sep 13 2013 09:31:10 +03:00

I am looking for an example for twitter streaming api using twitterizer.

 

metz2000
by metz2000 on Thursday, May 29 2014 21:08:31 +00:00
If using Result is wrong why is it used in a sample code what will be copied verbatim by hundreds of copycoders? Sample should use the right method what I assume is await in an async method.

New Comment