Sorted By: Month (5) and Year (2014)

Efficiently Streaming Large HTTP Responses With HttpClient

Downloading large files with HttpClient and you see that it takes lots of memory space? This post is probably for you. Let's see how to efficiently streaming large HTTP responses with HttpClient.
2014-05-11 12:56
Tugberk Ugurlu


I see common scenarios where people need to download large files (images, PDF files, etc.) on their .NET projects. What I mean by large files here is probably not what you think. It should be enough to call it large if it’s 500 KB as you will hit a memory limit once you try to download lots of files concurrently in a wrong way as below:

static async Task HttpGetForLargeFileInWrongWay()
{
    using (HttpClient client = new HttpClient())
    {
        const string url = "https://github.com/tugberkugurlu/ASPNETWebAPISamples/archive/master.zip";
        using (HttpResponseMessage response = await client.GetAsync(url))
        using (Stream streamToReadFrom = await response.Content.ReadAsStreamAsync())
        {
            string fileToWriteTo = Path.GetTempFileName();
            using (Stream streamToWriteTo = File.Open(fileToWriteTo, FileMode.Create))
            {
                await streamToReadFrom.CopyToAsync(streamToWriteTo);
            }

            response.Content = null;
        }
    }
}

By calling GetAsync method directly there, we are loading every single byte into memory. You can see this happening in a simple way by opening the Task Manager and observing the memory of the process.

2

We are calling ReadAsStreamAsync on HttpContent after the GetAsync method is completed. This will just get us the MemoryStream, so there is no point there:

Screenshot 2014-05-11 15.18.14

We need a way not to load the response body into memory and have the raw network stream so that we can pass the bytes into another stream without hitting the memory too hard. We can do it by just reading the headers of the response and then getting a handle for the network stream as below:

static async Task HttpGetForLargeFileInRightWay()
{
    using (HttpClient client = new HttpClient())
    {
        const string url = "https://github.com/tugberkugurlu/ASPNETWebAPISamples/archive/master.zip";
        using (HttpResponseMessage response = await client.GetAsync(url, HttpCompletionOption.ResponseHeadersRead))
        using (Stream streamToReadFrom = await response.Content.ReadAsStreamAsync())
        {
            string fileToWriteTo = Path.GetTempFileName();
            using (Stream streamToWriteTo = File.Open(fileToWriteTo, FileMode.Create))
            {
                await streamToReadFrom.CopyToAsync(streamToWriteTo);
            }
        }
    }
}

Notice that we are calling another overload of the GetAsync method by passing the HttpCompletionOption enumeration value as ResponseHeadersRead. This switch tells the HttpClient not to buffer the response. In other words, it will just read the headers and return the control back. This means that the HttpContent is not ready at the time when you get the control back. Afterwards, we are getting the stream and calling the CopyToAsync method on it by passing our FileStream. The result is much better:

3

Resources

Securing MongoDB Access with Username and Password

My MongoDb journey continues :) and I had my first attempt to put a username and password protection against a MongoDB instance. It went OK besides some hiccups along the way :) Let's see what I did.
2014-05-01 10:44
Tugberk Ugurlu


My MongoDb journey continues :) and I had my first attempt to put a username and password protection against a MongoDB instance. It went OK besides some hiccups along the way :) Let's see what I did.

First, I downloaded the latest (v2.6.0) MongoDB binaries as zip file and unzipped them. I put all MongoDB related stuff inside the c:\mongo directory for my development environment on windows and the structure of my c:\mongo directory is a little different:

Screenshot 2014-04-30 13.50.16

In order to set up the username and password authentication, first I need to get the mongod instance up and running with the authorization on. I achieved that by configuring it with the config file. Here is how it looks like:

dbpath = c:\mongo\data\db
port = 27017
logpath = c:\mongo\data\logs\mongo.log
auth = true

With this config file in place, I can get the mongod instance up:

image

First, I need to connect to this mongod instance and create the admin user. As you can see inside my config file, the server requires authentication. However, there is a localhost exception if there is no user defined inside the system. So, I can connect to my instance anonymously (as I’m running on port 27017 on localhost, I don’t need to define anything while firing up the mongo shell):

image

All great! Let’s create the system user administrator. As everything else, this chore is nicely documented, too:

use admin
db.createUser(
  {
    user: "tugberk",
    pwd: "12345678",
    roles:
    [
      {
        role: "userAdminAnyDatabase",
        db: "admin"
      }
    ]
  }
)

We are pretty much done. We have a user to administer our server now. Let’s disconnect from the mongo shell and reconnect to our mongod instance with our credentials:

mongo --host localhost --port 27017 -u tugberk -p 12345678 --authenticationDatabase admin

image

I’m all there and I can see what my privileges at the server with this user are.

If you try to connect to this MonogDB server anonymously, you will see that you are still able to connect to it. This’s really bad, isn’t it? Not at the level that you think it would be at. The real story is that MongoDB still allows you to connect to, but you won’t be able to do anything as the anonymous access is fully disabled.

image

The bad thing here is that your server existence is exposed which is still an important issue. Just be aware of this fact before getting started.

The user we created still has restricted access to MonogDB server. If you want to have a user with unrestricted access, you can create a user with root role assigned. In our case here, I will assign myself the root role:

use admin
db.grantRolesToUser("tugberk", ["root"])

Resources

Tags