This year, I decided not to miss on awesome developer conference NDC in Oslo and well, I attended! I'm actually still in Norway and I can tell that by looking at my pocket. I'm basically broken in terms of money :) Don't go to conferences if the city is ranked at the top of the World's most expensive cities (kidding, take a loan and go to NDC). If you miss this awesome developer conference, don't worry that much. The videos are already available online and I'm actually watching couple of the ones that I missed.

During the event, a few things were highlighted by many people over and over again which also made sense to me. Since this was a developer conference crowded by the World's top notch software developers, these are worth pointing out as bullet points for developers like me:

  • Mobile matters a lot (kind of obvious but worth highlighting it)!
  • JavaScript is big! No matter what programming language you use and what kind of development you do, it's going to end pretty badly for you if you keep ignoring JavaScript.
  • Learn a functional programming language. If you are a developer, you should (and I should) learn a functional programming language. It's kind of unavoidable in a World where concurrency matters this much.
  • Don't develop for the management, develop for the users of your product and your team mates (current and future).
  • SQL is still there but practically dead (there I said it!).
  • This is only what I felt during these days: you cannot survive in a software industry if you only know one general purpose programming language (well, you probably will but I don't think you will really *survive*. Catching my drift?).
  • Swift (not Taylor Swift) is a joke but it will be loved.

These are just my thoughts during these 3 days and the time will tell us whether those are actually valid points.

Efficiently Streaming Large HTTP Responses With HttpClient

Downloading large files with HttpClient and you see that it takes lots of memory space? This post is probably for you. Let's see how to efficiently streaming large HTTP responses with HttpClient.
2014-05-11 12:56
Tugberk Ugurlu


I see common scenarios where people need to download large files (images, PDF files, etc.) on their .NET projects. What I mean by large files here is probably not what you think. It should be enough to call it large if it’s 500 KB as you will hit a memory limit once you try to download lots of files concurrently in a wrong way as below:

static async Task HttpGetForLargeFileInWrongWay()
{
    using (HttpClient client = new HttpClient())
    {
        const string url = "https://github.com/tugberkugurlu/ASPNETWebAPISamples/archive/master.zip";
        using (HttpResponseMessage response = await client.GetAsync(url))
        using (Stream streamToReadFrom = await response.Content.ReadAsStreamAsync())
        {
            string fileToWriteTo = Path.GetTempFileName();
            using (Stream streamToWriteTo = File.Open(fileToWriteTo, FileMode.Create))
            {
                await streamToReadFrom.CopyToAsync(streamToWriteTo);
            }

            response.Content = null;
        }
    }
}

By calling GetAsync method directly there, we are loading every single byte into memory. You can see this happening in a simple way by opening the Task Manager and observing the memory of the process.

2

We are calling ReadAsStreamAsync on HttpContent after the GetAsync method is completed. This will just get us the MemoryStream, so there is no point there:

Screenshot 2014-05-11 15.18.14

We need a way not to load the response body into memory and have the raw network stream so that we can pass the bytes into another stream without hitting the memory too hard. We can do it by just reading the headers of the response and then getting a handle for the network stream as below:

static async Task HttpGetForLargeFileInRightWay()
{
    using (HttpClient client = new HttpClient())
    {
        const string url = "https://github.com/tugberkugurlu/ASPNETWebAPISamples/archive/master.zip";
        using (HttpResponseMessage response = await client.GetAsync(url, HttpCompletionOption.ResponseHeadersRead))
        using (Stream streamToReadFrom = await response.Content.ReadAsStreamAsync())
        {
            string fileToWriteTo = Path.GetTempFileName();
            using (Stream streamToWriteTo = File.Open(fileToWriteTo, FileMode.Create))
            {
                await streamToReadFrom.CopyToAsync(streamToWriteTo);
            }
        }
    }
}

Notice that we are calling another overload of the GetAsync method by passing the HttpCompletionOption enumeration value as ResponseHeadersRead. This switch tells the HttpClient not to buffer the response. In other words, it will just read the headers and return the control back. This means that the HttpContent is not ready at the time when you get the control back. Afterwards, we are getting the stream and calling the CopyToAsync method on it by passing our FileStream. The result is much better:

3

Resources

Securing MongoDB Access with Username and Password

My MongoDb journey continues :) and I had my first attempt to put a username and password protection against a MongoDB instance. It went OK besides some hiccups along the way :) Let's see what I did.
2014-05-01 10:44
Tugberk Ugurlu


My MongoDb journey continues :) and I had my first attempt to put a username and password protection against a MongoDB instance. It went OK besides some hiccups along the way :) Let's see what I did.

First, I downloaded the latest (v2.6.0) MongoDB binaries as zip file and unzipped them. I put all MongoDB related stuff inside the c:\mongo directory for my development environment on windows and the structure of my c:\mongo directory is a little different:

Screenshot 2014-04-30 13.50.16

In order to set up the username and password authentication, first I need to get the mongod instance up and running with the authorization on. I achieved that by configuring it with the config file. Here is how it looks like:

dbpath = c:\mongo\data\db
port = 27017
logpath = c:\mongo\data\logs\mongo.log
auth = true

With this config file in place, I can get the mongod instance up:

image

First, I need to connect to this mongod instance and create the admin user. As you can see inside my config file, the server requires authentication. However, there is a localhost exception if there is no user defined inside the system. So, I can connect to my instance anonymously (as I’m running on port 27017 on localhost, I don’t need to define anything while firing up the mongo shell):

image

All great! Let’s create the system user administrator. As everything else, this chore is nicely documented, too:

use admin
db.createUser(
  {
    user: "tugberk",
    pwd: "12345678",
    roles:
    [
      {
        role: "userAdminAnyDatabase",
        db: "admin"
      }
    ]
  }
)

We are pretty much done. We have a user to administer our server now. Let’s disconnect from the mongo shell and reconnect to our mongod instance with our credentials:

mongo --host localhost --port 27017 -u tugberk -p 12345678 --authenticationDatabase admin

image

I’m all there and I can see what my privileges at the server with this user are.

If you try to connect to this MonogDB server anonymously, you will see that you are still able to connect to it. This’s really bad, isn’t it? Not at the level that you think it would be at. The real story is that MongoDB still allows you to connect to, but you won’t be able to do anything as the anonymous access is fully disabled.

image

The bad thing here is that your server existence is exposed which is still an important issue. Just be aware of this fact before getting started.

The user we created still has restricted access to MonogDB server. If you want to have a user with unrestricted access, you can create a user with root role assigned. In our case here, I will assign myself the root role:

use admin
db.grantRolesToUser("tugberk", ["root"])

Resources

Challenge of Solving an ASP.NET Web API Self-Hosting Problem: No HTTP resource was found that matches the request URI

Couple of weeks ago, one of my former coworkers ran across a very weird problem when he was prototyping on some of his ideas with ASP.NET Web API: No HTTP resource was found that matches the request URI. Let's see what this issue was all about and what is the solution.
2014-04-16 08:28
Tugberk Ugurlu


Couple of weeks ago, one of my former coworkers ran across a very weird problem when he was prototyping on some of his ideas with ASP.NET Web API. He was hosting his ASP.NET Web API application on a console application using the Microsoft.Owin.Hosting components and Microsoft.Owin.Host.HttpListener host. His solution structure was also very simple. He put all of his controllers, message handlers, filters, etc. in one class library and all the hosting logic inside the console application. The below structure was pretty similar to what he did:

Screenshot 2014-04-16 10.36.01

Console application also has very little amount of code:

class Program
{
    static void Main(string[] args)
    {
        using (WebApp.Start("http://localhost:5555/", Start))
        {
            Console.WriteLine("Started listening on localhost:5555");
            Console.ReadLine();
            Console.WriteLine("Shutting down...");
        }
    }

    static void Start(IAppBuilder app)
    {
        HttpConfiguration config = new HttpConfiguration();
        config.Routes.MapHttpRoute("DefaultHttpRoute", "api/{controller}");
        app.UseWebApi(config);
    }
}

As you can see, it's all done by the book. However, he was constantly getting 404 when he fired up the application and sent a request to /api/cars:

Screenshot 2014-04-16 10.41.50

"No HTTP resource was found that matches the request URI". It's pretty strange. After I looked into the issue for a while, I was able to figure what the problem is:

Screenshot 2014-04-16 10.45.50

Let's make this a little bit interesting and have a look at the modules loaded into the AppDomain :)

Screenshot 2014-04-16 10.49.30

Notice that the WebApiStrangeConsoleHostSample.dll was never loaded into the AppDomain because we never used it even if it's referenced. As ASP.NET Web API uses reflection to determine the controller and the action, it never finds the CarsController. To prove our point here, I'll load the assembly manually:

static void Main(string[] args)
{
    Assembly.LoadFrom(Path.Combine(Environment.CurrentDirectory, "WebApiStrangeConsoleHostSample.dll"));
    using (WebApp.Start("http://localhost:5555/", Start))
    {
        Console.WriteLine("Started listening on localhost:5555");
        Console.ReadLine();
        Console.WriteLine("Shutting down...");
    }
}

The result is a success:

Screenshot 2014-04-16 10.56.56

However, this is not an ideal solution and I bet that you never run into this issue before. Why? Because, you wise developer keep your hosting agnostic bootstrap code inside the same assembly with you core ASP.NET Web API layer and you call this inside the host application. As soon as you call a method from the core layer assembly, that assembly will be loaded into your AppDomain.

public static class WebApiConfig
{
    public static void Configure(HttpConfiguration config)
    {
        config.Routes.MapHttpRoute("DefaultHttpRoute", "api/{controller}");
    }
}
class Program
{
    static void Main(string[] args)
    {
        using (WebApp.Start("http://localhost:5555/", Start))
        {
            Console.WriteLine("Started listening on localhost:5555");
            Console.ReadLine();
            Console.WriteLine("Shutting down...");
        }
    }

    static void Start(IAppBuilder app)
    {
        HttpConfiguration config = new HttpConfiguration();
        WebApiConfig.Configure(config);
        app.UseWebApi(config);
    }
}

There is one other potential solution to a problem which is similar to this one. That is to replace the IAssembliesResolver service as Filip did in this post.

Order of Fields Matters on MongoDB Indexes

Order of Fields Matters on MongoDB Indexes. Let's see how with an example.
2014-04-12 18:52
Tugberk Ugurlu


As my MongoDB journey continues, I discover new stuff along the way and one of them is about indexes in MongoDB. Let me try to explain it with a sample.

First, create the below four documents inside our users collection:

db.users.save({ 
	_id: 1, 
	name: "tugberk1", 
	login: [
		{ProviderName: "twitter", ProviderKey: "232"}, 
		{ProviderName: "facebook", ProviderKey: "423"}
	]
});
	
db.users.save({ 
	_id: 2, 
	name: "tugberk23", 
	login: [
		{ProviderName: "twitter", ProviderKey: "3443"}
	]
});

db.users.save({ 
	_id: 3, 
	name: "tugberk4343", 
	login: [
		{ProviderName: "dropbox", ProviderKey: "445345"}
	]
});

db.users.save({ 
	_id: 4, 
	name: "tugberk98", 
	login: [
		{ProviderName: "dropbox", ProviderKey: "3443"}, 
		{ProviderName: "facebook", ProviderKey: "768"}
	]
});

Let’s query the users collection by login.ProviderKey and login.ProviderName:

db.users.find({
	"login.ProviderKey": "232", 
	"login.ProviderName": "twitter"
}).pretty();

image

It found the document we wanted. Let’s see how it performed:

db.users.find({
	"login.ProviderKey": "232", 
	"login.ProviderName": "twitter"
}).explain();

image

Result is actually pretty bad. It scanned all four documents to find the one that we wanted to get. Let’s put an index to ProviderName and ProviderKey fields:

db.users.ensureIndex({
	"login.ProviderName": 1, 
	"login.ProviderKey": 1
});

Now, let’s see how it performs the query:

image

It’s better as it scanned only two documents. However, we had only one matching document for our query. As the chances that the providerKey will be more unique than the ProviderName, I want it to first look for the ProviderKey. To do that, I need to change the index:

db.users.dropIndex({
	"login.ProviderName": 1, 
	"login.ProviderKey": 1 
});
db.users.ensureIndex({ 
	"login.ProviderKey": 1, 
	"login.ProviderName": 1 
});

Let’s now see how it’s trying to find the matching documents:

db.users.find({
	"login.ProviderKey": "232", 
	"login.ProviderName": "twitter"
}).explain();

image

Boom! Only one document was scanned. This shows us how it’s important to put the fields in right order for our queries.

Resources

Tags