Sorted By: Tag (mongodb)

Setting up a MongoDB Replica Set with Docker and Connecting to It With a .NET Core App

Easily setting up realistic non-production (e.g. dev, test, QA, etc.) environments is really critical in order to reduce the feedback loop. In this blog post, I want to talk about how you can achieve this if your application relies on MongoDB Replica Set by showing you how to set it up with Docker for non-production environments.
2018-01-31 10:10
Tugberk Ugurlu


Easily setting up realistic non-production (e.g. dev, test, QA, etc.) environments is really critical in order to reduce the feedback loop. In this blog post, I want to talk about how you can achieve this if your application relies on MongoDB Replica Set by showing you how to set it up with Docker for non-production environments.

Hold on! I want to watch, not read!

I got you covered there! I have also recorded a ~5m covering the content of this blog post, where I also walks you through the steps visually. If you find this option useful, let me know through the comments below and I can aim harder to repeat that :)

What are we trying to do here and why?

Picture2

 

If you have an application which works against a MongoDB database, it’s very common to have a replica set in production. This approach ensures the high availability of the data, especially for read scenarios. However, applications mostly end up working against a single MongoDB instance, because setting up a Replica Set in isolation is a tedious process. As mentioned at the beginning of the post, we want to reflect the production environment to the process of developing or testing the software applications as much as possible. The reason for that is to catch unexpected behaviour which may only occur under a production environment. This approach is valuable because it would allow us to reduce the feedback loop on those exceptional cases.

Docker makes this all easy!

This is where Docker enters into the picture! Docker is containerization technology and it allows us to have repeatable process to provision environments in a declarative way. It also gives us a try and tear down model where we can experiment and easily start again from the initial state. Docker can also help us with easily setting up a MongoDB Replica Set. Within our Docker Host, we can create Docker Network which would give us the isolated DNS resolution across containers. Then we can start creating the MongoDB docker containers. They would initially be unaware of each other. However, we can initialise the replication by connecting to one of the containers and running the replica set initialisation command. Finally,  we can deploy our application container under the same docker network.

Picture1

There are a handful of advantages to setting up this with Docker and I want to specifically touch on some of them:

  •  It can be automated easily. This is especially crucial for test environments which are provisioned on demand.
  • It’s repeatable! The declarative nature of the Dockerfile makes it possible to end up with the same environment setup even if you run the scripts months later after your initial setup.
  • Familiarity! Docker is a widely known and used tool for lots of other purposes and familiarity to the tool is high. Of course, this may depend on your development environment

Let’s make it work!

First of all, I need to create a docker network. I can achieve this by running the "docker network create” command and giving it a unique name.

docker network create my-mongo-cluster

The next step is to create the MongoDB docker containers and start them. I can use “docker run” command for this. Also, MongoDB has an official image on Docker Hub. So, I can reuse that to simplify the acqusition of MongoDB. For convenience, I will name the container with a number suffix. The container also needs to be tied to the network we have previously created. Finally, I need to specify the name of the replica set for each container.

docker run --name mongo-node1 -d --net my-mongo-cluster mongo --replSet “rs0"

First container is created and I need to run the same command to create two more MongoDB containers. The only difference is with the container names.

docker run --name mongo-node2 -d --net my-mongo-cluster mongo --replSet "rs0"
docker run --name mongo-node3 -d --net my-mongo-cluster mongo --replSet “rs0"

I can see that all of my MongoDB containers are at the running state by executing the “docker ps” command.

Image

In order to form a replica set, I need to initialise the replication. I will do that by connecting to one of the containers through the “docker exec” command and starting the mongo shell client.

docker exec -it mongo-node1 mongo

Image

As I now have a connection to the server, I can initialise the replication. This requires me to declare a config object which will include connection details of all the servers.

config = {
      "_id" : "rs0",
      "members" : [
          {
              "_id" : 0,
              "host" : "mongo-node1:27017"
          },
          {
              "_id" : 1,
              "host" : "mongo-node2:27017"
          },
          {
              "_id" : 2,
              "host" : "mongo-node3:27017"
          }
      ]
  }

Finally, we can run “rs.initialize" command to complete the set up.

You will notice that the server I am connected to will be elected as the primary in the replica set shortly. By running “rs.status()”, I can view the status of other MongoDB servers within the replica set. We can see that there are two secondaries and one primary in the replica set.

.NET Core Application

As a scenario, I want to run my .NET Core application which writes data to a MongoDB database and start reading it in a loop. This application will be connecting to the MongoDB replica set which we have just created.  This is a standard .NET Core console application which you can create by running the following script:

dotnet new console

The csproj file for this application looks like below.

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp2.0</TargetFramework>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Bogus" Version="18.0.2" />
    <PackageReference Include="MongoDB.Driver" Version="2.4.4" />
    <PackageReference Include="Polly" Version="5.3.1" />
  </ItemGroup>
</Project>

Notice that I have two interesting dependencies there. Polly is used to retry the read calls to MongoDB based on defined policies. This bit is interesting as I would expect the MongoDB client to handle that for read calls. However, it might be also a good way of explicitly stating which calls can be retried inside your application. Bogus, on the other hand, is just here to be able to create fake names to make the application a bit more realistic :)

Finally, this is the code to make this application work:

partial class Program
{
    static void Main(string[] args)
    {
        var settings = new MongoClientSettings
        {
            Servers = new[]
            {
                new MongoServerAddress("mongo-node1", 27017),
                new MongoServerAddress("mongo-node2", 27017),
                new MongoServerAddress("mongo-node3", 27017)
            },
            ConnectionMode = ConnectionMode.ReplicaSet,
            ReplicaSetName = "rs0"
        };

        var client = new MongoClient(settings);
        var database = client.GetDatabase("mydatabase");
        var collection = database.GetCollection<User>("users");

        System.Console.WriteLine("Cluster Id: {0}", client.Cluster.ClusterId);
        client.Cluster.DescriptionChanged += (object sender, ClusterDescriptionChangedEventArgs foo) => 
        {
            System.Console.WriteLine("New Cluster Id: {0}", foo.NewClusterDescription.ClusterId);
        };

        for (int i = 0; i < 100; i++)
        {
            var user = new User { Id = ObjectId.GenerateNewId(), Name = new Bogus.Faker().Name.FullName() };
            collection.InsertOne(user);
        }

        while (true)
        {
            var randomUser = collection.GetRandom();
            Console.WriteLine(randomUser.Name);

            Thread.Sleep(500);
        }
    }
}

This is not the most beautiful and optimized code ever but should demonstrate what we are trying to achieve by having a replica set. It's actually the GetRandom method on the MongoDB collection object which handles the retry:

public static class CollectionExtensions 
{
    private readonly static Random random = new Random();

    public static T GetRandom<T>(this IMongoCollection<T> collection) 
    {
        var retryPolicy = Policy
            .Handle<MongoCommandException>()
            .Or<MongoConnectionException>()
            .WaitAndRetry(2, retryAttempt => 
                TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)) 
            );

        return retryPolicy.Execute(() => GetRandomImpl(collection));
    }

    private static T GetRandomImpl<T>(this IMongoCollection<T> collection)

    {
        return collection.Find(FilterDefinition<T>.Empty)
            .Limit(-1)
            .Skip(random.Next(99))
            .First();
    }
}

I will run this through docker as well and here is the dockerfile for this: 

FROM microsoft/dotnet:2-sdk

COPY ./mongodb-replica-set.csproj /app/
WORKDIR /app/
RUN dotnet --info
RUN dotnet restore
ADD ./ /app/
RUN dotnet publish -c DEBUG -o out
ENTRYPOINT ["dotnet", "out/mongodb-replica-set.dll"]

When it starts, we can see that it will output the result to the console:

Image

Prove that It Works!

In order to demonstrate the effect of the replica set, I want to take down the primary node. First of all, we need to have look at the output of rs.status command we have previously ran in order to identify the primary node. We can see that it’s node1! 

Image

Secondly, we need to get the container id for that node. 

Image

Finally, we can kill the container by running the “docker stop command”. Once the container is stopped, you will notice that application will gracefully recover and continue reading the data. 

Image

Integration Testing with MongoDB with MongoDB.Testing Library

I have put together a library, MongoDB.Testing, which makes it easy to stand up a MongoDB server, create a random database and clean up the resources afterwards. Here is how you can start using it.
2015-12-05 21:06
Tugberk Ugurlu


Considering the applications we produce today (small, targeted, "micro" applications), I value integration tests way more than unit tests (along with acceptance tests). They provide much more realistic testing on your application with the only downside of being hard to pinpoint which part of your code is the problem when you have failures. I have been writing integration tests for the .NET based HTTP applications which use MongoDB as the data storage system on same parts and I pulled out a helper into library which makes it easy to stand up a MongoDB server, create a random database and clean up the resources afterwards. The library is called MongoDB.Testing and it’s on NuGet, GitHub. Usage is also pretty simple and there is also a a few samples I have put together.

Install the library into your testing project through NuGet:

Install-Package MongoDB.Testing -pre

Write a mongod.exe locator:

public class MongodExeLocator : IMongoExeLocator
{
    public string Locate()
    {
        return @"C:\Program Files\MongoDB\Server\3.0\bin\mongod.exe";
    }
}

Finally, integrate this into your tests:

[Test]
public async Task HasEnoughRating_Should_Throw_When_The_User_Is_Not_Found()
{
    using (MongoTestServer server = MongoTestServer.Start(27017, new MongodExeLocator()))
    {
        // ARRANGE
        var collection = server.Database.GetCollection<UserEntity>("users");
        var service = new MyCounterService(collection);
        await collection.InsertOneAsync(new UserEntity
        {
            Id = ObjectId.GenerateNewId().ToString(),
            Name = "foo",
            Rating = 23
        });

        // ACT, ASSERT
        Assert.Throws<InvalidOperationException>(
            () => service.HasEnoughRating(ObjectId.GenerateNewId().ToString()));
    }
}

That’s basically all. MongoTestServer.Start will do the following for you:

  • Start a mongod instance and expose it through the specified port.
  • Creates a randomly named MongoDB database on the started instance and exposes it through the MongoTestServer instance returned from MongoTestServer.Start method.
  • Cleans up the resources, kills the mongod.exe instance when the MongoTestServer instance is disposed.

If you are doing a similar sort of testing with MongoDB, give this a shot. I want to improve this based on the needs. So, make sure to file issues and send some lovely pull requests.

ASP.NET 5 Identity MongoDB Implementation

ASP.NET Identity will have a new version with ASP.NET 5 which is going to be version 3.0.0 and I gave it shot to implement ASP.NET Identity MongoDB data store.
2015-11-05 20:04
Tugberk Ugurlu


As with everything, ASP.NET Identity will have a new version with ASP.NET 5 which is going to be version 3.0.0. There are some changes on the interfaces but it’s not as drastic as others. By default, it provides Entity Framework implementation which I assume going to be compatible with any data storage system that can plug into Entity Framework (which is good). However, you need to provide a custom implementation if you want to support a data storage system which doesn’t support Entity Framework. MongoDB is one of them and I gave it shot to implement ASP.NET Identity MongoDB store. The result was really good:

mongodb-aspnet-identity

Library is available on NuGet as Dnx.Identity.MongoDB package and it supports beta8 runtime release. For now, it’s part of a sample project I am working on but it will probably make it into its own repository soon. I also have a sample here which is the fork of the original Identity sample. You can look at this commit to see what I had to do to make it support my custom provider which is not that bad, if you ignore the dependency injection dance I had to make. That’s because my implementation of ASP.NET Identity library doesn’t support IRoleStore. Don’t worry, you will not need this as you already have IUserClaimStore and also, there is an open issue to change the dependency injection hook a bit so that the IRoleStore would be optional.

Here are a few more details about this ASP.NET 5 Identity MongoDB implementation:

  • Currently, there is no documentation.
  • MongoUserStore is thread safe. You can register this as a singleton.
  • At the moment, there is no logging and nice exception handling for the implementation.
  • The implementation only supports dnx452 or above and it doesn’t support corefx as MongoDB .NET Client has no support for that.
  • The library doesn’t support SetUserNameAsync as the user name is also the Id of the user. So, you cannot change the Id.
  • The implementation requires you to pass an implementation of IMongoDatabase through the constructor and it persists data into a collection which is named "users". There is going to be a way to change the name of the collection soon in upcoming releases.
  • E-mail uniqueness is ensured through a unique MongoDB index. However, this will not function properly if you shard the users collection.
  • The implementation doesn’t persist changes unless you call one of the UpdateAsync, CreateAsync or DeleteAsync methods (which is what UserManager does).

Give it a try and you can file issues here for now and send pull requests. Enjoy :)

Securing MongoDB Access with Username and Password

My MongoDb journey continues :) and I had my first attempt to put a username and password protection against a MongoDB instance. It went OK besides some hiccups along the way :) Let's see what I did.
2014-05-01 10:44
Tugberk Ugurlu


My MongoDb journey continues :) and I had my first attempt to put a username and password protection against a MongoDB instance. It went OK besides some hiccups along the way :) Let's see what I did.

First, I downloaded the latest (v2.6.0) MongoDB binaries as zip file and unzipped them. I put all MongoDB related stuff inside the c:\mongo directory for my development environment on windows and the structure of my c:\mongo directory is a little different:

Screenshot 2014-04-30 13.50.16

In order to set up the username and password authentication, first I need to get the mongod instance up and running with the authorization on. I achieved that by configuring it with the config file. Here is how it looks like:

dbpath = c:\mongo\data\db
port = 27017
logpath = c:\mongo\data\logs\mongo.log
auth = true

With this config file in place, I can get the mongod instance up:

image

First, I need to connect to this mongod instance and create the admin user. As you can see inside my config file, the server requires authentication. However, there is a localhost exception if there is no user defined inside the system. So, I can connect to my instance anonymously (as I’m running on port 27017 on localhost, I don’t need to define anything while firing up the mongo shell):

image

All great! Let’s create the system user administrator. As everything else, this chore is nicely documented, too:

use admin
db.createUser(
  {
    user: "tugberk",
    pwd: "12345678",
    roles:
    [
      {
        role: "userAdminAnyDatabase",
        db: "admin"
      }
    ]
  }
)

We are pretty much done. We have a user to administer our server now. Let’s disconnect from the mongo shell and reconnect to our mongod instance with our credentials:

mongo --host localhost --port 27017 -u tugberk -p 12345678 --authenticationDatabase admin

image

I’m all there and I can see what my privileges at the server with this user are.

If you try to connect to this MonogDB server anonymously, you will see that you are still able to connect to it. This’s really bad, isn’t it? Not at the level that you think it would be at. The real story is that MongoDB still allows you to connect to, but you won’t be able to do anything as the anonymous access is fully disabled.

image

The bad thing here is that your server existence is exposed which is still an important issue. Just be aware of this fact before getting started.

The user we created still has restricted access to MonogDB server. If you want to have a user with unrestricted access, you can create a user with root role assigned. In our case here, I will assign myself the root role:

use admin
db.grantRolesToUser("tugberk", ["root"])

Resources

Order of Fields Matters on MongoDB Indexes

Order of Fields Matters on MongoDB Indexes. Let's see how with an example.
2014-04-12 18:52
Tugberk Ugurlu


As my MongoDB journey continues, I discover new stuff along the way and one of them is about indexes in MongoDB. Let me try to explain it with a sample.

First, create the below four documents inside our users collection:

db.users.save({ 
	_id: 1, 
	name: "tugberk1", 
	login: [
		{ProviderName: "twitter", ProviderKey: "232"}, 
		{ProviderName: "facebook", ProviderKey: "423"}
	]
});
	
db.users.save({ 
	_id: 2, 
	name: "tugberk23", 
	login: [
		{ProviderName: "twitter", ProviderKey: "3443"}
	]
});

db.users.save({ 
	_id: 3, 
	name: "tugberk4343", 
	login: [
		{ProviderName: "dropbox", ProviderKey: "445345"}
	]
});

db.users.save({ 
	_id: 4, 
	name: "tugberk98", 
	login: [
		{ProviderName: "dropbox", ProviderKey: "3443"}, 
		{ProviderName: "facebook", ProviderKey: "768"}
	]
});

Let’s query the users collection by login.ProviderKey and login.ProviderName:

db.users.find({
	"login.ProviderKey": "232", 
	"login.ProviderName": "twitter"
}).pretty();

image

It found the document we wanted. Let’s see how it performed:

db.users.find({
	"login.ProviderKey": "232", 
	"login.ProviderName": "twitter"
}).explain();

image

Result is actually pretty bad. It scanned all four documents to find the one that we wanted to get. Let’s put an index to ProviderName and ProviderKey fields:

db.users.ensureIndex({
	"login.ProviderName": 1, 
	"login.ProviderKey": 1
});

Now, let’s see how it performs the query:

image

It’s better as it scanned only two documents. However, we had only one matching document for our query. As the chances that the providerKey will be more unique than the ProviderName, I want it to first look for the ProviderKey. To do that, I need to change the index:

db.users.dropIndex({
	"login.ProviderName": 1, 
	"login.ProviderKey": 1 
});
db.users.ensureIndex({ 
	"login.ProviderKey": 1, 
	"login.ProviderName": 1 
});

Let’s now see how it’s trying to find the matching documents:

db.users.find({
	"login.ProviderKey": "232", 
	"login.ProviderName": "twitter"
}).explain();

image

Boom! Only one document was scanned. This shows us how it’s important to put the fields in right order for our queries.

Resources

Tags