Having a Look at dotnet CLI Tool and .NET Native Compilation in Linux

dotnet CLI tool can be used for building .NET Core apps and for building libraries through your development flow (compiling, NuGet package management, running, testing, etc.) on various operating systems. Today, I will be looking at this tool in Linux, specifically its native compilation feature.
2016-01-03 18:20
Tugberk Ugurlu


I have been following ASP.NET 5 development from the very start and it has been an amazing experience so far. This new platform has seen so many changes both on libraries and concepts throughout but the biggest of all is about to come. The new command line tools that ASP.NET 5 brought to us like dnx and dnu will vanish soon. However, this doesn’t mean that we won’t have a command line first experience. Concepts of these tools will be carried over by a new command line tool: dotnet CLI.

Note that dotnet CLI is not even a beta yet. It’s very natural that some of the stuff that I show below may change or even be removed. So, be cautious.

image

Scott Hanselman gave an amazing introduction to this tool in his blog post. As indicated in that post, new dotnet CLI tool will give a very similar experience to us compared to other platforms like Go, Ruby, Python. This is very important because, again, this will remove another entry barrier for the newcomers.

You can think of this new CLI tool as combination of following three in terms of concepts:

  • csc.exe
  • msbuild.exe
  • nuget.exe

Of course, this is an understatement but it will help you get a grasp of what that tools can do. One other important aspect of the tool is to be able to bootstrap your code and execute it. Here is how:

In order to install dotnet CLI tool into my Ubuntu machine, I just followed the steps on the Getting Started guide for Ubuntu.

image

Step one is to create a project structure. My project has two files under "hello-dotnet" folder. Program.cs:

using System;

namespace ConsoleApplication
{
    public class Program
    {
        public static void Main(string[] args)
        {
            Console.WriteLine("Hello World!");
        }
    }
}

project.json:

{
    "version": "1.0.0-*",
    "compilationOptions": {
        "emitEntryPoint": true
    },

    "dependencies": {
        "Microsoft.NETCore.Runtime": "1.0.1-beta-*",
        "System.IO": "4.0.11-beta-*",
        "System.Console": "4.0.0-beta-*",
        "System.Runtime": "4.0.21-beta-*"
    },

    "frameworks": {
        "dnxcore50": { }
    }
}

These are the bare essentials that I need to get something outputted to my console window. One important piece here is the emitEntryPoint bit inside the project.json file which indicates that the module will have an entry point which is the static Main method by default.

The second step here is to restore the defined dependencies. This can be done through the "dotnet restore" command:

image

Finally, we can now execute the code that we have written and see that we can actually output some text to console. At the same path, just run "dotnet run" command to do that:

image

Very straight forward experience! Let’s just try to compile the code through "dotnet compile" command:

image

Notice the "hello-dotnet" file there. You can think of this file as dnx which can just run your app. It’s basically the bootstrapper just for your application.

image

So, we understand that we can just run this thing:

image

Very nice! However, that’s not all! This is still a .NET application which requires a few things to be in place to be executed. What we can also do here is to compile native, standalone executables (just like you can do with Go).

image

Do you see the "--native" switch? That will allow you to compile the native executable binary which will be specific to the acrhitecture that you are compiling on (in my case, it’s Ubuntu 14.04):

image

"hello-dotnet" file here can be executed same as the previous one but this time, it’s all machine code and everything is embedded (yes, even the .NET runtime). So, it’s very usual that you will see a significant increase in the size:

image

This is a promising start and amazing to see that we have a unified tool to rule them all (famous last words). The name of the tool is also great, it makes it straight forward to understand based on your experiences with other platforms and seeing this type of command line first architecture adopted outside of ASP.NET is also great and will bring consistency throughout the ecosystem. I will be watching this space as I am sure there will be more to come :)

Resources

Speaking at CodeMash 2016 in Sandusky, Ohio

I will be speaking at CodeMash 2016 in Sandusky, Ohio and I will be talking about ASP.NET 5 and Database Lifecycle Management. I hope to see some of you there :)
2016-01-03 13:58
Tugberk Ugurlu


Attending a developer conference is an amazing way to start a new year. I love conferences because it’s where I learn the most. I am a believer of experience driven life and the conferences is the best place where you can learn about other people’s experiences and different cultures. 2016 will start for me with CodeMash, a unique event that will educate developers on current practices, methodologies, and technology trends in a variety of platforms and development languages such as Java, .NET, Ruby, Python and PHP. The conference will be held in Sandusky, Ohio between 5th and 8th of January, 2016. I will be talking about two topics at the conference and both of those talks will be held on the last day of the conference, 8th of January, Friday.

image

First one is at 08:30 AM: ASP.NET 5: How to Get Your Cheese Back. This is mainly targeted for people who are interested in learning what are the main reasons to adopt ASP.NET 5 and what it will bring to the table. Even if you haven’t done any .NET development before, you will still find a lot of interesting things here as I believe that one of the biggest advantage of the new .NET ecosystem is that there are no big entry barriers for newcommers anymore.

My last talk is going to be on DLM (Database Lifecycle Management), at 11:00 AM: Database Lifecycle Management: Getting it Right. If you are working with an RDBMS in your daily job and want to automate the release process of the changes, you will definitely find something valuable in this talk. I will be mainly giving the examples for SQL Server changes by using DLM Automation Suite tools but this talk is not about tools and SQL Server. It’s all about concepts and challenges of managing the lifecycle of the database schema.

Unfortunately, I will be able to only attend the last two days of the conference but that’s better than nothing. I am so excited about the conference and if you are going to be there, ping me (through Twitter, LinkedIn, etc.) to have a chat and meet. See you there :)

Getting Started with Neo4j in .NET with Neo4jClient Library

I have been looking into Neo4j, a graph database, for a while and here is what impressed me the most while trying to work with it through the Neo4jClient .NET library.
2015-12-13 19:07
Tugberk Ugurlu


I am really in love with the side project I am working on now. It is broken down to little "micro" applications (a.k.a. microservices), uses multiple data storage technologies and being brought together through Docker. As a result, the entire solution feels very natural, not-restricted and feels so manageable.

One part of this solution requires to answer a question which involves going very deep inside the data hierarchy. To illustrate what I mean, have a look at the below graph:

movies-with-only-agency-employees-2

Here, we have an agency which has acquired some actors. Also, we have some movies which employed some actors. You can model this in various data storage systems in various ways but the question I want to answer is the following: "What are the movies which employed all of its actors from Agency-A?". Even thinking about the query you would write in T-SQL is enough to melt your brain for this one. It doesn’t mean that SQL Server, MySQL, etc. are bad data storage systems. It’s just that this type of questions are not among those data storage systems' strengths.

Enters: Neo4j

Neo4j is an open-source graph database implemented in Java and accessible from software written in other languages using the Cypher query language through a transactional HTTP endpoint (Wikipedia says). In Neo4j, your data set consists of nodes and relationships between these nodes which you can interact with through the Cypher query language. Cypher is a very powerful, declarative, SQL-inspired language for describing patterns in graphs. The biggest thing that stands out when working with Cypher is the relationships. Relationships are first class citizens in Cypher. Consider the following Cypher query which is brought from the movie sample in Neo4j web client:

You can bring up the this movie sample by just running ":play movie graph" from the Neo4j web client and walk through it.

MATCH (tom:Person {name: "Tom Hanks"})-[:ACTED_IN]->(tomHanksMovies) RETURN tom,tomHanksMovies

This will list all Tom Hanks movies. However, when you read it from left to right, you will pretty much understand what it will do anyway. The interesting part here is the ACTED_IN relationship inside the query. You may think at this point that this is not a big deal as it can probably translate the below T-SQL query:

SELECT * FROM Movies m
INNER JOIN MovieActors ma ON ma.MovieId = m.Id
WHERE ma.ActorId = 1;

However, you will start seeing the power as the questions get interesting. For example, let’s find out Tom Hanks’ co-actors from the every movie he acted in (again, from the same sample):

MATCH (tom:Person {name:"Tom Hanks"})-[:ACTED_IN]->(m)<-[:ACTED_IN]-(coActors) RETURN coActors.name

It’s just mind-blowingly complicated to retrieve this from a relational database but with Cypher, it is dead easy. You can start to see that it’s all about building up nodes and declaring the relationships to get the answer to your question in Neo4j.

Neo4j in .NET

As Neo4j communicates through HTTP, you can pretty much find a client implementation in every ecosystem and .NET is not an exception. Amazing people from Readify is maintaining Neo4jClient OSS project. It’s extremely easy to use this and the library has a very good documentation. I especially liked the part where they have documented the thread safety concerns of GraphClient. It is the first thing I wanted to find out and there it was.

Going back to my example which I mentioned at the beginning of this post, I tried to handle this through the .NET Client. Let’s walk through what I did.

You can find the below sample under my DotNetSamples GitHub repository.

First, I initiated the GraphClient and made some adjustments:

var client = new GraphClient(new Uri("http://localhost:7474/db/data"), "neo4j", "1234567890")
{
    JsonContractResolver = new CamelCasePropertyNamesContractResolver()
};

client.Connect();

I started with creating the agency.

var agencyA = new Agency { Name = "Agency-A" };
client.Cypher
    .Create("(agency:Agency {agencyA})")
    .WithParam("agencyA", agencyA)
    .ExecuteWithoutResultsAsync()
    .Wait();

Next is to create the actors and ACQUIRED relationship between the agency and some actors (in below case, only the odd numbered actors):

for (int i = 1; i <= 5; i++)
{
    var actor = new Person { Name = $"Actor-{i}" };

    if ((i % 2) == 0)
    {
        client.Cypher
            .Create("(actor:Person {newActor})")
            .WithParam("newActor", actor)
            .ExecuteWithoutResultsAsync()
            .Wait();
    }
    else
    {
        client.Cypher
            .Match("(agency:Agency)")
            .Where((Agency agency) => agency.Name == agencyA.Name)
            .Create("agency-[:ACQUIRED]->(actor:Person {newActor})")
            .WithParam("newActor", actor)
            .ExecuteWithoutResultsAsync()
            .Wait();
    }
}

Then, I have created the movies :

char[] chars = Enumerable.Range('a', 'z' - 'a' + 1).Select(i => (Char)i).ToArray();
for (int i = 0; i < 3; i++)
{
    var movie = new Movie { Name = $"Movie-{chars[i]}" };

    client.Cypher
        .Create("(movie:Movie {newMovie})")
        .WithParam("newMovie", movie)
        .ExecuteWithoutResultsAsync()
        .Wait();
}

Lastly, I have related existing movies and actors through the EMPLOYED relationship.

client.Cypher
    .Match("(movie:Movie)", "(actor1:Person)", "(actor5:Person)")
    .Where((Movie movie) => movie.Name == "Movie-a")
    .AndWhere((Person actor1) => actor1.Name == "Actor-1")
    .AndWhere((Person actor5) => actor5.Name == "Actor-5")
    .Create("(movie)-[:EMPLOYED]->(actor1), (movie)-[:EMPLOYED]->(actor5)")
    .ExecuteWithoutResultsAsync()
    .Wait();

client.Cypher
    .Match("(movie:Movie)", "(actor1:Person)", "(actor3:Person)", "(actor5:Person)")
    .Where((Movie movie) => movie.Name == "Movie-b")
    .AndWhere((Person actor1) => actor1.Name == "Actor-1")
    .AndWhere((Person actor3) => actor3.Name == "Actor-3")
    .AndWhere((Person actor5) => actor5.Name == "Actor-5")
    .Create("(movie)-[:EMPLOYED]->(actor1), (movie)-[:EMPLOYED]->(actor3), (movie)-[:EMPLOYED]->(actor5)")
    .ExecuteWithoutResultsAsync()
    .Wait();

client.Cypher
    .Match("(movie:Movie)", "(actor2:Person)", "(actor5:Person)")
    .Where((Movie movie) => movie.Name == "Movie-c")
    .AndWhere((Person actor2) => actor2.Name == "Actor-2")
    .AndWhere((Person actor5) => actor5.Name == "Actor-5")
    .Create("(movie)-[:EMPLOYED]->(actor2), (movie)-[:EMPLOYED]->(actor5)")
    .ExecuteWithoutResultsAsync()
    .Wait();

When I run this, I now have the data set that I can play with. I have jumped back to web client and ran the below query to retrieve the relations:

MATCH (agency:Agency)-[:ACQUIRED]->(actor:Person)<-[:EMPLOYED]-(movie:Movie)
RETURN agency, actor, movie

One of the greatest features of the web client is that you can view your query result in a graph representation. How cool is that? You can exactly see the smilarity between the below result and the graph I have put together above:

image

Of course, we can run the same above query through the .NET client and grab the results:

var results = client.Cypher
    .Match("(agency:Agency)-[:ACQUIRED]->(actor:Person)<-[:EMPLOYED]-(movie:Movie)")
    .Return((agency, actor, movie) => new
    {
        Agency = agency.As<Agency>(),
        Actor = actor.As<Person>(),
        Movie = movie.As<Movie>()
    }).Results;

Going Beyond

However, how can we answer my "What are the movies which employed all of its actors from Agency-A?" question? As I am very new to Neo4j, I struggled a lot with this. In fact, I was not even sure whether this was possible to do in Neo4J. I asked this as a question in Stackoverflow (as every stuck developer do) and Christophe Willemsen gave an amazing answer which literally blew my mind. I warn you now as the below query is a bit complex and I am still going through it piece by piece to try to understand it but it does the trick:

MATCH (agency:Agency { name:"Agency-A" })-[:ACQUIRED]->(actor:Person)<-[:EMPLOYED]-(movie:Movie)
WITH DISTINCT movie, collect(actor) AS actors
MATCH (movie)-[:EMPLOYED]->(allemployees:Person)
WITH movie, actors, count(allemployees) AS c
WHERE c = size(actors)
RETURN movie.name

The result is as you would expect:

image

Still Dipping My Toes

I am hooked but this doesn’t mean that Neo4j is the solution to my problems. I am still evaluating it by implementing a few features on top of it. There are a few parts which I haven’t been able to answer exactly yet:

  • How does this scale with large data sets?
  • Can I shard the data across servers?
  • Want are the hosted options?
  • What is the story on geo location queries?

However, the architecture I have in my solution allows me to evaluate this type of technologies. At worst case scenario, Neo4j will not work for me but I will be able to replace it with something else (which I doubt that it will be the case).

Resources

Integration Testing with MongoDB with MongoDB.Testing Library

I have put together a library, MongoDB.Testing, which makes it easy to stand up a MongoDB server, create a random database and clean up the resources afterwards. Here is how you can start using it.
2015-12-05 21:06
Tugberk Ugurlu


Considering the applications we produce today (small, targeted, "micro" applications), I value integration tests way more than unit tests (along with acceptance tests). They provide much more realistic testing on your application with the only downside of being hard to pinpoint which part of your code is the problem when you have failures. I have been writing integration tests for the .NET based HTTP applications which use MongoDB as the data storage system on same parts and I pulled out a helper into library which makes it easy to stand up a MongoDB server, create a random database and clean up the resources afterwards. The library is called MongoDB.Testing and it’s on NuGet, GitHub. Usage is also pretty simple and there is also a a few samples I have put together.

Install the library into your testing project through NuGet:

Install-Package MongoDB.Testing -pre

Write a mongod.exe locator:

public class MongodExeLocator : IMongoExeLocator
{
    public string Locate()
    {
        return @"C:\Program Files\MongoDB\Server\3.0\bin\mongod.exe";
    }
}

Finally, integrate this into your tests:

[Test]
public async Task HasEnoughRating_Should_Throw_When_The_User_Is_Not_Found()
{
    using (MongoTestServer server = MongoTestServer.Start(27017, new MongodExeLocator()))
    {
        // ARRANGE
        var collection = server.Database.GetCollection<UserEntity>("users");
        var service = new MyCounterService(collection);
        await collection.InsertOneAsync(new UserEntity
        {
            Id = ObjectId.GenerateNewId().ToString(),
            Name = "foo",
            Rating = 23
        });

        // ACT, ASSERT
        Assert.Throws<InvalidOperationException>(
            () => service.HasEnoughRating(ObjectId.GenerateNewId().ToString()));
    }
}

That’s basically all. MongoTestServer.Start will do the following for you:

  • Start a mongod instance and expose it through the specified port.
  • Creates a randomly named MongoDB database on the started instance and exposes it through the MongoTestServer instance returned from MongoTestServer.Start method.
  • Cleans up the resources, kills the mongod.exe instance when the MongoTestServer instance is disposed.

If you are doing a similar sort of testing with MongoDB, give this a shot. I want to improve this based on the needs. So, make sure to file issues and send some lovely pull requests.

My Talk on Profiling .NET Server Applications from Umbraco UK Festival 2015

I was at Umbraco UK Festival 2015 in London a few weeks ago to give a talk on Profiling .NET Server Applications and the session is now available to watch.
2015-11-11 13:13
Tugberk Ugurlu


I was at Umbraco UK Festival 2015 in London a few weeks ago to give a talk on Profiling .NET Server Applications. It was a really great experience for me as this was my first time presenting on this topic which I love and enjoy very much. Also, the conference venue was a church which made it really interesting for a presentation and I would be lying if I tell you that I didn’t feel like a deacon up there on the stage :) The fantastic news is that all sessions were recorded and all of them are available to watch now including my Profiling .NET Server Applications talk:

You can find the slides under my Speaker Deck account and I also encourage you to download the free e-book which gives you 52 quick tips and tricks for .NET application performance:

image

Finally, here are some further resources to look at if the talk was interesting for you:

Tags