Sorted By: Tag (elasticsearch)

My Talk on Profiling .NET Server Applications from Umbraco UK Festival 2015

I was at Umbraco UK Festival 2015 in London a few weeks ago to give a talk on Profiling .NET Server Applications and the session is now available to watch.
2015-11-11 13:13
Tugberk Ugurlu


I was at Umbraco UK Festival 2015 in London a few weeks ago to give a talk on Profiling .NET Server Applications. It was a really great experience for me as this was my first time presenting on this topic which I love and enjoy very much. Also, the conference venue was a church which made it really interesting for a presentation and I would be lying if I tell you that I didn’t feel like a deacon up there on the stage :) The fantastic news is that all sessions were recorded and all of them are available to watch now including my Profiling .NET Server Applications talk:

You can find the slides under my Speaker Deck account and I also encourage you to download the free e-book which gives you 52 quick tips and tricks for .NET application performance:

image

Finally, here are some further resources to look at if the talk was interesting for you:

ASP.NET 5 and Log Correlation by Request Id

ASP.NET 5 is full of big new features and enhancements but besides these, I am mostly impressed by little, tiny features of ASP.NET 5 Log Correlation which is provided out of the box. Let me quickly show you what it is in this post.
2015-10-28 00:44
Tugberk Ugurlu


ASP.NET 5 is full of big new features and enhancements like being able to run on multiple operating systems, incredible CLI tools, hassle-free building for multiple framework targets, build only dependencies and many more. Besides these, I am mostly impressed by little, tiny features of ASP.NET 5 because these generally tend to be ignored in this type of rearchitecturing works. One of these little features is log correlation. Let me quickly show you what it is and why it made me smile.

BIG ASS CAUTION! At the time of this writing, I am using DNX 1.0.0-beta8 version. As things are moving really fast in this new world, it’s very likely that the things explained here will have been changed as you read this post. So, be aware of this and try to explore the things that are changed to figure out what are the corresponding new things.

Also, inside this post I am referencing a lot of things from ASP.NET GitHub repositories. In order to be sure that the links won’t break in the future, I’m actually referring them by getting permanent links to the files on GitHub. So, these links are actually referring the files from the latest commit at the time of this writing and they have a potential to be changed, too. Read the "Getting permanent links to files" post to figure what this actually is.

Brief Introduction to Logging in ASP.NET 5 World

If you want to skip this part, you can directly go to "Log Correlation" section below.

As you probably know, ASP.NET 5 also has a great support for logging. The nicest thing about this new logging abstraction is that it’s the only logging abstraction which every provided library and framework is relying on. So, when you enable logging in your application, you will enable it in all components (which is perfect)! Here is a sample in my MVC 6 application. I am just adding MVC to pipeline here, enabling logging by hooking Serilog and configuring it to write the logs to console:

using System;
using Microsoft.AspNet.Builder;
using Microsoft.Framework.DependencyInjection;
using Microsoft.Framework.Logging;
using Serilog;

namespace LoggingCorrelationSample
{
    public class Startup
    {
        public Startup(ILoggerFactory loggerFactory)
        {
            var serilogLogger = new LoggerConfiguration()
                .WriteTo
                .TextWriter(Console.Out)
                .MinimumLevel.Verbose()
                .CreateLogger();

            loggerFactory.MinimumLevel = LogLevel.Debug;
            loggerFactory.AddSerilog(serilogLogger);
        }

        public void ConfigureServices(IServiceCollection services)
        {
            services.AddMvc();
        }

        public void Configure(IApplicationBuilder app)
        {
            app.UseMvc();
        }
    }
}

When I run the application and hit a valid endpoint, I will see bunch of things being logged to console:

image

Remember, I haven’t logged anything myself yet. It’s just the stuff I hooked in which were already relying on ASP.NET 5 logging infrastructure. This doesn’t mean I can’t though. Hooking into logging is super easy since an instance of ILoggerFactory is already inside the DI system. Here is an example class which I have for my application and it is responsible for getting the cars (forgive the stupid example here but I am sure you will get the idea):

public class CarsContext : IDisposable
{
    private readonly ILogger _logger;

    public CarsContext(ILoggerFactory loggerFactory)
    {
        _logger = loggerFactory.CreateLogger<CarsContext>();
        _logger.LogDebug("Constructing CarsContext");
    }

    public IEnumerable<string> GetCars()
    {
        _logger.LogInformation("Found 3 cars.");
        
        return new[]
        {
            "Car 1",
            "Car 2",
            "Car 3"
        };
    }
    
    public void Dispose()
    {
        _logger.LogDebug("Disposing CarsContext");
    }
}

I will register this class so that it can get the dependencies it needs and also, it can be injected into other places:

public void ConfigureServices(IServiceCollection services)
{
    services.AddMvc();
    services.AddScoped<CarsContext, CarsContext>();
}

Finally, I will use it inside my controller:

public class CarsController : Controller
{
    private readonly CarsContext _carsContext;
    
    public CarsController(CarsContext carsContext)
    {
        _carsContext = carsContext;
    }
    
    [Route("cars")]
    public IActionResult Get()
    {
        var cars = _carsContext.GetCars();
        return Ok(cars);
    }
}

Just seeing how beautifully things are coming together is really great! When I run the application and hit the /cars endpoint now, I will see my logs appearing along side the framework and library logs:

image

Same goes for your middlewares. You can naturally hook into logging system from your middleware thanks to first class middleware DI support.

public class RequestUrlLoggerMiddleware 
{
    private readonly RequestDelegate _next;
    private readonly Microsoft.Framework.Logging.ILogger _logger;
    
    public RequestUrlLoggerMiddleware(RequestDelegate next, ILoggerFactory loggerFactory) 
    {
        _next = next;
        _logger = loggerFactory.CreateLogger<RequestUrlLoggerMiddleware>();
    }
    
    public Task Invoke (HttpContext context)
    {
        _logger.LogInformation("{Method}: {Url}", context.Request.Method, context.Request.Path);
        return _next(context);
    }
}

Notice that we have a log message template rather the actual log message here. This is another great feature of the new logging system which is pretty much the same as what Serilog have had for log time.

When we run this, we should see the middleware log appear, too:

image

Log Correlation

Without doing anything else first, let me also write logs to Elasticsearch by pulling in Serilog Elasticsearch sink and hooking it in. After hitting the same endpoint, I have the below result inside my Elasticsearch index:

image

You can see that each log message has got richer and we can see new things like RequestId which will allow you to correlate your logs per request. This information is being logged because the hosting layer starts a new log scope for each request. RequestId is particularly useful when you have an unexpected behavior with an HTTP request and you want to see what was happening with that request. In order to take advantage this, you should send the the RequestId along side your response (ideally among the response headers). The below is a sample middleware which you can hook into your pipeline in order to add RequestId to your response:

public class RequestIdMiddleware
{
    private readonly RequestDelegate _next;

    public RequestIdMiddleware(RequestDelegate next)
    {
        _next = next;
    }

    public async Task Invoke(HttpContext context)
    {
        var requestIdFeature = context.Features.Get<IHttpRequestIdentifierFeature>();
        if (requestIdFeature?.TraceIdentifier != null)
        {
            context.Response.Headers["RequestId"] = requestIdFeature.TraceIdentifier;
        }

        await _next(context);
    }
}

Note that, IHttpRequestIdentifierFeature is the way to get a hold of RequestId in beta8 but in upcoming versions, it’s likely to change to HttpContext.TraceIdentifier.

If you look at the response headers now, you should see the RequestId header there:

HTTP/1.1 200 OK
Date: Wed, 28 Oct 2015 00:32:22 GMT
Content-Type: application/json; charset=utf-8
Server: Kestrel
RequestId: 0b66784c-eb98-4a53-9247-8563fad85857
Transfer-Encoding: chunked

Assuming that I have problems with this request and I have been handed the RequestId, I should be able to see what happened in that request by running a simple query on my Elasticsearch index:

image

That’s pretty much it and as mentioned, this is one those tiny features which was always possible but painful to get it all right. If you are also interested, you can find the full source code of the sample under my GitHub repository.

Elasticsearch Array Contains Search With Terms Filter

Here is a quick blog post on Elasticsearch and terms filter to achieve array contains search with terms filter
2015-07-14 22:17
Tugberk Ugurlu


Here is a quick blog post on Elasticsearch and terms filter while I still remember how the hell it works :) Yes, this is possibly the 20th time that I looked for how to achieve array contains functionality in Elasticseach and it's a clear sign for me that I need to blog about it :)

I created the index called movies (mostly borrowed from Joel's great Elasticsearch 101 blog post) and here is its mapping:

PUT movies/_mapping/movie
{
  "movie": {
    "properties": {
       "director": {
          "type": "string"
       },
       "genres": {
          "type": "string",
          "index": "not_analyzed"
       },
       "title": {
          "type": "string"
       },
       "year": {
          "type": "long"
       }
    }
  }
}

The genres field mapping is important here as it needs to be not analyzed. I also indexed a few stuff in it:

POST movies/movie
{
    "title": "Apocalypse Now",
    "director": "Francis Ford Coppola",
    "year": 1979,
    "genres": ["Drama", "War", "Foo"]
}

POST movies/movie
{
    "title": "Apocalypse Now",
    "director": "Francis Ford Coppola",
    "year": 1979,
    "genres": ["Drama", "War", "Foo", "Bar"]
}

POST movies/movie
{
    "title": "Apocalypse Now",
    "director": "Francis Ford Coppola",
    "year": 1979,
    "genres": ["Drama", "Bar"]
}

Now, I am interested in finding out the movies which is in War or Foo genre. The way to achieve that is the terms filter as mentioned:

GET movies/movie/_search
{
  "query": {
    "filtered": {
      "query": {
        "match_all": {}
      },
      "filter": {
        "terms": {
          "genres": ["War", "Foo"]
        }
      }
    }
  }
}

We will get us the following result:

"hits": [
   {
      "_index": "movies",
      "_type": "movie",
      "_id": "AU6OkygJidzUtyfB9L2D",
      "_score": 1,
      "_source": {
         "title": "Apocalypse Now",
         "director": "Francis Ford Coppola",
         "year": 1979,
         "genres": [
            "Drama",
            "War",
            "Foo",
            "Bar"
         ]
      }
   },
   {
      "_index": "movies",
      "_type": "movie",
      "_id": "AU6OkwUeidzUtyfB9L1q",
      "_score": 1,
      "_source": {
         "title": "Apocalypse Now",
         "director": "Francis Ford Coppola",
         "year": 1979,
         "genres": [
            "Drama",
            "War",
            "Foo"
         ]
      }
   }
]

What if we want to see the movies which is in War, Foo and Bar genres at the same time? Well, there are probably other ways of doing this but here is how I hacked it together with bool and term filter:

GET movies/movie/_search
{
  "query": {
    "filtered": {
      "query": {
        "match_all": {}
      },
      "filter": {
        "bool": {
          "must": [
            { 
              "term": {
                "genres": "War"
              }
            }
          ],
          
          "must": [
            { 
              "term": {
                "genres": "Foo"
              }
            }
          ],
          
          "must": [
            { 
              "term": {
                "genres": "Bar"
              }
            }
          ]
        }
      }
    }
  }
}

The result:

"hits": [
   {
      "_index": "movies",
      "_type": "movie",
      "_id": "AU6OkygJidzUtyfB9L2D",
      "_score": 1,
      "_source": {
         "title": "Apocalypse Now",
         "director": "Francis Ford Coppola",
         "year": 1979,
         "genres": [
            "Drama",
            "War",
            "Foo",
            "Bar"
         ]
      }
   }
]

The exact match is a whole different story :)

Elasticsearch Installation and a Few Core Concepts

So, I have been having my way with Elasticsearch for a few weeks now and it's time for me to blog about it :) In this post, I will only highlight a few things here which were useful to me at the beginning.
2014-09-25 15:01
Tugberk Ugurlu


So, I have been having my way with Elasticsearch for a few weeks now and it's time for me to blog about it :) I hear what you say :) This is yet another 101 blog post but settle down :) I have a few selfish reasons here. Blogging on a new technology is a way for me to grasp it better. Also, during this time, I have been also looking into Azure Search, an hosted search service solution by Microsoft. You should check this service out, too if what you want is to have an easily scalable hosted search service solution.

OK, what were we talking about? Oh yes, Elasticsearch :) It's awesome! I mean it, seriously. If you haven't looked at it, you should check it out. It's a search product which makes data search and analytics very easy. It's built on top of famous search engine library Apache Lucene. Elasticsearch also has a great documentation. So, I will only highlight a few things here which were useful to me at the beginning.

Setting it Up

To get started with Elasticsearch, you can download it from here. At the time of this writing, version 1.3.2 was the latest version. What you need to do is fairly simple: download the zip file, extract it, navigate to bin directory and run Elasticsearch from the command line.

1-Screenshot 2014-09-25 13.20.48

As we didn't specify any configuration values, Elasticsearch started with the configuration defined inside the /config/elasticsearch.yml file. If you didn't touched that file either, it will be exposed through localhost:9200. Elasticsearch server is exposed to the World through its HTTP API and when you send a GET request to the root level, you will get the server info:

2-Screenshot 2014-09-25 13.23.47

You can also see inside the response that the node even assigned itself a random name: Mole Man in this case. You can start working with this elasticsearch node using your choice of an HTTP client but I really recommend installing Marvel which ships with a developer console that allows you to easily issue calls to Elasticsearch’s HTTP API. To install marvel, you need to run the following command which you are under the elasticsearch/bin path:

plugin -i elasticsearch/marvel/latest

After you are done with the installation, you should restart the server and from there, you can navigate to localhost:9200/_plugin/marvel/sense/index.html on your favorite browser:

3-Screenshot 2014-09-25 13.43.02

Now, you can start issuing request to you Elasticsearch server right from your browser:

4-Screenshot 2014-09-25 13.47.13

The best part of this plugin is its autocomplete support while you are construction queries:

5-Screenshot 2014-09-25 13.49.29

It can even understand and provide you the options for your type fields which is pretty useful. Before going deeper, let's learn e few fundamentals about Elasticsearch.

Basics 101

When working with Elasticsearch, you will come across a few core concepts very often. I think knowing what they are beforehand will help you along the way.

Index is the container for all your documents. Each document inside your index will have a few shared properties. One of those properties is type. Each document inside your index will have a type and each type will have a defined mapping (sort of a schema). The type and its mapping is not required to be defined upfront. Elasticsearch can create the type and its mapping dynamically. However, there are lots of useful things you can achieve with altering the default mapping such as changing the analyzer for a field.

At first glance, it seems that we can only search on types but that's not the case actually. You can even search the entire Elasticsearch node if you want:

6-Screenshot 2014-09-25 14.08.02

I have a few indexes created here and those indexes have documents in one or multiple different types. When I search for a word "madrid"  here, I got 568 hits throughout the node. I have some hits as movie type from movies-test index and some hits as status type from my_twitter_river index. This could be pretty powerful depending on your scenario if you embrace Elasticsearch’s schema free nature. We can also search on one index and that would give us the ability to search for the types under that particular index.

I mentioned that each type has its mapping. You can see the mapping of a type by sending a GET request to /{index}/{type}/_mapping:

7-Screenshot 2014-09-25 17.23.17

Twitter River Plugin for Elasticsearch

If you are just getting started with Elasticsearch, what I recommend for you is to get some data in using the Twitter River Plugin for ElasticSearch. After you configure and run it for a few hours, you will have insane amount of data ready to play with. 

8-Screenshot 2014-09-25 17.28.10

Resources

Tags