Elasticsearch Array Contains Search With Terms Filter

Here is a quick blog post on Elasticsearch and terms filter to achieve array contains search with terms filter
2015-07-14 22:17
Tugberk Ugurlu


Here is a quick blog post on Elasticsearch and terms filter while I still remember how the hell it works :) Yes, this is possibly the 20th time that I looked for how to achieve array contains functionality in Elasticseach and it's a clear sign for me that I need to blog about it :)

I created the index called movies (mostly borrowed from Joel's great Elasticsearch 101 blog post) and here is its mapping:

PUT movies/_mapping/movie
{
  "movie": {
    "properties": {
       "director": {
          "type": "string"
       },
       "genres": {
          "type": "string",
          "index": "not_analyzed"
       },
       "title": {
          "type": "string"
       },
       "year": {
          "type": "long"
       }
    }
  }
}

The genres field mapping is important here as it needs to be not analyzed. I also indexed a few stuff in it:

POST movies/movie
{
    "title": "Apocalypse Now",
    "director": "Francis Ford Coppola",
    "year": 1979,
    "genres": ["Drama", "War", "Foo"]
}

POST movies/movie
{
    "title": "Apocalypse Now",
    "director": "Francis Ford Coppola",
    "year": 1979,
    "genres": ["Drama", "War", "Foo", "Bar"]
}

POST movies/movie
{
    "title": "Apocalypse Now",
    "director": "Francis Ford Coppola",
    "year": 1979,
    "genres": ["Drama", "Bar"]
}

Now, I am interested in finding out the movies which is in War or Foo genre. The way to achieve that is the terms filter as mentioned:

GET movies/movie/_search
{
  "query": {
    "filtered": {
      "query": {
        "match_all": {}
      },
      "filter": {
        "terms": {
          "genres": ["War", "Foo"]
        }
      }
    }
  }
}

We will get us the following result:

"hits": [
   {
      "_index": "movies",
      "_type": "movie",
      "_id": "AU6OkygJidzUtyfB9L2D",
      "_score": 1,
      "_source": {
         "title": "Apocalypse Now",
         "director": "Francis Ford Coppola",
         "year": 1979,
         "genres": [
            "Drama",
            "War",
            "Foo",
            "Bar"
         ]
      }
   },
   {
      "_index": "movies",
      "_type": "movie",
      "_id": "AU6OkwUeidzUtyfB9L1q",
      "_score": 1,
      "_source": {
         "title": "Apocalypse Now",
         "director": "Francis Ford Coppola",
         "year": 1979,
         "genres": [
            "Drama",
            "War",
            "Foo"
         ]
      }
   }
]

What if we want to see the movies which is in War, Foo and Bar genres at the same time? Well, there are probably other ways of doing this but here is how I hacked it together with bool and term filter:

GET movies/movie/_search
{
  "query": {
    "filtered": {
      "query": {
        "match_all": {}
      },
      "filter": {
        "bool": {
          "must": [
            { 
              "term": {
                "genres": "War"
              }
            }
          ],
          
          "must": [
            { 
              "term": {
                "genres": "Foo"
              }
            }
          ],
          
          "must": [
            { 
              "term": {
                "genres": "Bar"
              }
            }
          ]
        }
      }
    }
  }
}

The result:

"hits": [
   {
      "_index": "movies",
      "_type": "movie",
      "_id": "AU6OkygJidzUtyfB9L2D",
      "_score": 1,
      "_source": {
         "title": "Apocalypse Now",
         "director": "Francis Ford Coppola",
         "year": 1979,
         "genres": [
            "Drama",
            "War",
            "Foo",
            "Bar"
         ]
      }
   }
]

The exact match is a whole different story :)

Playing Around with Docker: Hello World, Development Environment and Your Application

I have been also looking into Docker for a while now. In this post, I am planning to cover what made me love Docker and where it shines for me.
2015-07-13 19:16
Tugberk Ugurlu


When you have a urge to blog about something, it's mostly the case that you have just learnt something :) Well, it's the case for me now. I have been looking into Linux space for a while now and I didn't expect it to go this smooth. It is wonderful and I must admit that I have been missing on too many nice stuff by sticking with Windows for my development environment. However, I have one thing, one big thing that I have no regret at all: .NET ecosystem. It's one of the great ecosystems to write application on top and as it's now so easy to get it into non-Windows environments, the entry door for those environments are now wide open to me.

I have been also looking into Docker for a while now and I was mostly trying to understand the concept as it's so distant to you if you were only developing on Windows for long time. After the concept was clear, the fun part started to take place. In this post, I am planning to cover that part and show you what made me love Docker.

Why Docker

Here is why I think Docker is useful for me (not much different than the other people's reasons):

  • Repeatable, declarative environments. This can get much better with Docker compose for your development, CI, QA (a.k.a. your Pre-Production) environments. 
  • Read, try and tear down when you are learning a new tool like Redis, RabbitMQ, etc. Just run the docker run command and create the container. Play with the tools on that container and remove the container at the end.
  • One way of deploying stuff. AWS, Azure, whatever. Wherever you go, you will use the same script to deploy your stuff.
  • Shifting your thinking to modularize the hell out of your solution (microservices, there I said it). This can open up insane opportunities. For example, developing each part of your application with the stack that is suitable for the job. Not only you will preserve your sanity by using the right tool but also you will have different parts inside your solution which can be developed separately by separate people who has different skill sets. I strongly suggest the .NET Rocks podcast on Building Microservices with Howard Dierking to understand more about this.
  • I am not sure about this but Docker also makes it really trivial for people to dockerize repro environments for issues.

There will be possibly more by these are the stuff that made me love Docker.

Hello World

I will assume that you installed Docker and you are ready to go. In my case here, I am using Ubuntu 14.04 LTS but I assume it should be the same on OS X as well.

As you can expect from Docker, the "Hello World" example is also declared and packaged up (A.K.A. dockerized). To get the "Hello World" example running, just run the below command:

docker run ubuntu:14.04 /bin/echo 'Hello world'

What happens when you do this is explained under Docker Hello World docs but briefly, you have a container based on Ubuntu 14.04 image and ran echo 'Hello World' in it and exited.

Screenshot from 2015-07-13 06^%10^%46

As mentioned, this container will not live after the echo is finished its job but it is still there to run. If you run the below command, you will see that container is there:

docker ps -a

Screenshot from 2015-07-13 06^%11^%51

We start the container by running the following command based on the container ID we retrieved from the docker ps output:

docker start --attach 6a174ac370a2

Screenshot from 2015-07-13 06^%17^%28

We also used --attach switch here to attach STDOUT/STDERR and forward signals and that's why we are able to see the hello world written in our console. Let's see a more realistic container example this time by getting an alive container up:

docker run -d ubuntu:14.04 /bin/sh -c "while true; do echo hello world; sleep 1; done"

This example is the exact same example you can find on "A daemonized Hello world" section of Docker Hello World doc. The interesting stuff here is the -d switch which tells Docker to run the container and put it in the background. If we run docker ps now, we should see that the container is still at the running state:

Screenshot from 2015-07-13 06^%38^%03

We can attach to a running container's STDOUT/STDERR by running the below command based on the container ID:

docker attach ff2847155ced

Screenshot from 2015-07-13 06^%39^%43

You can detach from the container and leave it running with CTRL-p CTRL-q.

Also, you should have noticed that the first run command we have run took a while to get it up because it downloaded all the images from the registry. However, the second one was instantaneous as Ubuntu 14.04 was already there. So, we can understand from here that Docker images are immutable and composable which is great. You can look at the images you have under your host by running docker images command.

Screenshot from 2015-07-13 06^%46^%10

Development Environment

As mentioned before, Docker makes it super easy to get stuff in and try them out. For example, Redis is on Docker registry and I can just run it as another container:

docker run --name my-redis -d redis:3.0.2

Screenshot from 2015-07-13 06^%59^%06

Screenshot from 2015-07-13 06^%59^%34

Screenshot from 2015-07-13 07^%01^%33

I am seeing here that the TCP port 6379 is also exposed which is the port that Redis is exposed. However, I need to know the IP address of this host to connect to this Redis instance from the host. We can figure out the IP address of a running container through the inspect command:

docker inspect --format '{{ .NetworkSettings.IPAddress }}' dfaf0cf33467

Screenshot from 2015-07-13 07^%08^%10

Now, I can connect to this Redis instance with redis-cli tool I have installed in my host:

redis-cli -h 172.17.0.10 -p 6379

Screenshot from 2015-07-13 07^%13^%38

I can play with this as long as I want and I can stop the container whenever I don't need it anymore. I can follow this process for nearly everything (e.g. Ruby, GoLang, Elasticsearch, MongoDB, RabbitMQ, you-name-your-thing, etc.). For example, get yourself a python development environment, you can run the following docker run command:

docker run -t -i python:2.7.10 /bin/bash

This will get you an interactive container with Python installed in it and you can do anything you want with it:

Screenshot from 2015-07-13 07^%37^%25

When you’re done you can use the exit command or enter Ctrl-D to finish your interactive session inside the container and effectively stop the container. This container is still in your easy reach. You can start it again by obtaining its ID by running docker ps -a and running the start command with the container ID.

You might wonder how tools like Redis, MongoDB and Elasticsearch fit into this world as they need to persist data on disk but Docker containers in nature are created and torn down without any worry. This is a well-thought problem and Docker has a solution for this problem with Data Volumes.

Your Application

This is all great and shiny but where does our application fit into this? There are some many different approaches that you can take with Docker for your application but let me show you a straight from and a powerful approach which will possibly give you an idea.

For an application example, I have chosen to dockerize Octopus Deploy Library web application which works on top of Node.js. The way to achieve this is through the Dockerfile. It has a well written documentation and here is how the Dockerfile for Octopus Deploy library would look like:

FROM node:0.12.7

RUN ["npm", "install", "gulp", "-g"]
RUN ["npm", "install", "bower", "-g"]
RUN ["npm", "install", "grunt-cli", "-g"]

COPY . /app
WORKDIR /app

RUN ["npm", "install"]
RUN ["bower", "--allow-root", "install"]

EXPOSE 4000

ENTRYPOINT ["gulp"]

In my opinion, self-descriptiveness is the the best part of this file. We are defining here that the application image should be based on node:0.12.7 image, which has node.js stuff in it. Then, we run a few npm commands to install what we need. Later, we copy our stuff and change the working directory. Lastly, we install dependencies, expose the TCP port 4000 and specify the entry point command.

Octopus Deploy does its magic and gets the server up when you run gulp default task. That's why it's our entry point here.

We can now build our application image:

docker build -t octopus-library .

Screenshot from 2015-07-13 08^%33^%50

This kicks off the build and creates the image if everything goes well. Lastly, we can get the container up and running under our host using the same run command:

docker run -t -d -p 4040:4000 octopus-library

We use -p option here to map the internal 4000 TCP port to 4040 TCP port of the host. With this way, you can access the running application from the host through 4040 TCP port:

Screenshot from 2015-07-13 08^%50^%02

You can repeat the same steps by cloning my fork of Octopus Library and switching to docker branch.

Resources

Progressive .NET Tutorials 2015 and Recording Videos of My ASP.NET 5 Talks

Last Friday, I was at Progressive .NET Tutorials 2015 in London and I gave two talks on ASP.NET 5. Here are the recording videos and slides of my two ASP.NET 5 talks!
2015-07-07 21:45
Tugberk Ugurlu


Last Friday, I was at Progressive .NET Tutorials 2015 in London and I gave two talks on ASP.NET 5. The conference was really good, I had a chance to meet with new great people. One nice thing about this conference is that the sessions were recorded. You can check out all the recordings here. My two talks on ASP.NET 5 were also recorded:

ASP.NET 5: How to Get Your Cheese Back

Slides for this talk is also available under my Speakerdeck account:

Going Further with ASP.NET 5

Here are the links to several resources related to these talks:

Enjoy! :)

Slides of Introduction to Database Lifecycle Management Talk at /dev/summer 2015

Yesterday, I was at /dev/summer 2015, Cambridge and gave a talk on DLM (Database Lifecycle Management). I have uploaded the slides under my Speakerdeck account.
2015-06-28 11:57
Tugberk Ugurlu


Yesterday, I was at /dev/summer 2015, Cambridge. It was a really good event and I had a chance to attend a few talks on Go Language, Web Profiling and Open Source.

2015-06-27 16.36.26-1

2015-06-27 12.46.58

2015-06-27 11.40.33

I also gave a talk on DLM (Database Lifecycle Management), which is what we have been working at Redgate for a while to make it easy to adopt.

2015-06-27 16.31.18

I have uploaded the slides under my Speaker Deck account.

Also, here are the links I have at the end of the slides and a few more:

Upcoming Conferences That I am Speaking at

I have a few speaking activities lined up in upcoming weeks on ASP.NET 5 and DLM and I thought it would be good to share these with you all :)
2015-06-16 20:44
Tugberk Ugurlu


I have a few speaking activities lined up in upcoming weeks and I thought it would be good to share those with you all :) I am very excited about them as I don’t have much international speaking experience and these will be a really good learning opportunities for me, while having a chance to share the knowledge at the same time. As you can imagine, most of these talks are related to ASP.NET 5 which is the best thing that has happened to .NET Web stack if you ask me :) Here are all of them:

/dev/summer 2015, Cambridge (27th of June)

/dev/summer is of the great developer events in Cambridge and it happens twice a year. I was lucky to attend the /dev/winter 2015 this year and I am looking forward to /dev/summer 2015, too. I will be talking about DLM (Database Lifecycle Management) and I will try to set a clear understanding of the advantages of embracing DLM, where your process fits today and how you can apply the described patterns and practices. You can register for the conference and get your ticket here. Also, you can use the discount code of "Speakersfriend" which will give you a 30% discount. You can alternatively use this link to get this discount. You can view the full event schedule here.

image

Progressive .NET Tutorials 2015, London (3rd of July)

Progressive .NET Tutorials 2015 is a 3-days .NET developer conference in London and has a really good content from awesome speakers. I will be giving two talks on the last day of the event and both of them are about ASP.NET 5. The first one is titled as "ASP.NET 5: How to Get Your Cheese Back" which I will talk about the biggest, drastic changes in the upcoming version of ASP.NET but I will try to prove that these changes will make you smile. The next one is "Going Further with ASP.NET 5" which will be slides-free, dig-deep-into-the-bowels-of-this-new-runtime kind of talk. Check out the full schedule here and you can register for the event here.

image

That Conference 2015, Wisconsin Dells (12th of August)

I will also be speaking at That conference 2015 about ASP.NET 5: ASP.NET 5: How to Get Your Cheese Back. "Which conference? That Conference!" (obligatory That Conference joke :p) I am particularly excited for this one as it will be my first trip to US and I will make sure to whine about jet lag stuff on Twitter when I get there :) Jokes aside, you can see the conference schedule here and register for the conference here.

image

Thanks to these conferences, I am hoping to meet bunch of new and old frıends. Make sure to come and say hi if you are at one of the events :)

Tags