Versioning Software Builds Based on Git Tags and Semantic Versioning (SemVer)

I have been using a technique to set the build version on my CI (continuous integration) system, Travis CI, based on Git tags and semantic versioning (SemVer). In this post, I want to share this with you and give you an implementation of that in Bash.
2016-04-16 14:20
Tugberk Ugurlu


Let's start this post by setting the stage first and then move onto the problem. When a build is kicked off for your application/library/etc. on a CI (continuous integration) system like Travis CI or AppVeyor, you are most probably flowing a version number for that build no matter what type of tech stack you use. This is mostly to relate the artifacts, which the build will produce (e.g. Docker images, NuGet packages, .NET assemblies, etc.), with a particular context. This is really useful to be able to communicate and correlate stuff. A few scenarios:

  • Hey Mark, please take a look at foobar-1.2.3-rc.657 from our CI Docker registry. That has the issue I have mentioned. You can check it on that image.
  • Ow, barfoo-2.2.3-beta.362 NuGet package content misses a few assemblies that should have been there. Let's go back to build logs for this and check what went wrong.

Convinced? Good :) Otherwise, you won't find the rest of the article useful.

The other case is to flow a version number when you actually want to produce a release for your defined environments (e.g. acceptance, staging, production). In this case, you usually don't want to give an arbitrary version to your artifacts because the version will carry the high level information about the changes. There are three important intentions you can give here:

  • I am releasing something which has no behavior changes
  • I am releasing a new feature which doesn't break my existing consumers
  • Dude, brace yourself! I will break the World into half!

You can see Semantic Versioning 2.0.0 for more information about this.

So, what happens here is that we want to let the CI system decide on the version at some cases and take control over which version number to flow in some other cases. Actually, the first statement is not quite correct because you still want to have partial control over what version number to flow for your non-release builds. Here is an example case to highlight what I mean:

  • You started developing your application and shipped version 1.0.0.
  • Your CI system started flowing prerelease version based on 1.0.0 and also attached the build number to that version (e.g. 1.0.0-beta.54). Notice that it's wrong at this stage because you already shipped v1.0.0. So, it should really be something like 1.0.1-beta-54.
  • Now, you are shipping version 1.1.0 as you introduced a new feature.
  • After that change, you keep building the software and CI system keeps flowing version 1.1.0 based versions. This is a bit bad as you now don't have the chronological order and version order correlation.

So, what we want here is to assign a version based on the latest release version, which means that you want to have control over this process of assigning a version number. I have seen people having a text file inside the repository to hold the latest release version but that's a bit manual. I assume you kick a release somewhere and you already assign a version at that stage for releases. So, wouldn't it be bad to leverage this?

So, you probably understood my problems here :) Now, let me introduce a few key pieces which will play a role to solve this problem and then later, I will move onto the actual implementation to solve the problem.

Git Tags

Tagging is a feature of Git which allows you to mark specific points in repository's history. As the Git manual also states, people typically use this functionality to mark release points. This is super convenient for our needs here and gets two important things sorted for us:

  • A kick-off point for releases. Ultimately, release process will be kicked off when you tag a repository and push that tag to your remote.
  • Deciding the base version based on the latest release version.

Semantic Versioning

So, we have the tags. However, it doesn't mean that every tag is a valid version and you can also use Git's tagging feature for some other purposes. This is where SemVer comes into picture and you can safely assume that any tag which is a valid SemVer is for a release. This makes your life so much easier as you can rely on built-in tools like node-semver to help you out (as we will see shortly).

The other thing we have in the mix is to be able to increment the build version after a release. For example, we release version 2.5.6. The next build right after the release should have the version number bigger than 2.5.6. Seems easy as you can just increment the patch version, right? No! 2.5.6-beta is also a valid SemVer. We can go further with 2.5.6-beta.5+736287 which is also a valid SemVer. So, there is a pre-defined spec here and we can again leverage tools like node-semver to work with this domain nicely.

Solution and Bash Implementation

OK, all this information is super useful but how to make it work? Let me walk you through a solution I have introduced recently on a few of the projects I am working on. It's very trivial but that useful at the same time. However, keep in mind that there might be a few things I might have missed as I have been applying this not for a long time. In fact, here might even be better techniques on this that you know. If so, please comment here. I would love to hear them!

I want to example this in two stages and bring them together at the end.

Deciding on a Base Version

When the build is kicked off, one of the first things to do is to decide a base version. This is fairly trivial and here is the flow chart to describe this decision making process:

base-version

Here is how the implementation looks like in Bash:

#!/bin/bash

baseVersion=0.0.0-0
if semver "ignorethis" $(git tag -l) &>/dev/null
then
    baseVersion=$(semver $((semver $(git tag -l)) | tail -n1) -i prerelease)
fi

Keep in mind that I am fairly new to Bash. So, there might be wrong/bad usages here.

To explain what happens here with a bit more details:

  • We get all the tags for the repository as a list by running git tag -l
  • We pass this list to semver command-line tool to filter the invalid SemVer strings. Notice that there is another parameter we pass to semver here called "ignorethis". It's just there to cover cases when there is no tag so that semver command-line tool can return non-zero exit code.
  • If semver command-line tool exits with 0, we know that there is at least one tag which is a valid SemVer. So, we run tail -n1 on the semver output to retrieve the latest version and we increment it on its prerelease identifier. This is now our base version.
  • If there are no valid SemVer tags on the repository, we set 0.0.0-0 as the base version.

Decide on a Build Version

Now we have a base version and we now need to decide on a build version based on that. This is a bit more involved but again, very trivial to implement. Here is another flow chart to describe this decision making process:

build-version

And, here is how the implementation looks like in Bash (specific to Travis CI as it uses Travis CI specific environment variables):

if [ -z "$TRAVIS_TAG" ];
then
    if [ -z "$TRAVIS_BRANCH" ];
    then
        # can add the build metadata to indicate this is pull request build
        echo export PROJECT_BUILD_VERSION="$baseVersion.$TRAVIS_BUILD_NUMBER";
    else
        # can add the build metadata to indicate this is a branch build
        echo export PROJECT_BUILD_VERSION="$baseVersion.$TRAVIS_BUILD_NUMBER";
    fi
else 
    if ! semver $TRAVIS_TAG &>/dev/null
    then
        # can add the build metadata to indicate this is a tag build which is not a SemVer
        echo export PROJECT_BUILD_VERSION="$baseVersion.$TRAVIS_BUILD_NUMBER";
    else
        echo export PROJECT_BUILD_VERSION=$(semver $TRAVIS_TAG);
    fi 
fi

Notice that I am echoing commands rather than directly calling them. This is because of a fact that Travis CI doesn't flow the exports which happens inside a script file. Maybe it does but I was not able to get it working. Anyways, I am calling this script inside my .travis.yml file by evaluating the output like this: eval $(./scripts/set-build-version.sh)

I am not going to separately explain how this works as the flow chart is very easy to grasp (also the Bash script). However, one thing which is worth mentioning is the branch check. After we check if the build is for a branch, we do the same operation no matter what. This is OK for my use case but you can add special metadata to your version in order to indicate which branch the build has happened or whether it was a pull request.

Conclusion

I find this solution very straight forward to pick the version of the build and have a central way of kicking of a release process. I applied this on AspNetCore.Identity.MongoDB project, a MongoDB data store adapter for ASP.NET Core identity. You can also see how I am setting the build version, how I am using it and how I am kicking off a release process.

To bring everything together, here is the entire script to set the build version:

#!/bin/bash

baseVersion=0.0.0-0
if semver "ignorethis" $(git tag -l) &>/dev/null
then
    baseVersion=$(semver $((semver $(git tag -l)) | tail -n1) -i prerelease)
fi

if [ -z "$TRAVIS_TAG" ];
then
    if [ -z "$TRAVIS_BRANCH" ];
    then
        # can add the build metadata to indicate this is pull request build
        echo export PROJECT_BUILD_VERSION="$baseVersion.$TRAVIS_BUILD_NUMBER";
    else
        # can add the build metadata to indicate this is a branch build
        echo export PROJECT_BUILD_VERSION="$baseVersion.$TRAVIS_BUILD_NUMBER";
    fi
else 
    if ! semver $TRAVIS_TAG &>/dev/null
    then
        # can add the build metadata to indicate this is a tag build which is not a SemVer
        echo export PROJECT_BUILD_VERSION="$baseVersion.$TRAVIS_BUILD_NUMBER";
    else
        echo export PROJECT_BUILD_VERSION=$(semver $TRAVIS_TAG);
    fi 
fi

I hope this will be useful to you in some way and as said, if you have a similar technique or a practice that you apply for this case, please share it. Now, go and enjoy this spectacular weekend ;)

node-semver: CLI Tool for Semantic Versioning 2.0.0

In this post, I want to tell you about node-semver, a SemVer 2.0.0 parser command-line tool and a node.js library (the one that npm uses).
2016-04-14 16:47
Tugberk Ugurlu


This is yet another post from me on a tiny but a very useful tool. Last time, I have written about SqlLocalDB.exe Utility Tool which is a command-line tool to manage SQL Server Express LocalDB instances. Today, I want to tell you about node-semver, a SemVer 2.0.0 parser command-line tool and a node.js library (the one that npm uses). I am huge fan of SemVer. You cannot imagine how many places you can you use semantic versioning to make the internal and external communication easy for your software product. I plan to write more about a few different cases I figured out.

Going back to our topic, the acquisition of this tool is amazingly simple. You basically install this as a global npm package by running "npm install semver -g" and from there, you have access to semver command everywhere.

Image

There are a lot of great things you can do with this but I will give you a few of the basic and useful samples. First useful thing I noticed was the ability to validate a string against a SemVer 2.0.0 spec.

Screenshot from 2016-04-14 16^%40^%13

If the validation fails, it exists with non-zero exit code.

Screenshot from 2016-04-14 16^%38^%41

You can also increment a version:

Image(1)

By default, it increments the patch version but you can customize this if you want:

Image(2)

Image(3)

One useful feature that I have been using is to giving bunch of string values to node-semver and getting back the valid semantic version strings back:

Image(4)

Notice that it doesn't only sanitize the input, it also orders the versions on the standard output which is very convenient. I have an upcoming blog post where I will show you how I leverage this feature and a few others for a legitimate use case.

Lastly, I want to give you an example on its range feature which allows you to check if a version (or which versions if multiple versions are given) satisfies a range check:

Image(5)

So, did you like it? I hear you screaming "YES!" :) So, go spread this amazing tiny utility by sharing this blog post. There are more but these examples should be enough to give you an idea.

Microsoft Build 2016 in a Nutshell

Two weeks ago, I had an amazing opportunity to be at Microsoft Build Conference in San Francisco and I would like to share my experience about the conference with you in this post by highlighting what has happened and giving you my personal takeaways.
2016-04-09 18:59
Tugberk Ugurlu


Two weeks ago, I had an amazing opportunity to be at Microsoft Build Conference in San Francisco as an attendee thanks to my amazing company Redgate. The experience was truly unique and amount of people I have met there was huge. A bit late but I would like to share my experience about the conference with you in this post by highlighting what has happened and giving you my personal takeaways. You can also check out my tweets for the Build conference.

CeznGcRWAAE_QMw

Announcements

There were bunch of big and small announcements throughout the conference from Microsoft. Some of these were highlighted during two keynotes and some other announcements were spread to three days. I tried to capture all of them here but it's very likely I missed some of them (mostly the small ones):

Ce46IwXUsAA0Wil

Sessions

2016-03-31 15.28.14

Here is the list of sessions I have attended:

As much as I wanted to attend some other sessions, I missed some of them mostly due to clashes with other sessions. Luckily recordings for all Build 2016 sessions are available up on Channel 9. Here is my list of sessions to catch up:

There were also many good Channel 9 Live interviews. You can find them here. Here is a personal list of a few which are worth listening to:

IMG_1870

Personal Takeaways

All in all it has been a great conference and as stated, I am still catching up on the stuff that I have missed. Throughout the conference, I have picked up a few key points and I want to end the post with those:

  • I have seen more from Microsoft to make developers lives easier and more productive by enabling new tools (Bash on Ubuntu on Windows), supporting multiple platforms (Service Fabric to run on every environment including AWS, on-premises, Azure Stack and preview of Service Fabric on Linux), open sourcing more (some parts of Xamarin have gone open source) and making existing paid tools available for free (Xamarin is now free).
  • Microsoft is more focused on getting their existing services together and trying to give a cohesive ecosystem for developers. Service Fabric, Cognitive Services, Data Lake is a few examples of this.
  • .NET Core and CoreCLR is approaching to finalization for v1. After RC2, I don't suppose there will be much more features added or concepts changing.
  • I think this is the first time I have seen stabilization on client Apps story for Microsoft. Universal Windows Platform (UWP) was the focus on this area this year and it was the same on previous year.
  • I am absolutely happy to see Microsoft abandoning Windows Phone day by day. There was no direct sessions on it during the conference.
  • There were more steps towards making software to manage people's lives in a better way. Skype Bot Framework was one of these steps.
  • Microsoft (mostly Azure group) invests on IoT solutions heavily. Azure Functions and new updates on Azure IoT Suite are just a few signs of this.
  • Azure Resource Manager (ARM) and ARM templates are getting a lot of love from Microsoft and it's the way they push forward. They even build new services on Azure on top of this.

My Summary of DevConf 2016

I had an amazing time in Johannesburg this week and had the privilege to attend DevConf 2016 as a speaker. I gave a talk on architecting polyglot-persistent solutions and you can find the slides and resource in this post.
2016-03-10 16:23
Tugberk Ugurlu


I had an amazing time in Johannesburg this week and had the privilege to attend DevConf 2016 as a speaker. I got to travel to South Africa for the first time and I really liked the country, especially the food :) Also, I tried to be a social developer and had a chance to meet with a few new amazing folks like Colin Dembovsky, Lisa Basel, Chris Tite, Mark Pearl and many others that I cannot remember right now after a few beers at Dubai airport while writing this blog post :)

During the conference, I attended a few other talks apart from giving one. Here are all of them:

  • A year of dealing with RabbitMQ
  • Lap around Azure Machine Learning
  • Getting Started with Analytics (GTM and GA)
  • Brownfield TDD: How to eat an elephant?
  • Adjust your behaviour and be surprised how much you can influence your team
  • Building real world microservices using Node.js

I generally like to summarize my impressions I gained throughout the conference (e.g. Codemash 2016, NDC Oslo 2014). Here are those from DevConf:

  • People are pulling away from so-called Microsoft stack as much as it makes sense, especially on the data storage part. This is a good thing for everyone, even for Microsoft. Microsoft being more open has the biggest effect on this.
  • People who experienced relational database architecture where the business logic sits inside the database seems to have learnt the lessons very well.
  • Microservices had its appearance throughout the conference again :)
  • Polyglot persistence seems to be what most of the people are applying today without knowing the term that much :) Not knowing the term is no problem at all. Remember: this type of terms (e.g. Microservices) help communicate during planning processes and discussions easier.
  • Application and Database Lifecycle management is more about the culture and less about the tools. People who want to adopt this culture seek advice from consulting companies, they need tools just to get the job done.
  • Infrastructure as code is grabbing more and more attention.

My Talk: Architecting Polyglot-persistent Solutions

Apart from a few logistical problems during my talk (like electricity going out completely!), it was really good overall. This was very first time I have given this talk and it was based on the experience I had over a few years on Zleek.

2016-03-08 10.29.04

You can find the slides under my Speakerdeck account here.

Here are also a few more links that you may find useful on this topic:

Special Thanks

I would like to thank all the DevConf team who made this conference happen and all the audience for their amazing attention for the conference. I am especially impressed with the attention to the detail that the DevConf team has shown.

Upcoming Conferences and Talks

I am going to be at a few conferences in upcoming weeks and I would like to share them here with you. If you are going to be around for any of the below events, let's meet and say hi to each other :)
2016-02-27 16:45
Tugberk Ugurlu


I am going to be at a few conferences in upcoming weeks and I would like to share them here with you. Main objective here is to tell you about where I am going to be and this should help meeting new people and learning about different experiences. Jeremy Clark, a friend I met at Codemash 2016, has an amazing blog post on becoming a social developer. I encourage you to check that out to see why and how.

DevConf, Johannesburg (8th of March)

devconf

I am very, very excited about DevConf. Source of this excitement is the talk I will deliver there and the content of the conference. I will be presenting on architecting polyglot-persistent solutions as part of the Persistence and Data track. This is a topic which is very close to my heart as I had the first hand experience while working on Zleek on what a big difference this type of architecture can make on your software product. It will also be the first time I will deliver this talk.

Rest of the agenda also looks pretty impressive. So, I am sure this will be well worth the long trip to South Africa :)

Microsoft Build 2016, San Francisco (30th of March - 1st of April)

image

At the end of March, I will also be in San Francisco to attend Microsoft Build conference. This is also very exciting for several reasons. Obvious one is that there will be a lot of existing and soon-to-be friends there from the community and this is a very developer centric conference in view of Microsoft products. Also, I don't get to attend conferences that often as an attendee. I am sure I will feel the comfort of gliding through the session rooms and not trying to prepare for a talk. If you add the fact that this is going to be my first trip to San Francisco, it will be a real fun :)

Last year, my wish for Build Conference announcements came true with Visual Studio Code and I am very much looking forward to this year's announcements, too.

I T.A.K.E. Unconference 2016, Bucharest (19th - 20th of May)

image

At I T.A.K.E. Unconference 2016, I will talk about two very interesting topics and I think both of them are very interesting considering the type of software solutions and the way we produce them nowadays.

I am very much looking forward both of them since it's going to be the first time that I present these sessions. It seems like it's still possible to register and you can also check the rest of the schedule out here.

If you are going to be around for any of the above events, let's meet and say hi to each other :)

Tags