Long-Running Asynchronous Operations, Displaying Their Events and Progress on Clients

I want to share a few thoughts that I have been keeping to myself on showing progress for long-running asyncronous operations on a system where individual events can be sent during ongoing operations.
2016-07-30 16:33
Tugberk Ugurlu


With today's modern applications, we end up with lots of asynchronous operations happening and we somehow have to let the user know what is happening. This, of course, depends on the requirements and business needs.

Let's look at the Amazon case. When you buy something on Amazon, Amazon tells you that your order has been taken. However, it doesn't really mean that you have actually purchased it as the process of taking the payment has not been kicked off yet. The payment process probably goes through a few stages and at the end, it has a few outcome options like success, faılure, etc. Amazon only gives the user the change to know about the outcome of this process which is done through e-mail. This is pretty straightforward to implement (famous last words?) as it doesn't require things like real-time process notifications, etc..

On the other hand, creating a VM on Microsoft Azure has different needs. When you kick the VM creation operation through an Azure client like Azure Portal, it can give you real-time notifications on the operation status to show you in what stage the operation is (e.g. provisioning, starting up, etc.).

In this post, I will look at the later option and try to make a few points to prove that the implementation is a bit tricky. However, we will see a potential solution at the end and that will hopefully be helpful to you as well.

Problematic Flow

Let's look at a problematic flow:

  • User starts an operation by sending a POST/PUT/PATCH request to an HTTP endpoint.
  • Server sends the 202 (Accepted) response and includes a some sort of operation identifier on the response body.
  • Client subscribes to a notification channel based on this operation identifier.
  • Client starts receiving events on that notification channel and it decides on how to display them on the user interface (UI).
  • There are special event types which give an indication that the operation has ended (e.g. completed, failed, abandoned, etc.).
    • The client needs to know about these event types and handle them specially.
    • Whenever the client receives an event which corresponds to operation end event, it should unsubscribe from the notification channel for that specific operation.

We have a few problems with the above flow:

  1. Between the operation start and the client subscribing to a notification channel, there could be events that potentially could happen and the client is going to miss those.
  2. If the user disconnects from the notification channel for a while (for various reasons), the client will miss the events that could happen during that disconnect period.
  3. Similar to above situation, the user might entirely shut its client and opens it up again at some point later to see the progress. In that case, we lost the events that has happened before subscribing to the notification channel on the new session.

Possible Solution

The solution to all of the above problems boils down to persisting the events and opening up an HTTP endpoint to pull all the past events per each operation. Let's assume that we are performing payment transactions and want to give transparent progress on this process. Our events endpoint would look similar to below:

GET /payments/eefcb363/events

[
     {
          "id": "c12c432c",
          "type": "ProcessingStarted",
          "message": "The payment 'eefcb363' has been started for processing by worker 'h8723h7d'.",
          "happenedAt": "2016-07-30T11:05:26.222Z"
          "details": {
               "paymentId": "eefcb363",
               "workerId": "h8723h7d"
          }
     },

     {
          "id": "6bbb1d50",
          "type": "FraudCheckStarted",
          "message": "The given credit card details are being checked against fraud for payment 'eefcb363' by worker 'h8723h7d'.",
          "happenedAt": "2016-07-30T11:05:28.779Z"
          "details": {
               "paymentId": "eefcb363",
               "workerId": "h8723h7d"
          }
     },

     {
          "id": "f9e09a83",
          "type": "ProgressUpdated",
          "message": "40% of the payment 'eefcb363' has been processed by worker 'h8723h7d'.",
          "happenedAt": "2016-07-30T11:05:29.892Z"
          "details": {
               "paymentId": "eefcb363",
               "workerId": "h8723h7d",
               "percentage": 40
          }
     }
]

This endpoint allows the client to pull the events that have already happened, and the client can mix these up with the events that it receives through the notification channel where the events are pushed in the same format.

With this approach, suggested flow for the client is to follow the below steps in order to get 100% correctness on the progress and events:

  • User starts an operation by sending a POST/PUT/PATCH request to an HTTP endpoint.
  • Server sends the 202 (Accepted) response and includes a some sort of operation identifier (which is 'eefcb363' in our case) on response body.
  • Client subscribes to a notification channel based on that operation identifier.
  • Client starts receiving events on that channel and puts them on a buffer list.
  • Client then sends a GET request based on the operation id to get all the events which have happened so far and puts them into the buffer list (only the ones that are not already in there).
  • Client can now start displaying the events from the buffer to the user (optionally in chronological order) and keeps receiving events through the notification channel; updates the UI simultaneously.

Obviously, at any display stage, the client needs to honor that there are special event types which give an indication that the operation has ended. The client needs to know about these types and handle them specially (e.g. show the completed view, etc.). Besides this, whenever the client receives an event which corresponds to operation end event, it should unsubscribe from the notification channel for that specific operation.

However, the situation can be a bit easier if you only care about percentage progress. If that's case, you may still want to persist all events but you only care about the latest event on the client. This makes your life easier and if you decide that your want to be more transparent about showing the progress, you can as you already have all the data for this. It is just a matter of exposing them with the above approach.

Conclusion

Under any circumstances, I would suggest you to persist operation events (you can even go further with event sourcing and make this a natural process). However, your use case may not require extensive and transparent progress reporting through the client. If that's the case, it will certainly make your implementation a lot simpler. However, you can change your mind later and go with a different progress reporting approach since you have been already persisting all the events.

Finally, please share your thoughts on this if you see a better way of handling this.

NDC Oslo 2016 in a Nutshell

I had the privilege to attend NDC Oslo 2016 as a speaker this year. It was a fabulous experience. With this post, I want to share my talk and the list of sessions that grabbed my attention.
2016-06-25 12:03
Tugberk Ugurlu


As I blogged earlier, I had the privilege to attend NDC Oslo 2016 as a speaker this year. It was a fabulous experience and I took a different approach at NDC Oslo this year and rather than improving my existing knowledge, I decided to take different perspective on problems. There were really a few areas I wanted to get more information on from the World class experts:

  • Machine Learning
  • Functional Programming
  • Containerization
  • Soft Skills

Of course, this is NDC and you have people like Mark Rendle and Rob Conery. So, it was inevitable that I ended up at sessions which were just fun :) Nevertheless, it was pretty useful 3 days for me at the conference and it feels like I achieved my end goal.

Sessions

2016-06-08 09.04.02

One other amazing aspect of NDC Conferences is that all of the talks are being recorded and nearly all of those recordings are now up on Vimeo for public consumption :)

Here is the list of sessions I had a chance to attend in person:

All of them were really helpful and gave me a different perspective on the topics. Aside from those sessions, here are the talks that I really wanted to attend but I couldn't + I will definitely watch:

The schedule was full of amazing talks as you can see and there are also some other talks that seem interesting but I will probably not have time to look at:

My Session

At the conference I gave a talk on Zero Downtime Deployments. I tried to give useful and practical guidance based on my true experiences. I hope it was useful for everyone attended the talk. The recording video of my talk is also available now.

Getting Into the Zero Downtime Deployment World - Tugberk Ugurlu from NDC Conferences on Vimeo.

The slides of the talk is also available under my Speakerdeck account.

Finally, you can find the demo sample I used in my talk to simulate a zero-downtime deployment process under my GitHub account. It also has great instructions on how to run the sample. So, definitely give it a go.

demo-0

Off to Oslo for NDC Developer Conference

Next week, I am off to Oslo for one of my favorite conferences: NDC Oslo and this time is a little bit more special as I am one of the speakers this year, talking about zero-downtime deployments.
2016-06-05 20:50
Tugberk Ugurlu


Next week, I am off to Oslo for one of my favorite conferences: NDC Oslo. I have been to NDC Oslo before in 2014 but this time is a little bit more special as I am one of the speakers this year. Seeing the list of awesome speakers at conference makes me so excited and nervous at the same time :)

As for my topic, I am going to be talking about zero-downtime deployments. I will tell you about some patterns, practices and techniques that make it this challenging task easier, such as semantic versioning and blue/green deployments. We’ll also walk through an end-to-end demo of how a high traffic web application can survive the challenge of deployments. If you are going to be there, I hope you will also join me at my talk.

image

I can see lots of people from my Twitter timeline already tweeting about NDC Oslo. So, I am really sure that this is going to be an exciting event. I hope to see lots of friends and make new friends while I am there. See you in Oslo :)

Moving to ASP.NET Core RC2: Tooling

.NET Core Runtime RC2 has been released a few days ago along with .NET Core SDK Preview 1. At the same time of .NET Core release, ASP.NET Core RC2 has also been released. While I am migrating my projects to RC2, I wanted to write about how I am getting each stage done. In this post, I will show you the tooling aspect of the changes.
2016-05-22 14:04
Tugberk Ugurlu


.NET Core Runtime RC2 has been released a few days ago along with .NET Core SDK Preview 1. At the same time of .NET Core release, ASP.NET Core RC2 has also been released. Today, I started doing the transition from RC1 to RC2 and I wanted to write about how I am getting each stage done. Hopefully, it will be somewhat useful to you as well. In this post, I want to talk about the tooling aspect of the transition.

Get the dotnet CLI Ready

One of the biggest shift from RC1 and RC2 is the tooling. Before, we had DNVM, DNX and DNU as command line tools. All of them are now gone (RIP). Instead, we have one command line tool: dotnet CLI. First, I installed dotnet CLI on my Ubuntu 14.04 VM by running the following script as explained here:

sudo sh -c 'echo "deb [arch=amd64] https://apt-mo.trafficmanager.net/repos/dotnet/ trusty main" > /etc/apt/sources.list.d/dotnetdev.list'
sudo apt-key adv --keyserver apt-mo.trafficmanager.net --recv-keys 417A0893
sudo apt-get update
sudo apt-get install dotnet-dev-1.0.0-preview1-002702

This installed dotnet-dev-1.0.0-preview1-002702 package on my machine and I am off to go:

image

You can also use apt-cache to see all available versions:

image

Just to make things clear, I deleted ~/.dnx folder entirely to get rid of all RC1 garbage.

Get Visual Studio Code Ready

At this stage, I had the C# extension installed on my VS Code instance on my Ubuntu VM which was only working for DNX based projects. So, I opened up VS Code and updated my C# extension to latest version (which is 1.0.11). After the upgrade, I opened up a project which was on RC1 and VS Code started downloading .NET Core Debugger.

image

image

That was a good experience, I didn't dig into how to do that but I am not sure at this point why it didn't come inside the extension itself. There is probably a reason for that but not my priority to dig into right now :)

Try out the Setup

Now, I am ready to blast off with .NET Core. I used dotnet CLI to create a sample project by typing "dotnet new --type=console" and opened up project folder with VS Code. As soon as VS Code is launched, it asked me to set the stage.

image

Which got me a few new files under .vscode folder.

image

I jumped into the debug pane, selected the correct option and hit the play button after putting a breakpoint inside the Program.cs file.

image

Boom! I am in business.

image

Now, I am moving to code changes which will involve more hair pulling I suppose.

Resources

Your Git Repository is Your Deployment Boundary

In this short post, I will try to explain why I think that your Git repository should be your deployment boundary.
2016-05-22 10:53
Tugberk Ugurlu


Deployment of the software has been a constant challenge possibly from the very start. It could be a web application, HTTP services, a Play Station app, an application running inside a Raspberry PI. All have the challenges of deploying new changes. It even makes you go down the route of a different architecture to make the deployments scale. One of the big challenges of software deployments is that there is not one generic rule or practice that you can apply and it will all be shinny. This doesn't mean that we don't have generic practices or techniques around it. However, the way your software lives in its destination(s) and the way it's being consumed are factors that you need to take into consideration before coming up with a deployment strategy.

There is a little bit of catch-22 here as you sometimes shape the architecture around the deployment strategy (not saying this is a good or bad way, not exactly sure yet). So, there is a fine balance you need to hit here to grab the robes of success.

So, what I am going to tell you here now might not seem to fit for everyone but I believe that it is the first step to make your deployment strategy a viable one. I will throw it here: your Git repository is your deployment boundary. This idea is so subtle, easy to grasp and easy to reason from. What this really means that all the things you have inside one repository will be part of only one deployment strategy. The main reason for this is versioning. Git allows you to tag your repositories and you use these feature to put you into different deployment pipelines based on you're the type of the change (e.g. minor, patch or major). See my previous post on versioning builds and releases through SemVer for a bit more detail on this.  However, it will get messy if you try to maintain multiple different versions inside the same repository for different components. You will easily lose touch on how two components relate to each other over time.

Ask yourself this when you are structuring your repositories: are any of the components inside a repository has a different release cadence? If the question is yes, try to think of why and if the reasons are legitimate, give that components a new, separate home.

Tags