Long-Running Asynchronous Operations, Displaying Their Events and Progress on Clients

I want to share a few thoughts that I have been keeping to myself on showing progress for long-running asyncronous operations on a system where individual events can be sent during ongoing operations.
2016-07-30 16:33
Tugberk Ugurlu


With today's modern applications, we end up with lots of asynchronous operations happening and we somehow have to let the user know what is happening. This, of course, depends on the requirements and business needs.

Let's look at the Amazon case. When you buy something on Amazon, Amazon tells you that your order has been taken. However, it doesn't really mean that you have actually purchased it as the process of taking the payment has not been kicked off yet. The payment process probably goes through a few stages and at the end, it has a few outcome options like success, fa─▒lure, etc. Amazon only gives the user the change to know about the outcome of this process which is done through e-mail. This is pretty straightforward to implement (famous last words?) as it doesn't require things like real-time process notifications, etc..

On the other hand, creating a VM on Microsoft Azure has different needs. When you kick the VM creation operation through an Azure client like Azure Portal, it can give you real-time notifications on the operation status to show you in what stage the operation is (e.g. provisioning, starting up, etc.).

In this post, I will look at the later option and try to make a few points to prove that the implementation is a bit tricky. However, we will see a potential solution at the end and that will hopefully be helpful to you as well.

Problematic Flow

Let's look at a problematic flow:

  • User starts an operation by sending a POST/PUT/PATCH request to an HTTP endpoint.
  • Server sends the 202 (Accepted) response and includes a some sort of operation identifier on the response body.
  • Client subscribes to a notification channel based on this operation identifier.
  • Client starts receiving events on that notification channel and it decides on how to display them on the user interface (UI).
  • There are special event types which give an indication that the operation has ended (e.g. completed, failed, abandoned, etc.).
    • The client needs to know about these event types and handle them specially.
    • Whenever the client receives an event which corresponds to operation end event, it should unsubscribe from the notification channel for that specific operation.

We have a few problems with the above flow:

  1. Between the operation start and the client subscribing to a notification channel, there could be events that potentially could happen and the client is going to miss those.
  2. If the user disconnects from the notification channel for a while (for various reasons), the client will miss the events that could happen during that disconnect period.
  3. Similar to above situation, the user might entirely shut its client and opens it up again at some point later to see the progress. In that case, we lost the events that has happened before subscribing to the notification channel on the new session.

Possible Solution

The solution to all of the above problems boils down to persisting the events and opening up an HTTP endpoint to pull all the past events per each operation. Let's assume that we are performing payment transactions and want to give transparent progress on this process. Our events endpoint would look similar to below:

GET /payments/eefcb363/events

[
     {
          "id": "c12c432c",
          "type": "ProcessingStarted",
          "message": "The payment 'eefcb363' has been started for processing by worker 'h8723h7d'.",
          "happenedAt": "2016-07-30T11:05:26.222Z"
          "details": {
               "paymentId": "eefcb363",
               "workerId": "h8723h7d"
          }
     },

     {
          "id": "6bbb1d50",
          "type": "FraudCheckStarted",
          "message": "The given credit card details are being checked against fraud for payment 'eefcb363' by worker 'h8723h7d'.",
          "happenedAt": "2016-07-30T11:05:28.779Z"
          "details": {
               "paymentId": "eefcb363",
               "workerId": "h8723h7d"
          }
     },

     {
          "id": "f9e09a83",
          "type": "ProgressUpdated",
          "message": "40% of the payment 'eefcb363' has been processed by worker 'h8723h7d'.",
          "happenedAt": "2016-07-30T11:05:29.892Z"
          "details": {
               "paymentId": "eefcb363",
               "workerId": "h8723h7d",
               "percentage": 40
          }
     }
]

This endpoint allows the client to pull the events that have already happened, and the client can mix these up with the events that it receives through the notification channel where the events are pushed in the same format.

With this approach, suggested flow for the client is to follow the below steps in order to get 100% correctness on the progress and events:

  • User starts an operation by sending a POST/PUT/PATCH request to an HTTP endpoint.
  • Server sends the 202 (Accepted) response and includes a some sort of operation identifier (which is 'eefcb363' in our case) on response body.
  • Client subscribes to a notification channel based on that operation identifier.
  • Client starts receiving events on that channel and puts them on a buffer list.
  • Client then sends a GET request based on the operation id to get all the events which have happened so far and puts them into the buffer list (only the ones that are not already in there).
  • Client can now start displaying the events from the buffer to the user (optionally in chronological order) and keeps receiving events through the notification channel; updates the UI simultaneously.

Obviously, at any display stage, the client needs to honor that there are special event types which give an indication that the operation has ended. The client needs to know about these types and handle them specially (e.g. show the completed view, etc.). Besides this, whenever the client receives an event which corresponds to operation end event, it should unsubscribe from the notification channel for that specific operation.

However, the situation can be a bit easier if you only care about percentage progress. If that's case, you may still want to persist all events but you only care about the latest event on the client. This makes your life easier and if you decide that your want to be more transparent about showing the progress, you can as you already have all the data for this. It is just a matter of exposing them with the above approach.

Conclusion

Under any circumstances, I would suggest you to persist operation events (you can even go further with event sourcing and make this a natural process). However, your use case may not require extensive and transparent progress reporting through the client. If that's the case, it will certainly make your implementation a lot simpler. However, you can change your mind later and go with a different progress reporting approach since you have been already persisting all the events.

Finally, please share your thoughts on this if you see a better way of handling this.



Comments

Call Girls in bangalore
by Call Girls in bangalore on Saturday, Aug 20 2016 19:33:59 +00:00
Very helpful article. I was searching for this information long time but there are very limited resources.
Hot Call Girls Amritsar
by Hot Call Girls Amritsar on Wednesday, Aug 31 2016 09:32:08 +00:00
I really enjoyed reading your article. I found this as an informative and interesting post, so i think it is very useful and knowledgeable. I would like to thank you for the effort you have made in writing this article.
google drive login
by google drive login on Saturday, Sep 10 2016 08:07:04 +00:00
I like your amazing post.thanks for share this with us. Good and full of knowledge.
blog article writing
by blog article writing on Tuesday, Sep 20 2016 14:36:41 +00:00
The client needs to honor that there are special event types which give an indication that the operation has ended
outlooksignin
by outlooksignin on Thursday, Sep 29 2016 07:05:03 +00:00
These things may sound ridiculous, the next time you dig into the many aspects of it would appear logical. Article is truely awfully impressive
gmail.com
by gmail.com on Friday, Oct 07 2016 11:44:41 +00:00
Very helpful article. Much thanks for this
instagram login
by instagram login on Thursday, Oct 13 2016 10:02:16 +00:00
I like your amazing post.thanks for share this with us. Good and full of knowledge.
mikel
by mikel on Friday, Oct 14 2016 06:42:12 +00:00
Red's fastest footwear brand, its commercial value can be imagined. The late former British Princess Diana also has a lot of JimmyChoo specially designed for her cheap christian louboutin shoes, the most unfortunate that when the Princess Diana was a car accident, no time to retrieve JimmyChoo for her handmade pair of flat heels, as the shoes Designers always regret. The graduated from the London Institute of the arts of the Malaysian temple-level women's shoes designer, has a profound insight on the shoes. Regardless of how the trend of change, high heels is the most able to reflect the charm of women's christian louboutin sale shoes. He is good at four inches high heels in the industry is quite famous, because four inches high (about 12 cm) high heels can make women's beautiful body to get the most perfect show. His shoes for the guests often "comfort, elegance, classic" principle, comfort is the design of the louboutin pas cher fit the curves of the feet, so walk more convenient, so JimmyChoo are hand-made shoes for customers. The elegance and classic is to keep the vitality of the shoes, so that after a few years after the shoes are still able to look consistent with the current trend of aesthetic.
mikel
by mikel on Friday, Oct 14 2016 06:42:50 +00:00
Red's fastest footwear brand, its commercial value can be imagined. The late former British Princess Diana also has a lot of JimmyChoo specially designed for her cheap christian louboutin shoes, the most unfortunate that when the Princess Diana was a car accident, no time to retrieve JimmyChoo for her handmade pair of flat heels, as the shoes Designers always regret. The graduated from the London Institute of the arts of the Malaysian temple-level women's shoes designer, has a profound insight on the shoes. Regardless of how the trend of change, high heels is the most able to reflect the charm of women's christian louboutin sale shoes. He is good at four inches high heels in the industry is quite famous, because four inches high (about 12 cm) high heels can make women's beautiful body to get the most perfect show. His shoes for the guests often "comfort, elegance, classic" principle, comfort is the design of the louboutin pas cher fit the curves of the feet, so walk more convenient, so JimmyChoo are hand-made shoes for customers. The elegance and classic is to keep the vitality of the shoes, so that after a few years after the shoes are still able to look consistent with the current trend of aesthetic.
zapya
by zapya on Tuesday, Oct 25 2016 11:13:59 +00:00

Tags