The why? Although, there have been great products that existing since a long time, which does exactly and more than what this project is trying to accomplish, namely – Celery. However, the basic premise of this effort is again based on a very simple idea, which happens to be the mainstay of python community and that is – KISS. Most of the time, a project begins humbly with small code based and small set of functionality and over time, tries to do a lot of things and while doing so, ends up accumulating huge sets of features that are overwhelming for a beginners to start with. Therefore, keeping the same problem in focus, I decided to make a celery like[…]

To begin, I would first like to answer the obvious question: Why would anyone want to use lua when already using python? The answer is simple, legacy code. This experiment arose from the evaluation of possible candidates to replace an existing c/c++ system with limited throughput in terms of request per second with something like python or erlang. In this post, we will be focussing on python, since it was an experiment borne out of my own curiosity. As for the usage of lua, let’s say there was a lot of business logic already written in it and the requirement is to reuse that code base. I chose to use LuaJIT, because it already proved to be fast enough for[…]

Lately, I had a task of creating a workflow manager and after much testing and feature evaluation of products such as Luigi, TaskFlow, Azkaban, etc. I settled on doing it with plain celery. I chose to use canvas functionality offered by celery and was floored by what I found. It was easy to create workflows using chain and group feature of celery and it was also very performant system. The configuration for system that was finalized had rabbitmq as broker and mongodb as result backend. Since the use case was to have completely asynchronous flows, there was hardly any need for result backend except for celery’s own use when handing subtask level errors. Since what we were building itself was[…]