The why?

Although, there have been great products that existing since a long time, which does exactly and more than what this project is trying to accomplish, namely – Celery. However, the basic premise of this effort is again based on a very simple idea, which happens to be the mainstay of python community and that is – KISS. Most of the time, a project begins humbly with small code based and small set of functionality and over time, tries to do a lot of things and while doing so, ends up accumulating huge sets of features that are overwhelming for a beginners to start with. Therefore, keeping the same problem in focus, I decided to make a celery like project which will do less instead of more. It will need less expertise with python (yes, the project requires python) for creating heavy backend services, by focussing on writing those backend services instead of the infrastructural code to run those services.

What it supports?

As of now, the project is aimed at using the usual suspects of existing middleware technologies such as rabbitmq, redis, zmq and kafka. As of writing this blog, rabbitmq and redis are supported and work is being done for the rest two. It is written using asyncio module in python 3.6 and that is the version of python recommended to use it. In coming sections we will see, how this project does enough with less requirements. As soon as zmq is supported, the need for another middleware system, like in the case of other three, will cease to exist. More on that later.

How to get it?

The source code for asyncexec is available on Github.com and it is released under MIT license, as it seems to be a very open one and without much hassles. One can go through the README to install and use the project and following is the what it says, in order to install it.

 

  1. Create a virtual environment:
    $> virtualenv venv -p /usr/bin/python3.6
  2. Activate the virtual
    $> source venv/bin/activate
  3. Use pip install to get the module and also install all the required modules it has as dependencies
  4. pip install git+https://github.com/arshadansari27/[email protected]

How to use it?

The repository contains two examples to show how to use the system, under the asyncexec/examples folder.

  1. simple_queue_based; shows how to use a simple rabbitmq based queueing system to run handlers on sending the events as rabbitmq messages
  2. workflows: expands on the previous example and shows how to break down tasks as multiple commands and use concurrency to achieve high performance using parallel group commands or create a workflow by combining it with sequential group commands.

Instead of describing the whole example, I would only like to point the attention to a very simple way of using the system, which is shown below.

#import the class
 from asyncexec.exec import  AsyncExecutor
 
 #Create the object and give configuration in constructor
 async_executor = AsyncExecutor({
     "rabbitmq": [
         ['pyasync_core_request', 'pyasync_core_result']
     ]
 })
 
 # Define handlers
 @async_executor.handler
 def test_method(data):
     print("Hello", data)
     return str(data) + ': response'
 
 # Start the process
 async_executor.start() 
 
 # Example message to send
 # {“event”: “test_method”, “data”: …}

That is it. The test_method handler will be added as an observer (Observer Pattern) and the listener will call the observer to handle the event they are listening on. This mapping between handler and the event is done by sending the event as a message, where the key “event” would be the name of the handler method and the data key will provide the handler method with a single argument input to use for its execution.

Coming soon

Although the main idea is to keep it small and simple for handling backend services, it is still lacking and is currently too simplistic. Although not fit for production use at the moment, very soon it will be. The following are the major upcoming features that I intend to include in this project:

  • Kafka and ZMQ support
  • Queue based handler distinction to decide which instance can run which all handlers
  • Multiprocessing improvement to enable the user to decide which handler should be run in a separate process/thread.

These are the few urgent things that will be worked on in near future, however; I’m open to inputs from all sides and would love any kind of contribution to make this a success. Until next time.

  • Anurag Agnihotri

    Nice post.

  • srahul07

    Nice… Keep on posting more about this.

    • Thanks. I will keep posting as it develops, while covering the features added or discussing the feature that are planned to be added.