Lately, I had a task of creating a workflow manager and after much testing and feature evaluation of products such as Luigi, TaskFlow, Azkaban, etc. I settled on doing it with plain celery. I chose to use canvas functionality offered by celery and was floored by what I found. It was easy to create workflows using chain and group feature of celery and it was also very performant system. The configuration for system that was finalized had rabbitmq as broker and mongodb as result backend. Since the use case was to have completely asynchronous flows, there was hardly any need for result backend except for celery’s own use when handing subtask level errors. Since what we were building itself was[…]

As much as we have come far in the field of AI, there is this sure gap between how generic humans capabilities are and how focused the AI is, regardless of how better they are at those specific areas of expertise. Following is my perception of what could be missing and how would that gap be filled to achieve Artificial Generic Intelligence. All the tools of AI: be it probabilistic graphical models, regressions or symbolic solvers, have one thing in common and that thing is bad input. To understand this let’s take an analogy, what if I write it article and then zip it before sharing? Everyone who would download it will know to use decompression before viewing the actual[…]