Project

General

Profile

Task #4169

Task #4163: [ce, ee] celery refactor + upgrade

re-architecture celery support

Added by Marcin Kuzminski [CTO] over 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Start date:
12.08.2016
Due date:
% Done:

0%

Estimated time:
Sorting:
Commit Number:

Description

Currently celery is a mess. We need to re-think how to handle the celery workers inside our application.

we should plan around:

  • start with fresh approach to celery, so we cen remove all the magic code we have now
  • upgrade celery to latest version
  • port our wrapper (ResultWrapper) on latest celery tasks code. This is to handle fallback execution (means execute sync when exceptions, or celery is down, or misconfigured connection to celery)
  • think about paster celeryd, probably we should create new executable, or we keep it this way for backward compat
  • We should design our new celery to make sure it has appl required registry items, as well as the pylons stuff so each task plays out nicely.

Few gotachas from old code.

  • ResultWrapper to handle taks nicer than standard celery
  • DBSession cleanups, on execution (the @dbsession decorator)

Also available in: Atom PDF