Project

General

Profile

Actions

Task #4139

closed

Task #4163: [ce, ee] celery refactor + upgrade

Bring celery workers up to shapen with initializing pyramid + pylons stack

Added by Marcin Kuzminski [CTO] over 7 years ago. Updated over 7 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
31.07.2016
Due date:
% Done:

0%

Estimated time:
Sorting:
Commit Number:

Description

It seems currently celery workers are not properly initialized and the stack is breaking on some tasks that require pyramid elements

Actions #1

Updated by Daniel D over 7 years ago

Celery task do not have access to pylons/pyramid elements such as:

  • url generator
  • request context
  • user that triggered a task

Problems caused by this:

  • create_repo at the moment triggers a CreateRepoEvent which requires the request context/user and url generator to create the url for the repo
Actions #2

Updated by Daniel D over 7 years ago

  • Status changed from New to In Progress
Actions #3

Updated by Daniel D over 7 years ago

After investigation our options are:

  1. Subclass celery.Task and make it pass the request object to celery worker, proxied and reapplied as a pyramid request context when the task is run, this will require upgrading celery as the version we run 2.2.10 lacks the capability to support this - although there seem to be some conflicts with ampq/kombu and paster here.

  2. Use pyramid_celery - this will require rewriting the entire celery configuration/deployment system we have now.

In both cases we will also need to hack pylons.url threadlocal until it is replaced with pyramid request.route_url.

Actions #7

Updated by Marcin Kuzminski [CTO] over 7 years ago

  • Status changed from Resolved to Feedback
Actions #8

Updated by Daniel D over 7 years ago

  • Status changed from Feedback to Closed

This particular problem has been fixed, a new task has been made here: https://issues.rhodecode.com/issues/4163 for planning the refactor of celery.

Actions #9

Updated by Daniel D over 7 years ago

  • Parent task set to #4163
Actions

Also available in: Atom PDF