Bug #4285
Intermittent error while trying to create or fork a repository
0%
Description
We're having trouble sometimes when trying to create or fork repositories. The reports I've heard are that it either never finishes or gets a 404 error. It seems like it does work at least some of the time, though.
We're running Community Edition 4.4 using Celery, RabbitMQ, and PostgreSQL.
I checked our Celery log file and we're seeing errors like this when we try to create/fork (this one is a fork, but the details are basically the same for create except for the task name):
[2016-10-17 11:19:46,408: ERROR/MainProcess] Task rhodecode.lib.celerylib.tasks.create_repo_fork[d71a3d33-d803-43cf-aa6d-0a6d48c5e935] raised exception: SQLAlchemyError('(DatabaseError) SSL error: decryption failed or bad record mac\n',) Traceback (most recent call last): File "/opt/rhodecode/store/32ay02r6m59qvxk7gvybpfsqmmhd810v-python2.7-celery-2.2.10/lib/python2.7/site-packages/celery/execute/trace.py", line 34, in trace return cls(states.SUCCESS, retval=fun(*args, **kwargs)) File "/opt/rhodecode/store/2nrqs5ylil69bsjb09j699ydahcxvvfn-python2.7-rhodecode-enterprise-ce-4.4.0/lib/python2.7/site-packages/rhodecode/lib/celerylib/__init__.py", line 115, in __call__ ip_addr=proxy_data['auth_user']['ip_addr']) File "/opt/rhodecode/store/2nrqs5ylil69bsjb09j699ydahcxvvfn-python2.7-rhodecode-enterprise-ce-4.4.0/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 796, in __init__ self.propagate_data() File "/opt/rhodecode/store/2nrqs5ylil69bsjb09j699ydahcxvvfn-python2.7-rhodecode-enterprise-ce-4.4.0/lib/python2.7/site-packages/rhodecode/lib/auth.py", line 851, in propagate_data anon_user = self.anonymous_user = User.get_default_user(cache=True) File "/opt/rhodecode/store/2nrqs5ylil69bsjb09j699ydahcxvvfn-python2.7-rhodecode-enterprise-ce-4.4.0/lib/python2.7/site-packages/rhodecode/model/db.py", line 810, in get_default_user user = User.get_by_username(User.DEFAULT_USER, cache=cache) File "/opt/rhodecode/store/2nrqs5ylil69bsjb09j699ydahcxvvfn-python2.7-rhodecode-enterprise-ce-4.4.0/lib/python2.7/site-packages/rhodecode/model/db.py", line 691, in get_by_username return q.scalar() File "/opt/rhodecode/store/rvbznw1hks4ix1pnlfmpfwh9sqmm287d-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2429, in scalar ret = self.one() File "/opt/rhodecode/store/rvbznw1hks4ix1pnlfmpfwh9sqmm287d-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2398, in one ret = list(self) File "/opt/rhodecode/store/2nrqs5ylil69bsjb09j699ydahcxvvfn-python2.7-rhodecode-enterprise-ce-4.4.0/lib/python2.7/site-packages/rhodecode/lib/caching_query.py", line 102, in __iter__ return self.get_value(createfunc=lambda: File "/opt/rhodecode/store/2nrqs5ylil69bsjb09j699ydahcxvvfn-python2.7-rhodecode-enterprise-ce-4.4.0/lib/python2.7/site-packages/rhodecode/lib/caching_query.py", line 121, in get_value ret = cache.get_value(cache_key, createfunc=createfunc) File "/opt/rhodecode/store/l7bv0fc6pv8fmmgmdbh39hzn4iskq3hj-python2.7-Beaker-1.7.0/lib/python2.7/site-packages/beaker/cache.py", line 317, in get return self._get_value(key, **kw).get_value() File "/opt/rhodecode/store/l7bv0fc6pv8fmmgmdbh39hzn4iskq3hj-python2.7-Beaker-1.7.0/lib/python2.7/site-packages/beaker/container.py", line 378, in get_value v = self.createfunc() File "/opt/rhodecode/store/2nrqs5ylil69bsjb09j699ydahcxvvfn-python2.7-rhodecode-enterprise-ce-4.4.0/lib/python2.7/site-packages/rhodecode/lib/caching_query.py", line 103, in <lambda> list(Query.__iter__(self))) File "/opt/rhodecode/store/rvbznw1hks4ix1pnlfmpfwh9sqmm287d-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2441, in __iter__ return self._execute_and_instances(context) File "/opt/rhodecode/store/rvbznw1hks4ix1pnlfmpfwh9sqmm287d-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2456, in _execute_and_instances result = conn.execute(querycontext.statement, self._params) File "/opt/rhodecode/store/rvbznw1hks4ix1pnlfmpfwh9sqmm287d-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 841, in execute return meth(self, multiparams, params) File "/opt/rhodecode/store/rvbznw1hks4ix1pnlfmpfwh9sqmm287d-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/opt/rhodecode/store/rvbznw1hks4ix1pnlfmpfwh9sqmm287d-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 938, in _execute_clauseelement compiled_sql, distilled_params File "/opt/rhodecode/store/rvbznw1hks4ix1pnlfmpfwh9sqmm287d-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1070, in _execute_context context) File "/opt/rhodecode/store/rvbznw1hks4ix1pnlfmpfwh9sqmm287d-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1271, in _handle_dbapi_exception exc_info File "/opt/rhodecode/store/rvbznw1hks4ix1pnlfmpfwh9sqmm287d-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause reraise(type(exception), exception, tb=exc_tb) File "/opt/rhodecode/store/rvbznw1hks4ix1pnlfmpfwh9sqmm287d-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1063, in _execute_context context) File "/opt/rhodecode/store/rvbznw1hks4ix1pnlfmpfwh9sqmm287d-python2.7-SQLAlchemy-0.9.9/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 442, in do_execute cursor.execute(statement, parameters) DatabaseError: (DatabaseError) SSL error: decryption failed or bad record mac
This seems like it could be related to this issue: https://github.com/celery/celery/issues/1564
Updated by Marcin Kuzminski [CTO] over 5 years ago
Hi,
This looks odd, looks like potential misconfiguration of caching/session library. Btw, could you try to upgrade to 4.4.2 release and see if this fixes the issue ?
Also potentially make sure if celery didn't leave any zombie processes after upgrade of RhodeCode from previous version having older code loaded.