RhodeCode CE/EE

The root cause of the problem is here:
https://internal-code.rhodecode.com/rhodecode-enterprise-ce/files/a92da8f244901a192e22530d3a8190a09dd4fef8/rhodecode/lib/middleware/simplevcs.py#L410

Some interesting points:

1. the recycle check only happens when checking out a connection from the pool
2. when no connections are checked out, meta.Session() will check out a new connection from the pool, also doing the recycle check, otherwise it will reuse a connection that exists - but it won't do a recycle check.
3. meta.Session.remove() is needed to close the connection and check it back in to the pool

The problem is although we have meta.Session.remove in the pylons tween: https://internal-code.rhodecode.com/rhodecode-enterprise-ce/files/a92da8f244901a192e22530d3a8190a09dd4fef8/rhodecode/tweens.py#L76

we are calling meta.Session() again, in the simplevcs finally clause of the response iterator...this 'bypasses' the original meta.Session.remove() finally clause and reopens the connection, leaving it open until the next request comes in.

On the next request that comes in, there will be no 'checkout' since the connection is open already....so if the time between last request (that was a push operation) and the next request is over mysql's wait_timeout seconds, then there will be no recycle, and we crash

nice write-up, actually maybe we could move the cache invalidation to other place ?

pullrequest created, https://internal-code.rhodecode.com/rhodecode-enterprise-ce/pull-request/2563, (status: under_review)

The problem is that this bug can happen easily anywhere - anything that creates a calls meta.Session() will check out a connection.

So we should wrap anything that would possibly use the database, and this is not limited to requests, the make_pyramid_app() function for example will check out a connection and leave it hanging, and also there seemed to be a subrequest issued when a http hooks daemon hook was being called.

pullrequest created, https://internal-code.rhodecode.com/rhodecode-enterprise-ce/pull-request/2569, (status: under_review)

pullrequest merged, https://internal-code.rhodecode.com/rhodecode-enterprise-ce/pull-request/2569, (status: approved)