Bug #5679
Data directory continues to grow until it fills disk partition
Start date:
28.03.2022
Due date:
% Done:
0%
Estimated time:
Sorting:
Commit Number:
Affected Version:
Description
Our data/cache directory will continue to grow until it fills the partition and the web interface of RC becomes inaccessible. We have been increasing the partition size (now up to 20G) and it still is filling/failing.
After removing clearing the cache entries from the data directory, was able to pull this exception from the Exception Tracker
Exception `OSError` generated on UTC date: 2022-03-28T13:15:03.975409 Request: POST https://127.0.0.1:5000/rc4/_admin/api 127.0.0.1 RhodeCode-CLI/1.4.0 Traceback (most recent call last): File "/opt/rhodecode/store/da2il1f83gci4333j0sd256nji9nlhpa-python2.7-pyramid-1.10.4/lib/python2.7/site-packages/pyramid/router.py", line 277, in default_execution_policy return router.invoke_request(request) File "/opt/rhodecode/store/da2il1f83gci4333j0sd256nji9nlhpa-python2.7-pyramid-1.10.4/lib/python2.7/site-packages/pyramid/router.py", line 252, in invoke_request request._process_response_callbacks(response) File "/opt/rhodecode/store/da2il1f83gci4333j0sd256nji9nlhpa-python2.7-pyramid-1.10.4/lib/python2.7/site-packages/pyramid/request.py", line 83, in _process_response_callbacks callback(self, response) File "/opt/rhodecode/store/7a23dpzkkqnqmai1kx8wfnflmngr7q9x-python2.7-rhodecode-enterprise-ce-4.26.0/lib/python2.7/site-packages/rhodecode/lib/rc_beaker.py", line 40, in session_callback self.persist() File "/opt/rhodecode/store/bq0n2vs76d5c5yxwjab17a7pfgsrfr9x-python2.7-beaker-1.9.1/lib/python2.7/site-packages/beaker/session.py", line 823, in persist self._session().save() File "/opt/rhodecode/store/bq0n2vs76d5c5yxwjab17a7pfgsrfr9x-python2.7-beaker-1.9.1/lib/python2.7/site-packages/beaker/session.py", line 463, in save self.namespace.acquire_write_lock(replace=True) File "/opt/rhodecode/store/bq0n2vs76d5c5yxwjab17a7pfgsrfr9x-python2.7-beaker-1.9.1/lib/python2.7/site-packages/beaker/container.py", line 220, in acquire_write_lock r = self.access_lock.acquire_write_lock(wait) File "/opt/rhodecode/store/bq0n2vs76d5c5yxwjab17a7pfgsrfr9x-python2.7-beaker-1.9.1/lib/python2.7/site-packages/beaker/synchronization.py", line 186, in acquire_write_lock x = self.do_acquire_write_lock(wait) File "/opt/rhodecode/store/bq0n2vs76d5c5yxwjab17a7pfgsrfr9x-python2.7-beaker-1.9.1/lib/python2.7/site-packages/beaker/synchronization.py", line 261, in do_acquire_write_lock filedescriptor = self._open(os.O_CREAT | os.O_WRONLY) File "/opt/rhodecode/store/bq0n2vs76d5c5yxwjab17a7pfgsrfr9x-python2.7-beaker-1.9.1/lib/python2.7/site-packages/beaker/synchronization.py", line 242, in _open filedescriptor = os.open(self.filename, mode) OSError: [Errno 28] No space left on device: '/home/rhodecode/.rccontrol/community-1/data/sessions/lock/5/5a/5a4e4f9d514d72d417d9cff05380f5b798e93b33.lock
$ rccontrol status RHODECODE CONTROL VERSION: 1.24.3 - NAME: community-1 - STATUS: RUNNING logs: /home/rhodecode/.rccontrol/community-1/community.log - VERSION: 4.26.0 Community - VCS: vcsserver-1 - URL: http://127.0.0.1:5000 - CONFIG: /home/rhodecode/.rccontrol/community-1/rhodecode.ini - NAME: vcsserver-1 - STATUS: RUNNING logs: /home/rhodecode/.rccontrol/vcsserver-1/vcsserver.log - VERSION: 4.26.0 VCSServer - URL: http://127.0.0.1:10010 - CONFIG: /home/rhodecode/.rccontrol/vcsserver-1/vcsserver.ini
Updated by lee loucks 3 months ago
Appears that our cache settings are "optimal" according to https://docs.rhodecode.com/RhodeCode-Enterprise/admin/system_admin/vcs-server.html#vcs-server-maintain
Updated by Marcin Kuzminski [CTO] 2 months ago
Hi,
Please note that this is limitation of the file based sessions, they don't get automatically cleaned.
- EE edition has an scheduler that can clean those
- you can switch sessions to Redis or DB backend so they don't pollute the disk space.