Actions
Bug #5679
openData directory continues to grow until it fills disk partition
Start date:
28.03.2022
Due date:
% Done:
0%
Estimated time:
Sorting:
Commit Number:
Affected Version:
Description
Our data/cache directory will continue to grow until it fills the partition and the web interface of RC becomes inaccessible. We have been increasing the partition size (now up to 20G) and it still is filling/failing.
After removing clearing the cache entries from the data directory, was able to pull this exception from the Exception Tracker
Exception `OSError` generated on UTC date: 2022-03-28T13:15:03.975409
Request: POST https://127.0.0.1:5000/rc4/_admin/api
127.0.0.1 RhodeCode-CLI/1.4.0
Traceback (most recent call last):
File "/opt/rhodecode/store/da2il1f83gci4333j0sd256nji9nlhpa-python2.7-pyramid-1.10.4/lib/python2.7/site-packages/pyramid/router.py", line 277, in default_execution_policy
return router.invoke_request(request)
File "/opt/rhodecode/store/da2il1f83gci4333j0sd256nji9nlhpa-python2.7-pyramid-1.10.4/lib/python2.7/site-packages/pyramid/router.py", line 252, in invoke_request
request._process_response_callbacks(response)
File "/opt/rhodecode/store/da2il1f83gci4333j0sd256nji9nlhpa-python2.7-pyramid-1.10.4/lib/python2.7/site-packages/pyramid/request.py", line 83, in _process_response_callbacks
callback(self, response)
File "/opt/rhodecode/store/7a23dpzkkqnqmai1kx8wfnflmngr7q9x-python2.7-rhodecode-enterprise-ce-4.26.0/lib/python2.7/site-packages/rhodecode/lib/rc_beaker.py", line 40, in session_callback
self.persist()
File "/opt/rhodecode/store/bq0n2vs76d5c5yxwjab17a7pfgsrfr9x-python2.7-beaker-1.9.1/lib/python2.7/site-packages/beaker/session.py", line 823, in persist
self._session().save()
File "/opt/rhodecode/store/bq0n2vs76d5c5yxwjab17a7pfgsrfr9x-python2.7-beaker-1.9.1/lib/python2.7/site-packages/beaker/session.py", line 463, in save
self.namespace.acquire_write_lock(replace=True)
File "/opt/rhodecode/store/bq0n2vs76d5c5yxwjab17a7pfgsrfr9x-python2.7-beaker-1.9.1/lib/python2.7/site-packages/beaker/container.py", line 220, in acquire_write_lock
r = self.access_lock.acquire_write_lock(wait)
File "/opt/rhodecode/store/bq0n2vs76d5c5yxwjab17a7pfgsrfr9x-python2.7-beaker-1.9.1/lib/python2.7/site-packages/beaker/synchronization.py", line 186, in acquire_write_lock
x = self.do_acquire_write_lock(wait)
File "/opt/rhodecode/store/bq0n2vs76d5c5yxwjab17a7pfgsrfr9x-python2.7-beaker-1.9.1/lib/python2.7/site-packages/beaker/synchronization.py", line 261, in do_acquire_write_lock
filedescriptor = self._open(os.O_CREAT | os.O_WRONLY)
File "/opt/rhodecode/store/bq0n2vs76d5c5yxwjab17a7pfgsrfr9x-python2.7-beaker-1.9.1/lib/python2.7/site-packages/beaker/synchronization.py", line 242, in _open
filedescriptor = os.open(self.filename, mode)
OSError: [Errno 28] No space left on device: '/home/rhodecode/.rccontrol/community-1/data/sessions/lock/5/5a/5a4e4f9d514d72d417d9cff05380f5b798e93b33.lock
$ rccontrol status
RHODECODE CONTROL VERSION: 1.24.3
- NAME: community-1
- STATUS: RUNNING
logs: /home/rhodecode/.rccontrol/community-1/community.log
- VERSION: 4.26.0 Community
- VCS: vcsserver-1
- URL: http://127.0.0.1:5000
- CONFIG: /home/rhodecode/.rccontrol/community-1/rhodecode.ini
- NAME: vcsserver-1
- STATUS: RUNNING
logs: /home/rhodecode/.rccontrol/vcsserver-1/vcsserver.log
- VERSION: 4.26.0 VCSServer
- URL: http://127.0.0.1:10010
- CONFIG: /home/rhodecode/.rccontrol/vcsserver-1/vcsserver.ini
Updated by lee loucks over 2 years ago
Appears that our cache settings are "optimal" according to https://docs.rhodecode.com/RhodeCode-Enterprise/admin/system_admin/vcs-server.html#vcs-server-maintain
Updated by Marcin Kuzminski [CTO] over 2 years ago
Hi,
Please note that this is limitation of the file based sessions, they don't get automatically cleaned.
- EE edition has an scheduler that can clean those
- you can switch sessions to Redis or DB backend so they don't pollute the disk space.
Actions