RhodeCode CE/EE

Separating python modules into /opt/rhodecode/store/<hash>-python2.7-<name>/ causes python to search for dependencies that are not in their relevant directory, causing hundreds of thousands of missing file errors.

To debug:
Edit one troubleshooting user in ~rcuser/.ssh/authorized_keys_rhodecode to include command="strace -f -o /tmp/rc-ssh-wrapper.log ~/.rccontrol/..."

$ less /tmp/rc-ssh-wrapper.log
...
34782      0.000131 stat("/opt/rhodecode/store/y2xhi7pkj4cycnaxm2wkw0hhcmfsn846-python2.7-celery-4.1.0/lib/python2.7/site-packages", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
34782      0.000140 stat("/opt/rhodecode/store/y2xhi7pkj4cycnaxm2wkw0hhcmfsn846-python2.7-celery-4.1.0/lib/python2.7/site-packages", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
34782      0.000134 stat("/opt/rhodecode/store/y2xhi7pkj4cycnaxm2wkw0hhcmfsn846-python2.7-celery-4.1.0/lib/python2.7/site-packages/os", 0x7ffd146ac630) = -1 ENOENT (No such file or directory)
34782      0.000133 open("/opt/rhodecode/store/y2xhi7pkj4cycnaxm2wkw0hhcmfsn846-python2.7-celery-4.1.0/lib/python2.7/site-packages/os.so", O_RDONLY) = -1 ENOENT (No such file or directory)
34782      0.000179 open("/opt/rhodecode/store/y2xhi7pkj4cycnaxm2wkw0hhcmfsn846-python2.7-celery-4.1.0/lib/python2.7/site-packages/osmodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
34782      0.000129 open("/opt/rhodecode/store/y2xhi7pkj4cycnaxm2wkw0hhcmfsn846-python2.7-celery-4.1.0/lib/python2.7/site-packages/os.py", O_RDONLY) = -1 ENOENT (No such file or directory)
34782      0.000142 open("/opt/rhodecode/store/y2xhi7pkj4cycnaxm2wkw0hhcmfsn846-python2.7-celery-4.1.0/lib/python2.7/site-packages/os.pyc", O_RDONLY) = -1 ENOENT (No such file or directory)
...

$ grep ENOENT /tmp/rc-ssh-wrapper.log  | wc -l
256522
$ grep ENOENT /tmp/rc-ssh-wrapper.log  | grep -v python | wc -l
965

Adapt the strace to -c:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 44.60    6.077716      675302         9         3 wait4
 33.16    4.519439          22    208836    202905 open
 15.13    2.061998          17    122457     53627 stat
  3.76    0.511863          18     28472        62 lstat

Re-ordering the PYTHONPATH may be an idea.