Project

General

Profile

Actions

Bug #5434

open

Locale problem

Added by Nikolay Yankin about 6 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Immediate
Assignee:
-
Category:
-
Target version:
Start date:
02.02.2018
Due date:
% Done:

0%

Estimated time:
Sorting:
Commit Number:
Affected Version:

Description

Same problem as described here https://issues.rhodecode.com/issues/5297 on Gentoo x64

Problem was appeared after upgrading from 4.9.1 to 4.11.1 (4.10.6 also affected)


Files

log.txt (6.43 KB) log.txt Nikolay Yankin, 02.02.2018 14:53
Actions #1

Updated by Nikolay Yankin about 6 years ago

I tried locales en_US.UTF-8 and ru_RU.UTF-8, both correctly installed in my system.

Actions #2

Updated by Marcin Kuzminski [CTO] about 6 years ago

have you tried to set LC_ALL ? to specific locale on your machine

What's the output of locale

Actions #3

Updated by Marcin Kuzminski [CTO] about 6 years ago

there were 0 changes on this code since 4.9, so i expect env changes that caused this... does install 4.9.0 alongside not have this problem?

Actions #4

Updated by Nikolay Yankin about 6 years ago

repo ~ # locale
LANG=en_US.utf8
LC_CTYPE="en_US.utf8"
LC_NUMERIC="en_US.utf8"
LC_TIME="en_US.utf8"
LC_COLLATE="en_US.utf8"
LC_MONETARY="en_US.utf8"
LC_MESSAGES="en_US.utf8"
LC_PAPER="en_US.utf8"
LC_NAME="en_US.utf8"
LC_ADDRESS="en_US.utf8"
LC_TELEPHONE="en_US.utf8"
LC_MEASUREMENT="en_US.utf8"
LC_IDENTIFICATION="en_US.utf8"
LC_ALL=

Actions #5

Updated by Nikolay Yankin about 6 years ago

Simple script

import locale
loc_list = [(a,b) for a,b in locale.locale_alias.items() ]
loc_size = len(loc_list)
print loc_size,'entries'

for loc in loc_list:
    try:
        locale.setlocale(locale.LC_TIME, loc[1])
        print 'SUCCES set {:12} ({})'.format(loc[1],loc[0])
    except:
        pass

ran by python test.py shows

970 entries
SUCCES set C            (c.ascii)
SUCCES set C            (c.en)
SUCCES set C            (c)
SUCCES set C            (posix-utf2)
SUCCES set C            (c_c)
SUCCES set C            (c_c.c)
SUCCES set en_US.UTF-8  (universal.utf8@ucs4)
SUCCES set C            (posix)
SUCCES set C            (english_united-states.437)
SUCCES set ru_RU.UTF-8  (ru_ru)
SUCCES set en_US.UTF-8  (c.utf8)
SUCCES set ru_RU.UTF-8  (ru)

but throught ishell shows only

970 entries
SUCCES set C            (posix-utf2)
SUCCES set C            (english_united-states.437)
SUCCES set C            (c_c.c)
SUCCES set C            (posix)
SUCCES set C            (c.en)
SUCCES set C            (c_c)
SUCCES set C            (c.ascii)
SUCCES set C            (c)
Actions #6

Updated by Marcin Kuzminski [CTO] about 6 years ago

can you check

~/.rccontrol/supervisord.ini ?

It can have this:

[supervisord]
minfds = 1024
minprocs = 200
loglevel = info
environment = HOME=/home/ubuntu,LANG=en_US.UTF-8,LC_ALL=en_US.UTF-8
strip_ansi = true
logfile = /home/ubuntu/.rccontrol/supervisor/supervisord.log
pidfile = /home/ubuntu/.rccontrol/supervisor/supervisord.pid
Actions #7

Updated by Nikolay Yankin about 6 years ago

[supervisord]
minfds = 1024
minprocs = 200
loglevel = info
environment = HOME=/root,LANG=en_US.UTF-8
strip_ansi = true
logfile = /root/.rccontrol/supervisor/supervisord.log
pidfile = /root/.rccontrol/supervisor/supervisord.pid

Actions #8

Updated by Marcin Kuzminski [CTO] about 6 years ago

can you add LC_ALL there, and run rccontrol self-stop && rccontrol self-init

Actions #9

Updated by Nikolay Yankin about 6 years ago

Nothing changed, but i tested 4.7.0, 4.8.0, 4.9.1 and problem appear on all these versions.

So, problem not only in RhodeCode, probably it appeared after upgrade glibc from 2.25 to 2.26

Actions #10

Updated by Marcin Kuzminski [CTO] about 6 years ago

might be, we got odd locale issues on multiple of our own server after some recent package updates...

But i'm not sure how to resolve this...

Actions #11

Updated by Nikolay Yankin about 6 years ago

Try to update python to 2.7.14 and glibc to 2.25 in you bundles. Do you have night or test builds? Or how i can update this packages manually?

Actions #12

Updated by Marcin Kuzminski [CTO] about 6 years ago

WE use a nix package manager, and it has pinned older python/glibc

  • we're on upgrading those but it's probably bigger task for next release

This ticket is interesting: https://github.com/NixOS/nix/issues/599

Maybe an option is to set LOCAL_ARCHIVES ?

But anyway it's odd that setting env LC_ALL via supervisor doesn't resolve that problem...

Actions #13

Updated by Nikolay Yankin about 6 years ago

Setting LOCALE_ARCHIVE=/usr/lib64/locale/locale-archive in .rccontrol/supervisor/supervisord.ini fixes the problem

Actions #14

Updated by Marcin Kuzminski [CTO] about 6 years ago

  • Status changed from New to Resolved
Actions #15

Updated by Sefter Aras almost 6 years ago

i have the same problem on openSuse 42.3, the LOCALE_ARCHIVE=/usr/lib64/locale/locale-archive doesn´t exists

my supervisord.ini:

[supervisord]
minfds = 1024
minprocs = 200
loglevel = info
environment = LOCALE_ARCHIVE=/usr/lib/locale/locale-archive,HOME=/root,LANG=en_US.UTF-8,LC_ALL=en_US.UTF-8
strip_ansi = true

it´s not working for me :-(

do you have any other ideas?

here my log:

Error: unsupported locale setting
[2018-04-12 15:31:08 +0000] [30837] [INFO] Worker exiting (pid: 30837)
Traceback (most recent call last):
File "/opt/rhodecode/store/9k60d775bg7fvbpvvbmxd9hz1lr2558j-python2.7-gunicorn-19.7.1/bin/.gunicorn-wrapped", line 12, in
sys.exit(run())
File "/opt/rhodecode/store/9k60d775bg7fvbpvvbmxd9hz1lr2558j-python2.7-gunicorn-19.7.1/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 74, in run
WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
File "/opt/rhodecode/store/9k60d775bg7fvbpvvbmxd9hz1lr2558j-python2.7-gunicorn-19.7.1/lib/python2.7/site-packages/gunicorn/app/base.py", line 203, in run
super(Application, self).run()
File "/opt/rhodecode/store/9k60d775bg7fvbpvvbmxd9hz1lr2558j-python2.7-gunicorn-19.7.1/lib/python2.7/site-packages/gunicorn/app/base.py", line 72, in run
Arbiter(self).run()
File "/opt/rhodecode/store/9k60d775bg7fvbpvvbmxd9hz1lr2558j-python2.7-gunicorn-19.7.1/lib/python2.7/site-packages/gunicorn/arbiter.py", line 231, in run
self.halt(reason=inst.reason, exit_status=inst.exit_status)
File "/opt/rhodecode/store/9k60d775bg7fvbpvvbmxd9hz1lr2558j-python2.7-gunicorn-19.7.1/lib/python2.7/site-packages/gunicorn/arbiter.py", line 344, in halt
self.stop()
File "/opt/rhodecode/store/9k60d775bg7fvbpvvbmxd9hz1lr2558j-python2.7-gunicorn-19.7.1/lib/python2.7/site-packages/gunicorn/arbiter.py", line 393, in stop
time.sleep(0.1)
File "/opt/rhodecode/store/9k60d775bg7fvbpvvbmxd9hz1lr2558j-python2.7-gunicorn-19.7.1/lib/python2.7/site-packages/gunicorn/arbiter.py", line 244, in handle_chld
self.reap_workers()
File "/opt/rhodecode/store/9k60d775bg7fvbpvvbmxd9hz1lr2558j-python2.7-gunicorn-19.7.1/lib/python2.7/site-packages/gunicorn/arbiter.py", line 524, in reap_workers
raise HaltServer(reason, self.WORKER_BOOT_ERROR)
gunicorn.errors.HaltServer:
/opt/rhodecode/store/bc656fqasmx1jmkglarvapy244bbh7dg-bash-4.3-p42/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
/opt/rhodecode/store/bc656fqasmx1jmkglarvapy244bbh7dg-bash-4.3-p42/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)

Actions #16

Updated by Marcin Kuzminski [CTO] almost 6 years ago

Hi,

Have you rebooted rhodecode after those changes ?

Maybe it's related to supervisord not loading those settings

Actions #17

Updated by Sefter Aras almost 6 years ago

yes

rccontrol self-stop && rccontrol self-init

it´s correct so? or did you mean server reboot?

Actions #18

Updated by Marcin Kuzminski [CTO] almost 6 years ago

Hmm, not sure then, maybe it's related to different Bash profile.

what's the output of locale when you run it via the profile rhodecode is running?

Actions #19

Updated by Sefter Aras almost 6 years ago

LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=

Actions #20

Updated by Marcin Kuzminski [CTO] almost 6 years ago

yes so problem is that LC_ALL isn't set.

Can you add LC_ALL in your profile maybe?

Actions #21

Updated by Sefter Aras almost 6 years ago

de_DE.UTF-8 or en_US.UTF-8 ?

Actions #22

Updated by Sefter Aras almost 6 years ago

if i set LC_ALL then i got this error on rccontrol start vcsserver-1

/opt/rhodecode/store/bc656fqasmx1jmkglarvapy244bbh7dg-bash-4.3-p42/bin/bash: warning: setlocale: LC_ALL: cannot change locale (de_DE.UTF-8)
/opt/rhodecode/store/bc656fqasmx1jmkglarvapy244bbh7dg-bash-4.3-p42/bin/bash: warning: setlocale: LC_ALL: cannot change locale (de_DE.UTF-8)

and i have the same error log entry above

Actions #23

Updated by Nikolay Yankin almost 6 years ago

Sefter Aras, In your system LOCALE_ARCHIVE need to be set to another path. Maybe to /usr/lib/locale/locale-archive
Try to run 'locate locale-archive'

Actions #24

Updated by Sefter Aras almost 6 years ago

locate locale-archive
If 'locate' is not a typo you can use command-not-found to lookup the package that contains it, like this:
cnf locate

doesnt work

Actions #25

Updated by Sefter Aras almost 6 years ago

OK, i have installed the locate util for opensuse

locate locale-archive
/usr/lib/locale/locale-archive

Actions #26

Updated by Sefter Aras almost 6 years ago

But this path i have already set, this change nothing

Actions #27

Updated by Sefter Aras almost 6 years ago

i feel that my settings doesn´t work:

my supervisord.ini:

[supervisord]
minfds = 1024
minprocs = 200
loglevel = info
environment = LOCALE_ARCHIVE=/usr/lib/locale/locale-archive,HOME=/root,LANG=en_US.UTF-8,LC_ALL=en_US.UTF-8
strip_ansi = true

Actions #28

Updated by Marcin Kuzminski [CTO] almost 6 years ago

after changes in supervisord.ini did you run rccontrol self-stop && rccontrol self-init ?

Actions #29

Updated by Marcin Kuzminski [CTO] over 5 years ago

We've identified the problem for now. It looks like since glibc 2.27 the generated locale broke backward compatibility. We've yet not found a solution to this problem, the only thing right now is to potentially either re-generate the locale-archives using older glibc or downgrade glibc to pre 2.27

Actions #30

Updated by Systems Administration over 5 years ago

I was using 4.12.4 and the problem appeared after upgrading from Ubuntu Xenial (16.04) to Ubuntu Bionic (18.04).

Actions #31

Updated by Marcin Kuzminski [CTO] over 5 years ago

Here's a workaround:

Fedora 23 / Ubuntu 18.04
------------------------

|RCC| has a know problem with locales, due to changes in glibc 2.27+ which affects
the local-archive format, which is now incompatible with our used glibc 2.26.


To work around this problem, you need set path to ``$LOCAL_ARCHIVE`` to the
locale package in older pre glibc 2.27 format, or set `LC_ALL=C` in your enviroment.

To use the pre 2.27 locale-archive fix follow these steps:

1. Download the pre 2.27 locale-archive package

.. code-block:: bash

    wget https://dls.rhodecode.com/assets/locale-archive


2. Point ``$LOCAL_ARCHIVE`` to the locale package.

.. code-block:: bash

    $ export LOCALE_ARCHIVE=/home/USER/locale-archive  # change to your path

This can either added in `~/.rccontrol/supervisor/supervisord.ini`
or in user .bashrc/.zshrc etc, or via a startup script that
runs `rccontrol self-init`
Actions #32

Updated by Oleg Schelykalnov over 2 years ago

It helps to fix python setlocale error but mercurial commit messages still shows as ?????

Actions

Also available in: Atom PDF