Bug #4035
open
failed to create whoosh index
0%
Description
I have rhodecode 4.1.2 installation. Running rhodecode-index on one of my repos produces following error traceback:
.rccontrol/enterprise-1/profile/bin/rhodecode-index --instance-name=enterprise-1 --force
Using <WhooshIndexBuilder(/root/.rccontrol/enterprise-1/data/index)> for building full text search
Indexing data types ['commits', 'files']
FETCH [unitry] repository configuration
FETCH [unitry] repository configuration, done [0.278s]
FETCH [unitry] commit info (1000 commits per call. Server reported 406 total commits, parse limit is -1)
Fetched 1000 commits [406 total][fetch:0001]
FETCH [unitry] commit info, done [1.745s]
FETCH [unitry] file tree info (@commit_id: 9a2e04c37b36637bcee44b47af8014eed494ae20)
FETCH [unitry] file tree info, done [2.425s]
PROCESSED [unitry] 1/3 repo process limit:-1
FETCH [design] repository configuration
FETCH [design] repository configuration, done [0.281s]
FETCH [design] commit info (1000 commits per call. Server reported 40 total commits, parse limit is -1)
Fetched 1000 commits [40 total][fetch:0001]
FETCH [design] commit info, done [0.318s]
FETCH [design] file tree info (@commit_id: c0f1dccf85e87585241aceaee0ce765ffbd5ef8b)
FETCH [design] file tree info, done [1.113s]
PROCESSED [design] 2/3 repo process limit:-1
FETCH [theram] repository configuration
FETCH [theram] repository configuration, done [0.369s]
FETCH [theram] commit info (1000 commits per call. Server reported 745 total commits, parse limit is -1)
Fetched 1000 commits [745 total][fetch:0001]
FETCH [theram] commit info, done [2.995s]
Traceback (most recent call last):
File "/opt/rhodecode/store/09rj6xan32awqyc73chc7ih3w9cqidp3-python2.7-rhodecode-tools-0.8.3/bin/.rhodecode-index-wrapped", line 12, in
sys.exit(cmd())
File "/opt/rhodecode/store/g0k5g59cxm7imw5lcqwj0zwp7as4qlna-python2.7-click-5.1/lib/python2.7/site-packages/click/core.py", line 700, in call
return self.main(*args, **kwargs)
File "/opt/rhodecode/store/g0k5g59cxm7imw5lcqwj0zwp7as4qlna-python2.7-click-5.1/lib/python2.7/site-packages/click/core.py", line 680, in main
rv = self.invoke(ctx)
File "/opt/rhodecode/store/g0k5g59cxm7imw5lcqwj0zwp7as4qlna-python2.7-click-5.1/lib/python2.7/site-packages/click/core.py", line 873, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/rhodecode/store/g0k5g59cxm7imw5lcqwj0zwp7as4qlna-python2.7-click-5.1/lib/python2.7/site-packages/click/core.py", line 508, in invoke
return callback(*args, **kwargs)
File "/opt/rhodecode/store/g0k5g59cxm7imw5lcqwj0zwp7as4qlna-python2.7-click-5.1/lib/python2.7/site-packages/click/decorators.py", line 63, in new_func
return ctx.invoke(f, obj, *args[1:], **kwargs)
File "/opt/rhodecode/store/g0k5g59cxm7imw5lcqwj0zwp7as4qlna-python2.7-click-5.1/lib/python2.7/site-packages/click/core.py", line 508, in invoke
return callback(*args, **kwargs)
File "/opt/rhodecode/store/09rj6xan32awqyc73chc7ih3w9cqidp3-python2.7-rhodecode-tools-0.8.3/lib/python2.7/site-packages/rhodecode_tools/commands/index.py", line 146, in cmd
commit_fetch_limit, commit_process_limit, repo_limit)
File "/opt/rhodecode/store/09rj6xan32awqyc73chc7ih3w9cqidp3-python2.7-rhodecode-tools-0.8.3/lib/python2.7/site-packages/rhodecode_tools/commands/index.py", line 388, in main
cli.run_with_lock(_build_index, ctx, 'index_build')
File "/opt/rhodecode/store/09rj6xan32awqyc73chc7ih3w9cqidp3-python2.7-rhodecode-tools-0.8.3/lib/python2.7/site-packages/rhodecode_tools/lib/cli.py", line 362, in run_with_lock
func(ctx)
File "/opt/rhodecode/store/09rj6xan32awqyc73chc7ih3w9cqidp3-python2.7-rhodecode-tools-0.8.3/lib/python2.7/site-packages/rhodecode_tools/commands/index.py", line 378, in _build_index
func()
File "/opt/rhodecode/store/09rj6xan32awqyc73chc7ih3w9cqidp3-python2.7-rhodecode-tools-0.8.3/lib/python2.7/site-packages/rhodecode_tools/commands/index.py", line 362, in __commits
indexer.build_commit_index(_repo_name, repo_data=api_data)
File "/opt/rhodecode/store/09rj6xan32awqyc73chc7ih3w9cqidp3-python2.7-rhodecode-tools-0.8.3/lib/python2.7/site-packages/rhodecode_tools/lib/fts_index/base.py", line 306, in build_commit_index
return self._build_commit_index(repo_name, repo_data)
File "/opt/rhodecode/store/09rj6xan32awqyc73chc7ih3w9cqidp3-python2.7-rhodecode-tools-0.8.3/lib/python2.7/site-packages/rhodecode_tools/lib/fts_index/whoosh_engine.py", line 230, in _build_commit_index
commit_idx_writer=commit_index_writer)
File "/opt/rhodecode/store/09rj6xan32awqyc73chc7ih3w9cqidp3-python2.7-rhodecode-tools-0.8.3/lib/python2.7/site-packages/rhodecode_tools/lib/fts_index/whoosh_engine.py", line 239, in index_commits
commit_idx_backend, repo_name, repo_data, start_rev,)
File "/opt/rhodecode/store/09rj6xan32awqyc73chc7ih3w9cqidp3-python2.7-rhodecode-tools-0.8.3/lib/python2.7/site-packages/rhodecode_tools/lib/fts_index/base.py", line 351, in index_commits
commit_idx_backend, repo_name, repo_data, _cs)
File "/opt/rhodecode/store/09rj6xan32awqyc73chc7ih3w9cqidp3-python2.7-rhodecode-tools-0.8.3/lib/python2.7/site-packages/rhodecode_tools/lib/fts_index/base.py", line 279, in add_commit_doc
parents=[cs['raw_id'] for cs in commit_data['parents']],
File "/opt/rhodecode/store/09rj6xan32awqyc73chc7ih3w9cqidp3-python2.7-rhodecode-tools-0.8.3/lib/python2.7/site-packages/rhodecode_tools/lib/fts_index/whoosh_engine.py", line 106, in add_doc
parents=u' '.join((commit_id for commit_id in parents)),
File "/opt/rhodecode/store/rj9y8mfdjmg59qdwr8wmn2j8x0by433p-python2.7-Whoosh-2.7.0/lib/python2.7/site-packages/whoosh/writing.py", line 750, in add_document
for tbytes, freq, weight, vbytes in items:
File "/opt/rhodecode/store/rj9y8mfdjmg59qdwr8wmn2j8x0by433p-python2.7-Whoosh-2.7.0/lib/python2.7/site-packages/whoosh/fields.py", line 668, in index
yield (self.to_bytes(num, shift), 1, 1.0, emptybytes)
File "/opt/rhodecode/store/rj9y8mfdjmg59qdwr8wmn2j8x0by433p-python2.7-Whoosh-2.7.0/lib/python2.7/site-packages/whoosh/fields.py", line 721, in to_bytes
x = self.prepare_number(x)
File "/opt/rhodecode/store/rj9y8mfdjmg59qdwr8wmn2j8x0by433p-python2.7-Whoosh-2.7.0/lib/python2.7/site-packages/whoosh/fields.py", line 691, in prepare_number
% (x, self.min_value, self.max_value))
ValueError: Numeric field value 9151470258 out of range [-2147483648, 2147483647]
Updated by Marcin Kuzminski [CTO] over 8 years ago
- Assignee set to Marcin Kuzminski [CTO]
- Target version set to v4.2
Updated by Marcin Kuzminski [CTO] over 8 years ago
Thanks for this report. Can you tell us a little bit more on the problematic repository ? Type/ how many commits/files etc.
Updated by Roman Valov over 8 years ago
Marcin Kuzminski [staff] wrote:
Thanks for this report. Can you tell us a little bit more on the problematic repository ? Type/ how many commits/files etc.
Hello, Marcin. Repository is mercurial based, at the moment there is about 750 commits and 150 files.
I can put some debug print statements to whoosh and rhodecode tools source to produce some useful information on the issue,
just tell me what information would you like to get or give me a patch with debug print statements.
Updated by Marcin Kuzminski [CTO] over 8 years ago
Hi,
It would be usefull to get the full set of params sent in here: File "/opt/rhodecode/store/09rj6xan32awqyc73chc7ih3w9cqidp3-python2.7-rhodecode-tools-0.8.3/lib/python2.7/site-packages/rhodecode_tools/lib/fts_index/whoosh_engine.py", line 106, in add_doc
There are 3 numeric fields:
repository_id=NUMERIC(unique=True, stored=True),
commit_idx=NUMERIC(stored=True, sortable=True),
date=NUMERIC(stored=True, sortable=True),
I wonder how it can happen that any of them is 9151470258
Best,
Updated by Roman Valov over 8 years ago
Well, with your hints I found that it was commit done during development of support datetime limits in our project and due to inaccuracy commit date is "Sun Jan 01 01:04:18 2260 +0300" with resulting date value 9151470258. I've patched mercurial a little bit to fix dates in my repo using convert extension. So, issue is not actual for me.
Updated by Marcin Kuzminski [CTO] over 8 years ago
- Status changed from New to In Progress