Bug #5399
open
Issues with Git LFS integration
0%
Description
We have recently enabled the Git LFS feature in RhodeCode, but we are having problems getting it working.
This may be a client issue, but I'm curious if you've ever seen this before. We run RhodeCode behind Nginx.
We are running RhodeCode 4.9.0 and git 1.8.3.1 on RHEL 6.
Here is the output when we attempt to use git-lfs to push a file:
$ GIT_TRACE=1 git lfs push origin master
trace: exec: 'git-lfs' 'push' 'origin' 'master'
trace: run_command: 'git-lfs' 'push' 'origin' 'master'
16:42:38.036384 trace git-lfs: run_command: 'git' version
16:42:38.046224 trace git-lfs: run_command: 'git' config -l
16:42:38.050043 trace git-lfs: tq: running as batched queue, batch size of 100
16:42:38.050093 trace git-lfs: Upload refs [master] to remote origin
16:42:39.054944 trace git-lfs: run_command: git rev-list --objects master --not --remotes=origin --
16:42:39.060222 trace git-lfs: run_command: git cat-file --batch
16:42:39.062265 trace git-lfs: tq: sending batch of size 1
16:42:39.062534 trace git-lfs: api: batch 1 files
16:42:39.113381 trace git-lfs: HTTP: POST https://username@server.example.com/game-of-life.git/info/lfs/objects/batch
16:42:39.846223 trace git-lfs: HTTP: 200
16:42:39.850511 trace git-lfs: HTTP: {"transfer": "basic", "objects": [{"oid": "30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58", "size": 1048576, "authenticated": true, "actions": {"upload": {"header": {"Authorization": "Basic bGFodWdoZXM6", "Transfer-Encoding": "chunked"}, "href": "http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58"}, "verify": {"header": {"Authorization": "Basic bGFodWdoZXM6"}, "href": "http://server.example.com/game
16:42:39.850669 trace git-lfs: HTTP: -of-life.git/info/lfs/verify"}}}]}
16:42:39.850798 trace git-lfs: tq: starting transfer adapter "basic"
Git LFS: (0 of 1 files) 0 B / 1024.00 KB
16:42:39.851141 trace git-lfs: HTTP: PUT http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
16:42:39.857225 trace git-lfs: HTTP: 301
16:42:39.857257 trace git-lfs: api: redirect PUT http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58 to https://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
16:42:39.857267 trace git-lfs: HTTP: PUT https://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
16:42:39.857532 trace git-lfs: tq: retrying object 30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58: LFS: Put https://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58: read /home/username/game-of-life/.git/lfs/objects/30/e1/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58: file already closed
16:42:39.857558 trace git-lfs: tq: enqueue retry #1 for "30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58" (size: 1048576)
16:42:39.857587 trace git-lfs: tq: sending batch of size 1
16:42:39.857659 trace git-lfs: api: batch 1 files
16:42:39.857746 trace git-lfs: HTTP: POST https://username@server.example.com/game-of-life.git/info/lfs/objects/batch
16:42:40.487590 trace git-lfs: HTTP: 200
16:42:40.490088 trace git-lfs: HTTP: {"transfer": "basic", "objects": [{"oid": "30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58", "size": 1048576, "authenticated": true, "actions": {"upload": {"header": {"Authorization": "Basic bGFodWdoZXM6", "Transfer-Encoding": "chunked"}, "href": "http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58"}, "verify": {"header": {"Authorization": "Basic bGFodWdoZXM6"}, "href": "http://server.example.com/game
16:42:40.490339 trace git-lfs: HTTP: -of-life.git/info/lfs/verify"}}}]}
16:42:40.490679 trace git-lfs: HTTP: PUT http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
Git LFS: (0 of 1 files) 96.00 KB / 1024.00 KB
16:42:40.496941 trace git-lfs: HTTP: 301
16:42:40.496977 trace git-lfs: api: redirect PUT http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58 to https://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
16:42:40.497007 trace git-lfs: HTTP: PUT https://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
16:42:40.497149 trace git-lfs: tq: retrying object 30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58: LFS: Put https://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58: read /home/username/game-of-life/.git/lfs/objects/30/e1/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58: file already closed
16:42:40.497184 trace git-lfs: tq: enqueue retry #2 for "30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58" (size: 1048576)
16:42:40.497215 trace git-lfs: tq: sending batch of size 1
16:42:40.497321 trace git-lfs: api: batch 1 files
16:42:40.497404 trace git-lfs: HTTP: POST https://username@server.example.com/game-of-life.git/info/lfs/objects/batch
16:42:41.137811 trace git-lfs: HTTP: 200
16:42:41.140226 trace git-lfs: HTTP: {"transfer": "basic", "objects": [{"oid": "30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58", "size": 1048576, "authenticated": true, "actions": {"upload": {"header": {"Authorization": "Basic bGFodWdoZXM6", "Transfer-Encoding": "chunked"}, "href": "http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58"}, "verify": {"header": {"Authorization": "Basic bGFodWdoZXM6"}, "href": "http://server.example.com/game
16:42:41.140636 trace git-lfs: HTTP: -of-life.git/info/lfs/verify"}}}]}
Git LFS: (0 of 1 files) 160.00 KB / 1024.00 KB
16:42:41.142918 trace git-lfs: HTTP: PUT http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
16:42:41.149051 trace git-lfs: HTTP: 301
16:42:41.149083 trace git-lfs: api: redirect PUT http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58 to https://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
16:42:41.149113 trace git-lfs: HTTP: PUT https://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
16:42:41.149239 trace git-lfs: tq: retrying object 30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58: LFS: Put https://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58: read /home/username/game-of-life/.git/lfs/objects/30/e1/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58: file already closed
16:42:41.149264 trace git-lfs: tq: enqueue retry #3 for "30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58" (size: 1048576)
16:42:41.149289 trace git-lfs: tq: sending batch of size 1
16:42:41.149368 trace git-lfs: api: batch 1 files
16:42:41.149450 trace git-lfs: HTTP: POST https://username@server.example.com/game-of-life.git/info/lfs/objects/batch
16:42:41.706401 trace git-lfs: HTTP: 200
16:42:41.709235 trace git-lfs: HTTP: {"transfer": "basic", "objects": [{"oid": "30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58", "size": 1048576, "authenticated": true, "actions": {"upload": {"header": {"Authorization": "Basic bGFodWdoZXM6", "Transfer-Encoding": "chunked"}, "href": "http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58"}, "verify": {"header": {"Authorization": "Basic bGFodWdoZXM6"}, "href": "http://server.example.com/game
16:42:41.709513 trace git-lfs: HTTP: -of-life.git/info/lfs/verify"}}}]}
16:42:41.709949 trace git-lfs: HTTP: PUT http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
Git LFS: (0 of 1 files) 416.00 KB / 1024.00 KB
16:42:41.714517 trace git-lfs: HTTP: 301
16:42:41.714555 trace git-lfs: api: redirect PUT http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58 to https://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
16:42:41.714565 trace git-lfs: HTTP: PUT https://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
16:42:41.714709 trace git-lfs: tq: retrying object 30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58: LFS: Put https://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58: read /home/username/game-of-life/.git/lfs/objects/30/e1/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58: file already closed
16:42:41.714734 trace git-lfs: tq: enqueue retry #4 for "30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58" (size: 1048576)
16:42:41.714762 trace git-lfs: tq: sending batch of size 1
16:42:41.714844 trace git-lfs: api: batch 1 files
16:42:41.714938 trace git-lfs: HTTP: POST https://username@server.example.com/game-of-life.git/info/lfs/objects/batch
16:42:42.180936 trace git-lfs: HTTP: 200
16:42:42.183428 trace git-lfs: HTTP: {"transfer": "basic", "objects": [{"oid": "30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58", "size": 1048576, "authenticated": true, "actions": {"upload": {"header": {"Authorization": "Basic bGFodWdoZXM6", "Transfer-Encoding": "chunked"}, "href": "http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58"}, "verify": {"header": {"Authorization": "Basic bGFodWdoZXM6"}, "href": "http://server.example.com/game
16:42:42.183707 trace git-lfs: HTTP: -of-life.git/info/lfs/verify"}}}]}
16:42:42.183952 trace git-lfs: HTTP: PUT http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
Git LFS: (0 of 1 files) 480.00 KB / 1024.00 KB
16:42:42.189689 trace git-lfs: HTTP: 301
16:42:42.189734 trace git-lfs: api: redirect PUT http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58 to https://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
16:42:42.189744 trace git-lfs: HTTP: PUT https://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
16:42:42.189953 trace git-lfs: tq: retrying object 30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58: LFS: Put https://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58: read /home/username/game-of-life/.git/lfs/objects/30/e1/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58: file already closed
16:42:42.189985 trace git-lfs: tq: enqueue retry #5 for "30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58" (size: 1048576)
16:42:42.190012 trace git-lfs: tq: sending batch of size 1
16:42:42.190106 trace git-lfs: api: batch 1 files
16:42:42.190188 trace git-lfs: HTTP: POST https://username@server.example.com/game-of-life.git/info/lfs/objects/batch
16:42:42.655481 trace git-lfs: HTTP: 200
16:42:42.659439 trace git-lfs: HTTP: {"transfer": "basic", "objects": [{"oid": "30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58", "size": 1048576, "authenticated": true, "actions": {"upload": {"header": {"Authorization": "Basic bGFodWdoZXM6", "Transfer-Encoding": "chunked"}, "href": "http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58"}, "verify": {"header": {"Authorization": "Basic bGFodWdoZXM6"}, "href": "http://server.example.com/game
16:42:42.659521 trace git-lfs: HTTP: -of-life.git/info/lfs/verify"}}}]}
16:42:42.659801 trace git-lfs: HTTP: PUT http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
Git LFS: (0 of 1 files) 704.00 KB / 1024.00 KB
16:42:42.676028 trace git-lfs: HTTP: 301
16:42:42.676084 trace git-lfs: api: redirect PUT http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58 to https://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
16:42:42.676100 trace git-lfs: HTTP: PUT https://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
16:42:42.676299 trace git-lfs: tq: retrying object 30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58: LFS: Put https://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58: read /home/username/game-of-life/.git/lfs/objects/30/e1/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58: file already closed
16:42:42.676329 trace git-lfs: tq: enqueue retry #6 for "30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58" (size: 1048576)
16:42:42.676355 trace git-lfs: tq: sending batch of size 1
16:42:42.676421 trace git-lfs: api: batch 1 files
16:42:42.676533 trace git-lfs: HTTP: POST https://username@server.example.com/game-of-life.git/info/lfs/objects/batch
16:42:43.355280 trace git-lfs: HTTP: 200
16:42:43.358107 trace git-lfs: HTTP: {"transfer": "basic", "objects": [{"oid": "30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58", "size": 1048576, "authenticated": true, "actions": {"upload": {"header": {"Authorization": "Basic bGFodWdoZXM6", "Transfer-Encoding": "chunked"}, "href": "http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58"}, "verify": {"header": {"Authorization": "Basic bGFodWdoZXM6"}, "href": "http://server.example.com/game
16:42:43.358636 trace git-lfs: HTTP: -of-life.git/info/lfs/verify"}}}]}
16:42:43.359125 trace git-lfs: HTTP: PUT http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
Git LFS: (0 of 1 files) 1.16 MB / 1024.00 KB
16:42:43.366469 trace git-lfs: HTTP: 301
16:42:43.366501 trace git-lfs: api: redirect PUT http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58 to https://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
16:42:43.366510 trace git-lfs: HTTP: PUT https://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
16:42:43.366648 trace git-lfs: tq: retrying object 30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58: LFS: Put https://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58: read /home/username/game-of-life/.git/lfs/objects/30/e1/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58: file already closed
16:42:43.366711 trace git-lfs: tq: enqueue retry #7 for "30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58" (size: 1048576)
16:42:43.366739 trace git-lfs: tq: sending batch of size 1
16:42:43.366806 trace git-lfs: api: batch 1 files
16:42:43.366882 trace git-lfs: HTTP: POST https://username@server.example.com/game-of-life.git/info/lfs/objects/batch
16:42:43.971651 trace git-lfs: HTTP: 200
16:42:43.974222 trace git-lfs: HTTP: {"transfer": "basic", "objects": [{"oid": "30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58", "size": 1048576, "authenticated": true, "actions": {"upload": {"header": {"Authorization": "Basic bGFodWdoZXM6", "Transfer-Encoding": "chunked"}, "href": "http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58"}, "verify": {"header": {"Authorization": "Basic bGFodWdoZXM6"}, "href": "http://server.example.com/game
16:42:43.974333 trace git-lfs: HTTP: -of-life.git/info/lfs/verify"}}}]}
16:42:43.974566 trace git-lfs: HTTP: PUT http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
Git LFS: (0 of 1 files) 1.47 MB / 1024.00 KB
16:42:43.979574 trace git-lfs: HTTP: 301
16:42:43.979608 trace git-lfs: api: redirect PUT http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58 to https://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
16:42:43.979618 trace git-lfs: HTTP: PUT https://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
16:42:43.979848 trace git-lfs: tq: retrying object 30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58: LFS: Put https://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58: read /home/username/game-of-life/.git/lfs/objects/30/e1/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58: file already closed
16:42:43.979878 trace git-lfs: tq: enqueue retry #8 for "30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58" (size: 1048576)
16:42:43.979906 trace git-lfs: tq: sending batch of size 1
16:42:43.979970 trace git-lfs: api: batch 1 files
16:42:43.980048 trace git-lfs: HTTP: POST https://username@server.example.com/game-of-life.git/info/lfs/objects/batch
16:42:44.622013 trace git-lfs: HTTP: 200
16:42:44.624631 trace git-lfs: HTTP: {"transfer": "basic", "objects": [{"oid": "30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58", "size": 1048576, "authenticated": true, "actions": {"upload": {"header": {"Authorization": "Basic bGFodWdoZXM6", "Transfer-Encoding": "chunked"}, "href": "http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58"}, "verify": {"header": {"Authorization": "Basic bGFodWdoZXM6"}, "href": "http://server.example.com/game
16:42:44.624758 trace git-lfs: HTTP: -of-life.git/info/lfs/verify"}}}]}
16:42:44.625233 trace git-lfs: HTTP: PUT http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
Git LFS: (0 of 1 files) 1.69 MB / 1024.00 KB
16:42:44.630536 trace git-lfs: HTTP: 301
16:42:44.630571 trace git-lfs: api: redirect PUT http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58 to https://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
16:42:44.630580 trace git-lfs: HTTP: PUT https://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
16:42:44.630828 trace git-lfs: tq: refusing to retry "30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58", too many retries (8)
Git LFS: (0 of 1 files) 1.75 MB / 1024.00 KB
LFS: Put https://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58: read /home/username/game-of-life/.git/lfs/objects/30/e1/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58: file already closed
These lines are interesting:
16:42:39.851141 trace git-lfs: HTTP: PUT http://server.example.com/game-of-life.git/info/lfs/objects/30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58
I don't understand why git-lfs is trying to put the data to http and not https.
Any help is appreciated.
Updated by Ricardo Rodrigues over 5 years ago
I'm also having this issue. Were you able to fix it? How?
It seems that when it tries to redirect, the file is already closed.
Updated by Marcin Kuzminski [CTO] over 5 years ago
Hi Rocardo,
Have you tried to set force_ssl in rhodecode.ini configuration ?
Maybe that would solve it.
Updated by Ricardo Rodrigues over 5 years ago
Hi Marcin, thank you for your reply.
I've tried with both force_ssl false and true. Either way git-lfs tries to use HTTP instead of HTTPS.
The problem can arise from several places. It is a git-lfs bug? Is it a file permission problem? Is it a Rhodecode bug?
I'm running Rhodecode 4.16.2 Community on a Debian 8.11 jessie. If you want I can run some tests here.
PS: The option "Require SSL for vcs operations" always fails to allow pulls and pushes. Is it related? I don't know how to fix it, but that should be another issue.
Updated by Ricardo Rodrigues over 5 years ago
Ok, here's an update.
The issue is in the LFS server implementation regarding how it deals with HTTP to HTTPS redirects. I posted an issue on git-lfs Github and this is the response:
Thanks for this information.
The problem here is that your server is producing a 302 in this case. Go's HTTP redirection support changes the request from a PUT to a GET in this case (since that's what web browsers do), so even if we rewound the body, this request wouldn't work, since GET requests can't have a body and the server wouldn't accept the request. You'll need to adjust your LFS server implementation so that it provides the correct URL (one using HTTPS) in the batch request. Git LFS doesn't provide a way to rewrite these requests, since the server is presumed to know where the data should be sent.
If the response were a 307 or 308, we could add support for that in Git LFS, but a 301 or 302 won't work here.
Here is the link for the conversation: https://github.com/git-lfs/git-lfs/issues/3698#issuecomment-507415888
Updated by Marcin Kuzminski [CTO] over 5 years ago
HI,
Thanks for the provided details. Our GIT-LFS server doesn't generate any 3XX codes at all: https://code.rhodecode.com/rhodecode-vcsserver/files/90644f979fa029551ff1daf50ed5ba123925519c/vcsserver/git_lfs/app.py
We only generate an url for obj in this call:
request.route_url('lfs_objects_oid', repo=repo, oid=oid)
This can generate either HTTP or HTTPS url based on headers sent from Apache/Nginx web-server.
Please check if you have in your Apache:
RequestHeader set X-Forwarded-Proto "https"
SetEnvIf X-Url-Scheme https HTTPS=1
Updated by Marcin Kuzminski [CTO] over 5 years ago
On second thought it might be actually a problem with RhodeCode.
Since vcsserver <-> rc communicate over http the URLs generate might be in fact wrong.
We'll confirm this and see if HTTPS headers are sent correctly from RhodeCode into vcsserver, so the URL generation is ok.
Updated by Marcin Kuzminski [CTO] over 5 years ago
You could try to modify that inside vcsserver source code followed by restart of vcsserver
from
request.route_url('lfs_objects_oid', repo=repo, oid=oid)
to
request.route_url('lfs_objects_oid', repo=repo, oid=oid, _scheme='https')
to quickly check !
Updated by Marcin Kuzminski [CTO] over 5 years ago
- Status changed from New to In Progress
- Priority changed from Normal to High
- Target version set to v4.17
Confirmed it's a bug, and we'll add a proper fix to the next release.
Updated by Ricardo Rodrigues over 5 years ago
Newer update. (sorry for the spam)
It's both an error from my end and Lyndsi.
I am serving Rhodecode with Apache and using Proxy mod. Meaning that when accessing the website url (through port 443 HTTPS) I proxy it to 10002 (where Rhodecode is running).
My problem is that, when I received a request for port 80, I made Apache redirect it to 443. Therefore I was breaking the git-lfs push.
To fix it, instead of redirecting I proxy port 80 to port 10002, and in the Rhodecode.ini toggle the force_https = true. This fixes the issue and shows it is not related with Rhodecode, but with its setup.
Updated by Ricardo Rodrigues over 5 years ago
From what you wrote it does seem to be a bug. Mainly because if I blocked every HTTP request to my server it wouldn't work right?
Because instead of using the HTTPS urls, it would create a HTTP instead and be blocked because of it.
My "fix" simply allows HTTP requests to be called. Correct?
Updated by Marcin Kuzminski [CTO] over 5 years ago
Yes, it seems that your fix just allowed communication over http. We discovered that the LFS object urls generated that are sent to the git client are always using HTTP. The simple change i posted would fix that. We have a fix already in place so it will be included in the next release.
Updated by Redmine Integration over 5 years ago
Commit 43af4e52b104 by Daniel Dourvaris daniel@rhodecode.com on default branch references this issue.
https://code.rhodecode.com/rhodecode-vcsserver/changeset/43af4e52b104b7272ba3a54cb1064900b1ce380c
Updated by Redmine Integration over 5 years ago
Commit b4126386c108 by Daniel Dourvaris daniel@rhodecode.com on default branch references this issue.
https://code.rhodecode.com/rhodecode-enterprise-ce/changeset/b4126386c108692e32713750ccb5874a9e59ecc5