request should be able to retry upon broken connections #993
Description
What needs to get done
When using remote-build service in rockcraft, there are often build failures caused by network instabilities.
The network instability causes the unhandled ChunkedEncodingError exception inside craft_application/services/request.py", line 71, in download_chunks.
When the *craft tool is downloading the artifact from the build farm, it should be able to retry upon broken connections, such that when using *craft tools in CI, the occasional failures happening during the download would not lead to a full rebuild.
A log of such an error is attached below:
2026-01-23 17:16:25.270 Succeeded: amd64, arm64, armhf, ppc64el, riscv64, s390x
2026-01-23 17:16:25.271 Fetching build artifacts...
2026-01-23 17:16:31.933 Downloading 6 files (--->)
2026-01-23 17:17:38.222 Downloading 6 files (<---)
2026-01-23 17:17:38.228 rockcraft internal error: ChunkedEncodingError(ProtocolError('Connection broken: IncompleteRead(14610378 bytes read, 25147446 more expected)', IncompleteRead(14610378 bytes read, 25147446 more expected)))
2026-01-23 17:17:38.255 Traceback (most recent call last):
2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/urllib3/response.py", line 779, in _error_catcher
2026-01-23 17:17:38.255 yield
2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/urllib3/response.py", line 904, in _raw_read
2026-01-23 17:17:38.255 data = self._fp_read(amt, read1=read1) if not fp_closed else b""
2026-01-23 17:17:38.255 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/urllib3/response.py", line 887, in _fp_read
2026-01-23 17:17:38.255 return self._fp.read(amt) if amt is not None else self._fp.read()
2026-01-23 17:17:38.255 ^^^^^^^^^^^^^^^
2026-01-23 17:17:38.255 File "/snap/rockcraft/current/usr/lib/python3.12/http/client.py", line 495, in read
2026-01-23 17:17:38.255 s = self._safe_read(self.length)
2026-01-23 17:17:38.255 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-01-23 17:17:38.255 File "/snap/rockcraft/current/usr/lib/python3.12/http/client.py", line 642, in _safe_read
2026-01-23 17:17:38.255 raise IncompleteRead(data, amt-len(data))
2026-01-23 17:17:38.255 http.client.IncompleteRead: IncompleteRead(14610378 bytes read, 25147446 more expected)
2026-01-23 17:17:38.255
2026-01-23 17:17:38.255 The above exception was the direct cause of the following exception:
2026-01-23 17:17:38.255 Traceback (most recent call last):
2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/requests/models.py", line 820, in generate
2026-01-23 17:17:38.255 yield from self.raw.stream(chunk_size, decode_content=True)
2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/urllib3/response.py", line 1091, in stream
2026-01-23 17:17:38.255 data = self.read(amt=amt, decode_content=decode_content)
2026-01-23 17:17:38.255 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/urllib3/response.py", line 980, in read
2026-01-23 17:17:38.255 data = self._raw_read(amt)
2026-01-23 17:17:38.255 ^^^^^^^^^^^^^^^^^^^
2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/urllib3/response.py", line 903, in _raw_read
2026-01-23 17:17:38.255 with self._error_catcher():
2026-01-23 17:17:38.255 File "/snap/rockcraft/current/usr/lib/python3.12/contextlib.py", line 158, in __exit__
2026-01-23 17:17:38.255 self.gen.throw(value)
2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/urllib3/response.py", line 806, in _error_catcher
2026-01-23 17:17:38.255 raise ProtocolError(f"Connection broken: {e!r}", e) from e
2026-01-23 17:17:38.255 urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(14610378 bytes read, 25147446 more expected)', IncompleteRead(14610378 bytes read, 25147446 more expected))
2026-01-23 17:17:38.255
2026-01-23 17:17:38.255 During handling of the above exception, another exception occurred:
2026-01-23 17:17:38.255 Traceback (most recent call last):
2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/craft_application/application.py", line 669, in run
2026-01-23 17:17:38.255 return_code = self._run_inner()
2026-01-23 17:17:38.255 ^^^^^^^^^^^^^^^^^
2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/craft_application/application.py", line 646, in _run_inner
2026-01-23 17:17:38.255 return_code = dispatcher.run() or os.EX_OK
2026-01-23 17:17:38.255 ^^^^^^^^^^^^^^^^
2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/craft_cli/dispatcher.py", line 564, in run
2026-01-23 17:17:38.255 return self._loaded_command.run(self._parsed_command_args)
2026-01-23 17:17:38.255 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/craft_application/commands/base.py", line 222, in run
2026-01-23 17:17:38.255 result = self._run(parsed_args, **kwargs) or result
2026-01-23 17:17:38.255 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/craft_application/commands/remote.py", line 165, in _run
2026-01-23 17:17:38.255 returncode = self._monitor_and_complete(builds=builds)
2026-01-23 17:17:38.255 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/craft_application/commands/remote.py", line 224, in _monitor_and_complete
2026-01-23 17:17:38.255 artifacts = builder.fetch_artifacts(pathlib.Path.cwd())
2026-01-23 17:17:38.255 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/craft_application/services/remotebuild.py", line 207, in fetch_artifacts
2026-01-23 17:17:38.255 return self.request.download_files_with_progress(artifact_downloads).values()
2026-01-23 17:17:38.255 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/craft_application/services/request.py", line 109, in download_files_with_progress
2026-01-23 17:17:38.255 for chunk_size in dl:
2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/craft_application/services/request.py", line 71, in download_chunks
2026-01-23 17:17:38.255 for chunk in download.iter_content(None):
2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/requests/models.py", line 822, in generate
2026-01-23 17:17:38.255 raise ChunkedEncodingError(e)
2026-01-23 17:17:38.255 requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(14610378 bytes read, 25147446 more expected)', IncompleteRead(14610378 bytes read, 25147446 more expected))
Why it needs to get done
As remote-build can take a long time to be built, purging a successful build with a single failed download attempt seems very inefficient and wasteful. We should let *craft tools give multiple tries to improve the chances of successful builds.