fix(connectivity): replace flaky example.org DNS probe with dual-stack fallback - #975
fix(connectivity): replace flaky example.org DNS probe with dual-stack fallback#975odesenfans wants to merge 3 commits into
Conversation
…k fallback example.org is an RFC 2606 documentation domain, never meant as a live resolution target, and has had DNS flakiness (notably AAAA). Both the node's host-status DNS check and the diagnostic VM's /dns endpoint resolved it and required both A and AAAA records, so a single flaky lookup failed /status/check/fastapi -> broke the droplet integration test (curl --fail). Replace the single hostname with an ordered list of reliable dual-stack hosts from two independent providers (Cloudflare one.one.one.one, then Google dns.google), tried until one resolves -- mirroring the existing CONNECTIVITY_HTTP_URLS fallback. A single provider's DNS hiccup no longer fails the check, and the value stays env-overridable. - conf.py: CONNECTIVITY_DNS_HOSTNAME (str) -> CONNECTIVITY_DNS_HOSTNAMES (list) - host_status.check_dns_ipv4/ipv6: first-success loop, tolerate per-host OSError - example_fastapi /dns: loop hosts, 200 once a host yields both families else 503 - tests: cover first-host success, fallback on no-address / on error, all-fail
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #975 +/- ##
==========================================
+ Coverage 73.02% 73.09% +0.06%
==========================================
Files 118 118
Lines 14631 14675 +44
Branches 1151 1156 +5
==========================================
+ Hits 10685 10727 +42
- Misses 3643 3644 +1
- Partials 303 304 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
foxpatch-aleph
left a comment
foxpatch-aleph
left a comment
There was a problem hiding this comment.
Replaces single DNS hostname with a fallback list in connectivity checks and fixes both check_dns_ipv4 and check_dns_ipv6 to iterate through hostnames with proper error handling, improving reliability. Tests cover first-success, fallback, and all-fail scenarios correctly.
…_PROGRAM The droplet integration tests fail with total DNS breakage inside guest VMs (socket.gaierror EAI_AGAIN for every hostname, raw-IP egress fine). Root cause: DNS auto-detection on the droplet now picks up DigitalOcean's VPC-internal resolver (10.110.15.254), which guests cannot reach through the host NAT. Pin ALEPH_VM_DNS_NAMESERVERS to public resolvers (1.1.1.1, 8.8.8.8) in both CI supervisor.env blocks so guest VMs get reachable nameservers. Separately, the "Test new runtime" job has been silently running the network-published diagnostic VM (and downloaded runtime) instead of the local ones: cli.py unconditionally wrote args.fake_data_program (argparse default None) over the settings, discarding ALEPH_VM_FAKE_DATA_PROGRAM from supervisor.env since c827c20 (Oct 2023). The startup settings dump in CI confirmed FAKE_DATA_PROGRAM = None, and the guest traceback matched the January-published example code (dec8f47), not the repo checkout. Only override the setting when -f/--fake-data-program is actually passed, so the env var works again and the job tests the code it claims to test. Verified: systemd EnvironmentFile passes the JSON list value through verbatim and pydantic-settings parses it; supervisor unit suites pass.
|
Same CI root-cause analysis and fixes as #974 (comment) (cherry-picked as b8ce139): pin guest DNS to public resolvers in CI (droplet auto-detection now hands guests an unreachable DigitalOcean VPC resolver), and fix cli.py discarding ALEPH_VM_FAKE_DATA_PROGRAM (dead since Oct 2023) so the new-runtime job tests local code again. |
With ALEPH_VM_FAKE_DATA_PROGRAM working again, the supervisor's startup check asserts that the example venv volume exists (FAKE_DATA_VOLUME, /opt/examples/volumes/volume-venv.squashfs) -- it is a built artifact, not a checked-in file, so `scp -pr ./examples` alone cannot provide it. Reuse the artifact from the existing build_example_venv_volume job: download it into examples/volumes/ before the scp so it lands where the assert (and the fake volume mount) expects it.
foxpatch-aleph
left a comment
foxpatch-aleph
left a comment
There was a problem hiding this comment.
The test suite for DNS check fallback logic is well-structured and correctly validates all key behaviors: first-host success, fallback on no-address, fallback on raised exception, and total failure returning False. All patches use the correct settings variable name (CONNECTIVITY_DNS_HOSTNAMES, plural) matching the source code.
tests/supervisor/test_status.py (line 176): This IPv6 test covers the "first host has no IPv6" case but not the "first host raises" case (unlike the IPv4 tests which cover both). Consider adding test_check_dns_ipv6_falls_back_when_first_host_raises for symmetry.
2 participants
Backport of #974 to
main.Problem
The droplet integration test (
curl --fail .../status/check/fastapi) is flaky because the DNS connectivity probe resolvesexample.org— an RFC 2606 documentation domain not meant as a live, dual-stack target, which has had recurring DNS flakiness (notably AAAA). The diagnostic VM's/dnsrequires both A and AAAA, so a flaky lookup returns 503 →check_dnsFalse →status_check_fastapi503 → job fails.Fix
Replace the single hostname with an ordered list of reliable dual-stack hosts from two independent providers, tried until one resolves — mirroring the existing
CONNECTIVITY_HTTP_URLSfallback:one.one.one.onefirst (matchesCONNECTIVITY_IPV4_URL = https://1.1.1.1/), Googledns.googlefallback (independent failure domain).host_status.check_dns_ipv4/ipv6: first-success loop, tolerating per-hostOSError./dns: loops hosts, 200 once a host yields both families else 503.Identical change to #974 (the
devPR).vm.example.orginhaproxy.pyis a string sentinel, not a DNS lookup — left unchanged.Tests
tests/supervisor/test_status.py: 11 passed (added first-host-success, fallback-on-no-address, fallback-on-error, all-fail cases).