v055x: Simple mode WireGuard: "Proxy Error" for all requests

[Terrance]

This started while the phone was idle (and low battery, if relevant). Stopping WireGuard would restore connections, but starting it again would cause all requests to error out again. Stopping/starting Rethink as a whole got things fully working again. WireGuard config is split-tunnel, but both routed and non-routed addresses error out.

All non-blocked requests would show _Proxy Error_ when opened from the Network logs:

Rethink's IP version is set to Auto. Android VPN slot is always-on and blocking non-VPN connections. Most other settings should be default.

This is the second time I've spotted this, but I had a moment to capture and sanitise all the built-in debug info this time (I didn't think to examine ip route etc. before restarting, sorry):

📎 log_1784656762825.txt
📊 General, Proc, Stacktrace

(Aside: it would be useful if we could export an anonymised log with domains and public IPs randomised/redacted -- I have a somewhat hacky script to try and do it myself currently.)

I haven't manually updated to v055y yet, but if you think this is something likely resolved there then I can do and watch for a reoccurrence.

[ignoramous]

From the About -> General output (Dialer4:true, Dialer6:true) and the diagnostic msg in the screenshot (notice he? true, which means the network engine is expecting the source app to perform "happy eyeballs"), the "proxy error" is expected, in that, IPv4 egress is being dropped in favour of some other (presumably) active IPv6 connection (which might already be inflight from the same app to the same endpoint but over IPv6).

May be the fact that you've Configure -> DNS -> Advanced DNS filtering turned on might be confusing the network engine (egress) dialers, but I haven't taken a critical look at that (nor is there any hint of anything going wrong in the logs shared). And the behaviour is also different on Android 12+ versus prior versions (Android 11 & below).

When you see these "Proxy error" messages, does internet simply not work for the apps (against which these happen)?

I haven't manually updated to v055y yet, but if you think this is something likely resolved there then I can do and watch for a reoccurrence.

I don't think v055y would do behave any differently here.

(Aside: it would be useful if we could export an anonymised log with domains and public IPs randomised/redacted -- I have a somewhat hacky script to try and do it myself currently.)

It is too much work to get it done efficiently (but we should)... let's see. Simply masking IPs and domains may make the bug report (depending on the issue being reported) useless, so that's there too.

• One-tap export of configuration summary / logs #1523

[Terrance]

Both the WireGuard connection and the connected wi-fi network (both the same network overall; WG tunnels to a device on the same network) were dual-stack, so I'd expect IPv6 and happy eyeballs to work.

When you see these "Proxy error" messages, does internet simply not work for the apps (against which these happen)?

From other apps' perspective, seemed like everything was just timing out.

IPv4 egress is being dropped in favour of some other (presumably) active IPv6 connection

Interesting. Both v4 and v6 were seemingly failing -- both inadvertently dropped in favour of the other? The majority of requests were probably v4-only (e.g. all the internal network ones over WG, which use internal DNS with both A and AAAA but I'm not sure I've ever seen v6 get picked). Syncthing was failing on both, though some amount of Proxy Error for that comes in normally as it probes all the networks including Rethink's internal IPs.

[ignoramous]

Stopping/starting Rethink as a whole got things fully working again. WireGuard config is split-tunnel, but both routed and non-routed addresses error out.

Quite possible the underlying TCP/IP stack implementation we use (of Google's gVisor netstack) is acting up. Or, we aren't making the correct use of its APIs.

From other apps' perspective, seemed like everything was just timing out.

I have seen the behaviour you describe while building v055o (which went on for a year) and other subsequent versions, and I vividly remember it was easy to hit it with Android's Block connections without VPN turned on (but we never got to the bottom of why it happened).

Can you see if setting Configure -> Network -> Connection change policy to "Sensitive" eliminates these timeouts? The Sensitive setting instruct Rethink to restart its network engine (gVisor netstack) on most network changes (like routes, MTU, availability/loss, etc). Today, in the default "Auto" setting, the network engine is never restarted.

Note that, "Sensitive" itself might be buggy (or restart the engine far too aggressively) simply for the fact that it hasn't been tested as extensively as "Auto". If so, you can try "Relaxed" instead.

Also, do you change any setting in the section Configure -> Network -> TCP/IP from the defaults (other than Choose IP version)?

[Terrance]

Can you see if setting Configure -> Network -> Connection change policy to "Sensitive" eliminates these timeouts?

Now set; we'll see if it happens again.

Also, do you change any setting in the section Configure -> Network -> TCP/IP from the defaults (other than Choose IP version)?

I don't think so:

• Shorter TCP keepalive: disabled
• Endpoint-independent mapping: disabled
• Idle timeout: disabled
• Bandwidth booster: disabled
• Socket Buffer Size: 512 KB
• Choose IP version: Auto
• Perform connectivity checks: None
• Custom Interface IPs for VPN: Auto

[Terrance]

Had another case today, same symptoms (again started after being left idle, not low battery this time though) -- turning wi-fi off resolved it, which I guess is the connection change policy kicking in.

[ignoramous]

turning wi-fi off resolved it, which I guess is the connection change policy kicking in.

I think you'll have to remain on Sensitive (perhaps, Relaxed may also help). May be, we should make it the default on Android 14 or below, once we test it enough...

I'm disappointed that Rethink did not auto-recover (that you had to turn off/on WiFi) and that you still hit this issue with Connection change policy set to "Sensitive".

same symptoms (again started after being left idle, not low battery this time though

From what I recall, assuming it is the same issue you're seeing, the OS fails to correctly add routing rules for VPN packets (which can be seen in ip route show table all & ip rule) when Block connections without VPN is turned on alongside Rethink's Configure -> Network -> Stall on network loss and/or Configure -> Network -> Use all available networks settings. Though, from the About -> General output shared in the first post, it looks like you do not use the latter. Do you use the former, though?

Another surprising thing I spot in About -> General output shared above is that the primary network (WiFi) reports 0 MTU, which Rethink treat as 1500 (as instructed in the official Android SDK docs). Though, mismatch in MTUs (of the VPN tunnel setup by Rethink and of the underlying network like WiFi / Mobile) should only really affect UDP (packet fragmentation) if at all, and not TCP (as it does its own "Max Segment Size" negotiation), but who knows... This one reason why Configure -> Network -> Bandwidth Booster even works (it always sets Rethink's VPN tunnel MTU to 10000, the maximum allowed by Android).

All that said, it really could simply be a bug in how Rethink uses gVisor's netstack. I'll keep looking in gVisor's code (though, I can barely keep up given the complexity, to an untrained eye, in that part of the codebase).

[Terrance]

the OS fails to correctly add routing rules for VPN packets (which can be seen in ip route show table all & ip rule)

I'll try and remember to grab those next time 😓

when Block connections without VPN is turned on alongside Rethink's Configure -> Network -> Stall on network loss and/or Configure -> Network -> Use all available networks settings

Neither of Stall on network loss or Use all available networks are switched on; both are greyed out from being in lockdown mode (I assume this is the same Block All Connections Not Using VPN as OnePlus calls it) in any case.

the primary network (WiFi) reports 0 MTU, which Rethink treat as 1500 (as instructed in the official Android SDK docs)

That agrees with what I see via adb at least:

$ ip link | grep -E 'wlan|tun'
19: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DORMANT group default qlen 3000
136: tun0: <POINTOPOINT,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 500

[ignoramous]

I'll try and remember to grab those next time 😓

I did see a reproduction of this on my test Android 14 (easier on those devices) with Advanced mode WireGuard on both dual-stack and IPv4-only networks (with Loopback mode turned on or off & Stall on network loss turned on or off), when user presence is registered (device unlocked) after a significantly long time (like 5h+).

Only restarting the full tunnel (via STOP / START) internet connectivity recovers instantly. Setting Connection change policy to "Sensitive" didn't seem to recover an already spooked tunnel.

The ip rule and ip route show table all don't seem to indicate Rethink is hitting the previously discovered AOSP VPN routing bug (triggered by a combination of VpnService.setUnderlyingNetworks, implementation of "seamless handover" between 2 VPN tunnels (owned by the same VPN app) with Android's "Block connection without VPN" is turned on): It seems like an entirely new issue (I think, perhaps a regression) that's resurfaced, because I used to hit similar issue often (on this same Android 14 device) during v055o development process.

[ignoramous]

@Terrance I went "Relaxed" on Configure -> Network -> Connection change policy (on latest, v055z release candidate) with Configure -> Network -> Stall on network loss turned off (not sure if it matters) and haven't seen this issue since on the test device. Can you try that (even on v055x)?