Both of us are on att fiber/1gbit. I’m bypassed on gpon. Remote client sits behind a bgw320, also on gpon. Connection to all remote endpoints is over wg tunnel.
Had a zfs replication task fail at exactly 30m - it was a single snapshot that had ~180GB of new data since the last snapshot. Most of it completed. This was a daily scheduled overnight replication task in TN. Replication task is configured using ssh+nc.
This morning i ran a simple single stream iperf3 test to the same target. It too failed at 30m (30m20s to be exact).
I then reran the same iperf3 test from the same local client to my vps endpoint. No issues after 30m. Speeds were more sporadic given it’s a vps.
From my the local truenas instance to remote tn I get consistent 890mbps upload over wg. Test to the vps test fluctuated between 400 and 700 mbps.
Note, after both failures, it took 6-7 min to re-establish the wg connection. Remote logs indicated no internet connectivity at the remote end. It could ping the default gateway ip (192.168.1.254), but not the internet and no dns (server configured at 192.168.1.254) resolution.
Pings to 192.168.1.254 were successful indicating the bgw did not reboot. Also, bgw’s uptime does not indicate a crash.
An additional iperf3 test performed, this time limiting upload bandwidth to 500 mbps (-b 500M) was successful. I then tried another one at full speed with same 30m failure.
I don’t have much remote access to the bgw other than to ask the person running it to check certain things. Given the test to vps was successful suggests that att has some limits implemented on the bgw. I do recall some other replication tasks in the past that moved even more data (600gb) successfully, but that was scattered over multiple datasets and snapshots. I don’t believe there was a single snapshot exceeding 100GB.
Is this a known limitation with att or is there some other factor at play?
Is the AT&T link using CGN - carrier grade NAT? If so they might have timeout for a connection even if it active. I have seen that in the past.
Can you create another type of connection, let’s say SSH and does it drop? If you TCPdump the traffic at the 29 minute mark (to prevent making the cap to big) what do you see TCP wise?
I don’t understand your second paragraph. Isn’t ssh+nc and iperf3 considered different types of traffic? What are you suggesting to capture with tcpdump, and where (local or remote end). The only connection I have to the remote end is over wg. There are no open ports, it’s strictly outbound to my public ip with wg running on pfsense.
The question is can you use some other type of TCP connection between the 2 sides and keep it open for the 30 minute window and see if that drops outside of the WG tunnel? The goal is to verify it is all connections being reset and not just the tunnel. For the TCPdump, you can do it on your side. The goal is to see if the connection just drops or do you get a reset from the remote side.
When you say there is no internet connectivity at the remote end; clarification here - you have access to the logs on the remote side and it says the internet dropped? What device is on the other side, another PFsense box?
I have not looked at PFsense in a few years but there are some logging tools for connections that you should look at. Is there a log of the WG interface dropping?
Also if you search you will find that ATT does use CGN on some of its offerings so it was worth asking.
The remote side has no inbound ports open. It initiates the wg connection to my pfsense box by querying a particular fqdn (which pfsense manages and updates A record if there’s a change). To perform this test I’d need to open some kind of listener on my end.
I did do another test after posting the OP yesterday. In this scenario I limited the bw to 10mbps and set time interval for 1hr. It completed successfully with no drops.
As indicated in the OP, remote TN box sits behind a bgw320 box.
I left an ongoing ping to 1.1.1.1 in a tmux session on the remote end. Did an iperf test to trigger the failure. After failure reviewed the ping log. Sure enough ping failed at the exact same timestamp as the connection dropped. Ping did not resume for ~6 minutes, which also aligns with when the wg connection was re-established.
It’s a question if the outage is limited to only my tn box at the remote end or the entire lan loses connectivity. I believe the latter because the remote admin indicated other services also lost connectivity at timestamps which align with my own.
On the local end, nothing in pf suggests any failures or service restarts.
The typical send speed over wg is 890mbps. This is sustained for 9s, drops to 500-600 mbps at t=10s, then picks back up to 890mbps. This cycle is ongoing and repetitive. Given the great tunnel speeds I never bothered to figure out which end was causing this, or try to mitigate it.
Today I tested again but limiting the bw to 800 mbps and letting it run for an hr - -t 3600 -b 1800M --timestamps. I expected a failure at 1800 as before, but this time it completed the whole test with no failures. The bgw320 is rated to handle 5gb wan speeds. Now whether almost 1gb linespeed using udp is causing it to crap out or if there some other isp induced block that’s triggered by x% of line speed for time y is unknown. I’m leaning towards the bgw being a turd.
I’m effectively on the same tier as the remote end except there’s no isp equipment in path. Fiber comes in directly to the sfp ont module (in a managed switch) then goes out to the pfsense firewall over gigabit ethernet. Possible isp has some kind of qos implemented on the bgw that i’m unaffected by - recall same test to vps succeeded.
Running another test now with tcpdump running on the firewall to monitor both the wg interface and wan interface for Reset packets. Getting inconclusive results. It’s passed 1800s (30min) and still going at line speed. Yesterday this was not the case.
Sounds like you are doing the right bit of troubleshooting. The fact that a lower speed works would and the internet breaks completely on the remote side does point to the BGW or something else on that side as you surmised.
There is a tool called zettarepl that can bandwidth limit the replication task.
I am still stuck on the 30m thing. I feels like there is some timer or state that has to be timing out, or some buffer that fills at close to line rate that causes a crash.
I assume the BGW is in passthrough mode and the remove PFsense has the public IP?
Google seems to say that model has some issues with DHCP renewal in passthrough mode. Hum, when you get a lease on the PFSense box, what is the lease time? Imagine the DHCP renewal break because the pipe is full? To be clear, I am tossing darts here.
When the internet drops on the remote side can the local admin reach the management IP of the BGW? The default is 192.168.1.254 and you would have to put a forwarding rule on the PFSense to say packets from internal to this single address are forwarded out WAN interface. Now if the interface has lost its DHCP this would not work.
There is no pfsense on the remote side, just the truenas box behind the bgw as a standard client (192.168.1.0/24 ip). No passthrough.
Local management (192.168.1.254) is pingable when the wg connection drops. I didn’t try accessing the UI page. The ping was running in the tmux session on the tn box. Client (TN) lease time is 24h.
@truenas-fan You’re being too generous with att. Their garbage gateways offer no true bridge mode, just the passthrough garbage which passes the public ip in a pseudo 1:1 nat arrangement (except certain ports are restricted) to a single lan client.
This is why on my end I have no att equipment at all except the fiber line, giving me full control over my network.
Still, what’s really strange was today’s test results, no disconnects with full speed iperf3 test. I let it run for about 40 min just to see if it would eventually fail, it didn’t. Yesterday (8/19) this was not the case. It dropped multiple times at exactly the 30m mark (+ a few sec).
I don’t have remote admin access to the bgw, but can curl some diag pages that require no auth. Last reboot was ~18 days ago. It’s on Software Version 6.34.7.
I think at this point I will leave it as is and just deal with the occasional resume if it happens again. TN (local) sends out plenty of emails to let me know the replication task failed.
This whole arrangement has been in place for over a year now, albeit with a different box through july of this year - older TN, but same general config with tcp over wireguard. The only main factor is this one large (180GB) dataset snapshot. All others have been < 100GB.
I’m familiar with the att passthrough mode and it should work fine, but then,
what are the routers on both ends?
Network monitoring goes a long way for these things, but if not available, easiest test would be to have several continuous pings going and see which fail at some point:
dev A pings internet address of dev B
dev A pings local router
dev A pings google/internet
dev B pings internet address of dev A
dev B pings local router
dev B pings google/internet
TCP should reconnect a brief interruption, but not a long one.
You could simply increase the number of retries in the Replication task, and force it to retry long enough until connectivity comes back.
Interesting tidbit regarding the hr long test that did not fail.
iperf3 command line used -b 1800M , according to ai this implements some sort of qos to control packet flow
Why -b 1800M (or any high cap) Actually Helps
It sounds counterintuitive that setting a target of 1800M (higher than your gigabit link) keeps the connection stable, but it changes how iperf3 spaces out its packet pacing:
Pacing Control: When you specify a target bitrate with -b, iperf3 uses an internal token bucket or pacing mechanism to smooth out the packet transmission rather than dumping raw packets into the OS socket buffer as fast as the CPU can process them.
Reduced Micro-Bursts: Even though your actual throughput is still around 900 Mbps, the gentle pacing prevents sudden, massive packet bursts that overwhelm the WireGuard encapsulation queue or state tables on your hardware router. This keeps packet loss at absolute zero over long durations.
I retested again this morning without the -b 1800 and sure enough it failed as before.
@truenas-fan pfsense on the sending side, bgw320 on the receiving side, already mentioned earlier in the thread.
Considering it takes 6-7 min after a failure to regain connectivity, doubt even setting retries would work as each retry would fail to reach the target. WG is down during this time. Another option is to schedule a clone of this replication task to run an hr or two later. If the first task was successful then second one would finish instantly. If not, it would resume where the first one failed.
There is a network monitoring service on the remote tn box that checks for connectivity every minute. Part of the logic is if the bgw is accessible, but wg is down, restart wg. The flaw in the logic is not verifying if internet is also accessible.
I don’t think this matters too much - when the connection drops and there’s no internet, even if wg was using the cached ip of the FQDN target (my local pfsense wan ip), it would still fail to reconnect.
I’m still 99.9% confident the bgw at the remote end is the cause of all this. Firmware likely not robust enough to handle long duration udp traffic, and/or maybe there’s artificial qos implemented by the isp on the CPE.
