Application-Specific Network Issue Identification
March 25, 2024
Situational Awareness: Nefarious Activity Observed over the Memorial Day Long Weekend from a Rogue Nation
September 27, 2024
Moving to a new area often comes with the challenge of availing reliable internet service. Recently, I experienced this firsthand when I relocated to the outskirts of Vacaville, California. Used to the high-quality service of fiber internet bundled with my phone plan, I was hopeful of maintaining similar standards. I previously had fiber and was hoping to keep that same high quality of service, but the new area I live in is on the outskirts of a smaller town and fiber is not available. In fact, no wired internet is available from this ISP in my area. Instead, they had me switch over to a type of home internet that operates on the 5G cellular network.
The Shift to 5G Home Internet
A standard internet connection will typically be wired and will go through a cross-box located somewhere close by in the neighborhood so that the ISP can run cable to this box to provide wired internet to your home. This will often be one of those green-looking boxes with some unspecified hardware living inside that we mostly ignore in our day-to-day lives. Certain ISPs are now experimenting with eliminating this type of infrastructure and instead building home internet services into their existing 5G networks which also handle all their cellular data. To do this, they provide a device that acts as a Wi-Fi router and can connect to this 5G network. This will save on costs since they no longer need to troubleshoot and maintain so much distributed infrastructure. Maintaining one cell tower is cheaper than maintaining dozens of cross boxes servicing hundreds of homes, especially since they already have the cell towers.
I work from home frequently, so I need good quality internet to collaborate with my team which includes video calls, messaging, and sharing files. I was skeptical this type of internet would work well for me in my work-from-home capacity but was also curious to find out more about how it works.
On the left is the router provided by the ISP which connects to a cell tower on the 5G network. On the right is my Wi-Fi mesh network (TP-Link Deco).
Ennetix xVisor
At Ennetix, we developed xVisor, a software solution designed to actively probe, measure network performance, and estimate end user experience based on these measurements. As I was skeptical about the performance of 5G home Internet, throughout the process of getting set up with 5G home Internet, I’ve used this to enhance my troubleshooting and show evidence of issues in the network.
The way xVisor works to measure a network is to send out probes (network packets) periodically and to capture and analyze the responses to give an estimation of the performance of the network. We do this in various ways to analyze several distinct metrics:
- Latency
- Loss
- Bandwidth
- Jitter
- Load balanced routes
- Link latency/loss
To get visibility from a specific access point on the network, this software is deployed as an agent that can run on any computer in a home, office, or university campus.
Frequently we run this type of active probing on a Raspberry Pi (an affordable mini-computer) since the software is lightweight. From our xVisor web dashboard, getting set up for probing is just a few clicks and then running a few commands on the agent terminal.
Our software confirmed what I suspected: that it wouldn’t work for my home office usage when I work from home. The internet provided by this 5G service performs poorly on almost every metric we measure. As a reminder, this is on the outskirts of the small town I just moved to, and this service may work perfectly fine in other areas.
Our software can be used by end users of internet services as evidence to their provider that they are not meeting expectations. Customers should get what they pay for, and we can empower them to do so.
xVisor can also be used by ISPs directly to get a better idea of what areas will work well for their 5G services, and which areas still need work and shouldn’t be offered to customers in that area.
Performance Issues Observed using xVisor
The latency (ping) is often 1000+ milliseconds. Standard ping on a Windows laptop gives the same results.
Jitter is a measure of the timing variability of packet delivery. It is most significant for media streaming and for VoIP calls, like Zoom or Microsoft Teams video calls. Measurements around the 50ms range are good and should have no visible issues to end users. More than that, especially in the 100+ range is likely to result in noticeable quality issues for users on calls or streaming video. The jitter I observed peaks at around 1000 milliseconds, and I was unable to have any meaningful calls on Microsoft Teams under these conditions.
Periodic bandwidth measurements, similar to the internet speed test by Ookla, show that the bandwidth varies between 20-40 Mbps. The ISP did not give specific guarantees about what kind of bandwidth to expect, but in 2024 I would expect much more than this. To cross check, I ran Ookla speed test on speedtest.net that also showed the same thing: the download speeds are ~30Mbps.
The quality issues were most prominent in the afternoon and evening when networks are also typically the most congested. Even early in the morning, at the least congested times, the quality was still sub-par.
xVisor provides charts for these various metrics and our product has detected many alerts and quality issues starting in the afternoon.
We can see here that the first hop on the ISP network already has significant latency. The nodes on the left in my home network have much lower latency, so we can isolate the issue to the ISP 5G network. If the cell tower itself responded to pings, we could troubleshoot further, but even with this level of visibility, we can be confident where the issue is.
Troubleshooting with the ISP Technician
The way this 5G router works is that it must establish an adequate connection to the network before registration can officially be completed. The end user takes the router and places it in different rooms, plugging in to see how the signal looks from that position. There is a built-in speed test that displays results on the front when you first plug in the router and wait for boot-up.
When I went through this process, the best speed I managed to get was 40Mbps download and 9Mbps upload. This was when I had the router up in a window with direct line of sight to the cell tower. I don’t know the specific numbers that constitute “adequate” to this ISP, but this was apparently not enough. I was unable to complete registration under these conditions. My laptop was able to connect fine to the router, but it showed a “sign in required” notification. Surprisingly, I was still able to browse, but with the challenges of poor network quality.
Here is a view of the cell tower to the south-west of my house. I have a window with a direct line of sight to this tower.
At this point I called the ISP for help, thinking that there may be something to do on their end to fully enable service. I hoped that this would clear up the poor performance issues and give me reasonable internet to use for work. I had a few different technicians help me troubleshoot over the course of some days and they all had me do variations of the same thing, which is to move the router around to different locations to see if signal or bandwidth improved. We still had no luck after trying all the rooms in my house oncemore. Eventually after ~2 hours of phone calls for troubleshooting, the technician told me they would be unable to offer me reasonable internet and suggested that I cancel and that I wouldn’t be charged. Specifically, he said that the density of houses combined with congestion of the cell towers was causing my low speeds and other performance issues.
I was surprised by their recommendation to cancel, thinking they would put in considerable effort to keep customers and not have them cancel and move to other providers. Since no other internet option is offered by this ISP in my area, I had no choice but to switch.
