Blur Busters Forums

Advice: Retest VPNs every time you move to a new location/new ISP/new country

One thing I noticed many years ago.
- If I am in one city and use a VPN (e.g. NordVPN or ExpressVPN) I notice lag drops in one city.
- But when I fly to a different city and use the same VPN brand, I notice my lag increases in that city.

It's kinda location-specific crapshoot, so you have to play VPN roulette again when you move to a different city to live in.

TL;DR: One brand of VPN can work better in one city/ISP/country, a different brand of VPN can work better in another city/ISP/country.

Boop wrote: ↑

17 Jan 2021, 08:07

Overall, I saw a 30ms reduction in latency, which is huge for games that have projectiles such as Quake Live. In Quake Live's case, once you go above 80ms, the lag compensation starts to fall apart and you find yourself at a huge disadvantage in fights against low ping players. The backwards reconciliation with hit-scan weapons also becomes a problem beyond this threshold.

That's a big reduction in lag!

That said, I've seen bigger lag reductions than that before -- sometimes over 50ms less lag in random tests a long time ago.

(Many years ago before I thought to be important to write about it -- shame I didn't record all the data.)

But I've also seen lag increase too, it depends on location/city/country, depends on VPN, and depends on servers.

Crossposting (from another thread) some useful VPN insight from Boop:

Some older engines discarded out-of-order packets that are solved by a gaming-friendly VPN.

Boop wrote: ↑

21 Jan 2021, 02:43

Another issue that can be solved by using a VPN is when an ISP sends out of order UDP packets to a game server.

I spent quite some time troubleshooting issues with jitter and packet loss to servers hosted in AWS. I eventually found threads of many Fortnite players (FN is hosted in AWS) reporting packet loss on my ISP's forums. The ISP would only perform basic troubleshooting and just send a tech out to their homes to tell them there is no problem with their network.

I took all of this information and escalated the issue to the ISP executives through a friend. Epic Games was eventually contacted to help troubleshoot the problem. Shortly thereafter, Epic Games posted this thread on Reddit:
https://www.reddit.com/r/FortNiteBR/com ... ical_post/

TLDR: Unreal Engine was discarding packets that were received out of order!

I confirmed this was the issue all along by creating an EC2 Instance in AWS (Virtual Server) and performing some network tests using iPerf. The logs from iPerf showed there was 0% packet loss but the packets were received out of order!

The changes made to the Unreal Engine by Epic Games made a big difference and the ordering issue with my ISP eventually went away.

That said, this is not the only thing that a gaming-friendly VPN helps with (running retro games that has poor out-of-order packet handling) -- there can be other things that a gaming-friendly VPN can unwittingly/unexpectedly fix.

I built my own proxy for playing an ancient game hosted in China from west US. I would call it a proxy because it's not a VPN but it helps with internet lag. There are some interesting findings.

The biggest advantage a gaming VPN, or an alternative network stack brings is actually from software optimization. For example, sendto(127.0.0.1) can be faster than sendto some remote host. My game is especially sensitive to this because it was built around 2004 and it uses APC (asynchronous procedure call) to handle network. Everything - control, simulation, display, network, audio are handled in a single thread. I believe this is also what some gaming ethernet cards (like killer) do. Even with multithreading, this can still reduce thread contention - sometimes I heard people increased fps by using a VPN.

In my proxy, I was using LSP (layered service provider), a long deprecated interface for implementing winsock middlewares. One thing I found is that by only installing an empty LSP hooking all the functions but without any network redirection can make gameplay much better. LSP does a lot of things for historical reasons, like using IOCP (I/O completion port) to simulate APC, and no wonder why Microsoft deprecated it. There are other ways of implementing a proxy I haven't explored, like WFP and virtual network interfaces.

Network links with low latencies exist. For example, there are routes from Seattle to Tokyo with 82ms latency. The latency of the default route on public internet is usually higher. For anywhere on the world where internet exists other than (evil) China, it's easy to get near optimal routes from big (evil) cloud providers. One trick here is the routes are often optimized for bandwidth and aggregated with ECMP. The underlying routes usually have different latencies. You can manually pick the lowest latency one by changing ports. This is not recommended for building service but fun to do it personally.

I also tried UDP multipath, where the client sends one packet twice through different routes to the proxy server, and the proxy server dedupes the packet. FEC (forward error correction) is also interesting to try but my service has not grown to that scale - with low bandwidth FEC is the same as sending the packet twice.

Chief Blur Buster wrote: ↑

21 Jan 2021, 16:09

Some older engines discarded out-of-order packets that are solved by a gaming-friendly VPN.

Exactly the same thing happens in, most likely all, both newer & older FPS games & engines.
The FPS games i have been playing in the last two decades:

CS 1.6
CS:GO
Overwatch
PUBG
Black Squad
Apex Legends
Fortnite (very briefly, most likely before the out-of-order packet issue being addressed)

All of them had these issue and the worst, most painful and depressing experience i ever encountered was when playing PUBG. All these games have different netcodes, so some newer titles are able to better hide certain issues, but all of them are still present. CS 1.6 is a perfect example and test case to show how ridiculous this networking issue has become over the last two decades. I'm also not sure if our-of-order packets are the only thing that causes lags & bad behaviour in FPS games online. I'm also not sure if Fortnite patch actually fixed everything or is it just a workaround that still adds delay because of additional processing being needed to address the this issue. We also don't know how much, UDP packets that are being dropped completely (still 100% ISP dependency in relation to UDP packet handling), contribute to this problem.

I was very busy recently, and even after seeing Boop's post i still wasn't able to test everything i wanted to test. It will most likely take weeks, but I'm happy to report that based on my preliminary testing (ExpressVPN servers only) that indeed, VPN is able to address the internet lag issue in online FPS games. I still haven't found a server which would actually fix all these problems completely, but certain servers, even despite higher pings than my home Fibre 1Gb/s, perform much better in all aspects, but most importantly the perceived network performance is much more stable. By "fix" i specifically mean, all the problems going away and player having LAN like experience or something closest to LAN-like experience as possible. All of this means only one thing. You need to get lucky with a VPN in the exactly same way as you need to get lucky with your ISP in the place you live in. As explained in many different topics & posts on this forum, the issue with internet & first person shooter games exists on a wide spectrum in severity. One of the most starking differences i noticed was that one of the VPN servers that i connected to, has fixed enemy player positioning & their visibility 100%. It actually showed their true & accurate positions, so 1v1 encounters actually became equal.

This has lead me to realization that people need to rethink completely their understanding about playing online, because this situation shows that nothing in this matter is equal between players and online gaming & results are a complete joke. As mentioned in other posts, this has nothing to do with ping & packet loss reported by the game itself & other measurement tools. This problem covers the entire gaming community, which includes pro-gaming, competitive gaming and casual gaming on pubs. It is a complete joke because of the staggering difference in how different ISP's behave and process UDP packets and what the consequences of that happening are. The difference can be so big, that on a bad internet when you peek or you are getting peeked on, then you die instantly without even having a chance of reacting, aiming and shooting. Or maybe you manage to shoot (first or late) and nothing is getting registered. With a good internet there is a complete reversal, which means that once everything is fixed (or performs much better in terms of enemy model positioning) the exact same player who was killing you, now has literally no chance to react and shoot at you ! And this can be achieved while going from bad internet with ~10 ping to good internet with ping ~45-60 which is a ridiculous to even suggest that as a possibility, and yet this is exactly what happens. When you keep playing again and again, against the same people, you tend to know them in terms of their skills, meaning reflex, aiming, movement etc. When i played using my Fibre 1Gb/s i often encountered situations (getting killed) which i would describe as impossible to happen, which would not occur even once in a million if i would be having good internet or playing on LAN. When you actually see this issues improving drastically and some of them getting fixed completely with the use of VPN, then you start to understand how big of a mess the entire online gaming is. This thing trumps any & all hardware, system and peripheral modifications (all these things combined) that are supposed to "improve" gaming or reduce latency. These modifications can only be useful (and make difference), either when playing on LAN or when you are not significantly affected by this networking issue or when you have a near LAN-like experience when playing online.

PS: I was testing Gaming VPNs before, years ago, but i never tested them in a sufficient manner (silly me...) as i was expecting a starking difference right of the bat. And back then i was still clinging to the idea that lowest ping the better, so i actually never tried most VPN servers with higher pings. To anyone who is reading, if you are luckly with your region and place you live in, you might find a VPN provider with large number of local servers to choose from. That might finally address all the internet problems that you have been having. Try multiple VPN providers if needed, as you will basically get access to a large number of ISP's that way.

The problem is you are not taking into account ping compensation if you go from 10ms without VPN to 40-50ms with VPN.
To rule this out you would need one more test + add latency without VPN and test this vs VPN with same end latency.

dervu wrote: ↑

31 Jan 2021, 10:33

The problem is you are not taking into account ping compensation if you go from 10ms without VPN to 40-50ms with VPN.
To rule this out you would need one more test + add latency without VPN and test this vs VPN with same end latency.

I see plenty of players not having these issues, regardless of their ping, so it is not a ping compensation algorithm. Also, not every foreign VPN server that i have tried (pings 40-60) behaves the same or improves these problems. I see it in both games that i'm currently testing on (CS 1.6 and Apex Legends). What i need to do now is to find local VPN servers that will allow me to still have pings around 10-20 and see if i can find one that improves these issues drastically.

dervu wrote: ↑

31 Jan 2021, 10:33

The problem is you are not taking into account ping compensation if you go from 10ms without VPN to 40-50ms with VPN. To rule this out you would need one more test + add latency without VPN and test this vs VPN with same end latency.

Not all games have ping compensation algorithms affected by this -- but ping compensation algorithms need to be tested in more detail. This could be good stuff for Battle(non)sense to take upon.

One can test a network conditioner app (add intentional latency in both directions) if you need to intentionally increase FTTH latency to be equal to your other players.

The FTTH users who report better games playing instead of LTE connections, should also test network conditioning apps or build interim boxes (or even build a custom pfsense box with an adjustable-lag adjustment). One advantage many fiber users (FTTH) has is ultralow ping jitter, so your wonderful 8ms lag can be essentially temporarily reprogrammed to a near-perfect 30ms lag (+/- 0.5ms jitter!), to hit the sweet spot of specific gaming servers (to bypass "LPB-handicapping" algorithms applied to you)

Also -- Not all VPNs add latency. There are some of us getting 30-40ms LESS latency WITH the VPN. It's because of better routes to the gaming server than the original ISP. It's ISP-dependant, location-dependant, and VPN-dependant, so you have to roll the lag dice trying multiple VPNs.

I've used VPN's for about 6 years now for routing reasons. Obviously for most people it won't make a difference, or make things potentially worse.
Ping for some of the games I played wasn't a huge issue for me, 250ms was playable but anything higher wasn't enjoyable.
I've used various over the years testing Nord, Express, Vypr, AirVPN, Mullvad, Windscribe, WTFast, ExitLag and Mudfish.

Here are my reviews;

WTFast and Exitlag were pretty much useless and half the time didn't even work, I spent most of my time talking to tech support. They don't support community servers on various games, only dedicated. These were also the most costly with almost zero control and you know... Not actually being a VPN.

Nord - While I can't say for all servers, the ones I tried weren't entirely stable. I went through at least 100. Has an insanely large amount of servers to choose from though.

Express - Quite pricey but does a good job and an easy to use program. No complaints

Vypr - Not 100% stable pingtimes but good overall. Would get the occasional dropped packet. - Granted this was 3 years ago so things might have gotten better since then.

AirVPN - A solid choice. Never had an issue.

Mullvad - Another solid choice. A shame they don't have promotions as this was the priciest one of the bunch. Though I did find I got banned from a community server for having a matching IP address which I found funny.

Windscribe - Avoid for gaming. Good price though.

Mudfish - An abundance of servers but a bit finicky for new users. Pay for data that doesn't expire (which is really nice if you don't need a VPN subscription and once in a blue moon your routing goes to shit). Gets me the lowest ping to any server I want.

Hopefully this helped. Mudfish is a must and if you need a dedicated one, Mullvad is a solid choice. Cost wise AirVPN or Express deals can get you good results too.

More on the out of order packets issue, which looks more and more like the root cause behind everything and ISP's network resilience, devices & configuration being the main culprit behind poor UDP packet handling:

• "Out of Order Packets" thread @ reddit

...
Essentially this is packet loss, even though the game does not alert the user of the issue (packet loss indicator) and there is no way to diagnose this problem as of right now. Packet loss or extensive out of order packets would result in degraded simulation accuracy and make it very difficult for people to predict where people are going, register shots, and otherwise participate in a real time fashion. This doesn't just effect Fortnite, it effects all games and how extensive depends on the engine and the netcode. I did not know this was a huge issue, but apparently it is.

Currently there does not seem to be any sort of ways of diagnosing this.
...

• "Valid reasons for out-of-order packets?" thread @ reddit

• "Why Can Some Advanced Ethernet NICs Cause Packet Reordering?"

• "Sorting Reordered Packets with Interrupt Coalescing"

• "Turtles on the wire: understanding how the OS uses the modern NIC"