ESPORTS: Latency Perception, Temporal Ventriloquism & Horizon of Simultaneity

[diakou]

So I saw this post of Chief responding to someone over at I have the new Razer 8000 Hz prototype gaming mouse on my desk. and instead of posting a reply to him there, I decided to maybe just open the discussion on it even though I am not ready to present it clearly with proper formatting & more. Instead, I want to think of this as a crowd-sourcing/open discussion moment where there can be criticism, feedback and discussion on the topic in regards to Latency Perception, its impacts on Pro Players (and casuals) and how things are even more "insane" than we originally thought... due to our incredible bodies.

Here's a Secret...
Nobody knows they want an 8000 hz mouse. Yet.
It will hit the esports sphere first. But when 360 Hz monitors are cheaper in 2025, the 8000 Hz mouse will be a coveted accessory by then for users picky about smoothest motion. Getting money's worth out of 360Hz

That has been how it has always worked. I genuinely believe most people either consciously or subconsciously try to "deny the truth" on the basis of not ruining their own casual or competitive mentality by saying that "these things do not matter or are negligible". Or, they genuinely can not actually notice the difference due to something I'll come back to later down this post (Latency Perception)

I know not a single gamer out of the ~1000 gamers I have met, that would be fine with going back to a 60Hz monitor after their 120/144, let alone 240Hz. Yet it was only a few years ago where there was still arguments for how the impact and/or effect is not worth the effort(in this case, the price.)

With a sample size of 30 in the most unscientific way, I have also done an interesting test where the results has shown me:

If a player can notice 120Hz to 240Hz (or 60 to 120)
They are never able recalibrate to a lower standard at peak/close to peak performance - their performance (can) be semi-permanently affected in this scenario.
If a player however can NOT notice the instant effect of 60 to 120Hz or 120 to 240Hz, they are able to notice the change if they play on the new standard for a week then swap to lower Hz, however they are still able to recalibrate again to the lower Hz without being significantly affected.

However, (from limited unscientific sample) There has not been a single person to be capable of NOT noticing, as both groups has had the capability to notice/perceive changes if you gave them at least a week and then "downclocked" them.

Nowadays the argument stopped being "it's not a big difference" to "it's a negligible difference"

I.E 240hz to 360hz. From personal testing, Nvidia reported a raw 4% performance increase.


Here's the punch line; Sports research teams / performance specialists both legally and illegally has for YEARS and YEARS spent MILLIONS and MILLIONS for performance increases for game deciding differences of 0.3-1.5%

It is a bit frustrating to go in circles about this topic, as if there is no past documentation and history. But if there is one thing to learn from history, it is that we are very efficient at conveniently forgetting what is available to us and what we know from the past.

Lastly, a disclaimer: There are many, many variables to turn and improve before looking at top-end hardware and equipment/gear as the "problem". But it has always been baffling to ignore this once you are at a level where the investment cost easily justifies the reward. What is ~0.01% of an Esports Athletes salary for 4% performance increase? I honestly blame the top competitors more for not speaking out, for not raising awareness or for simply having poor understanding or bad(maybe it's good in this case?) latency perception. The environment and direction has always changed very fast once "influencers" and the like starts speaking about or using things. I do somewhat believe in a conspiracy that the current climate of Esports pros are people with relatively high "genetical" baseline thresholds of latency tolerance due to...Temporal ventriloquism and the Horizon of simultaneity.

I want to kick the door open very soon on a truly big and fascinating thing that I have been constantly thinking about the last year and a half, but I think it requires more context, proper formatting, strong research and deeper association for the "common folk" and a different variant for the scientifically educated, for it to have strong impact. However I'll try to leave as many breadcrumbs for people and create an open discussion around it and possibly inspire someone much more qualified than myself to research this.

Latency Perception, Temporal Ventriloquism & Horizon of Simultaneity in Esports

The absolute first article that opened my eyes to this (and is so far the only article to touch upon this subject for Gaming and Esports competition) was by a person named Eryk Banatt (Twitter here) who has a B.A in Cognitive Science. The article is called Delays Have Dangerous Ends: On Monitors in SSBM
This article was sparked due to the Super Smash Bros Melee community wanting to use monitors for accessibility and logistics reasons instead of their long-time precious CRTs. There was a lot of arguments about the monitors being "lagless" and "imperceiveable" from a CRT, thus the negligible performance difference did "not matter." Banatt tried to reason with the community about why this thinking is not as straight forward as it seems not only because of how impactful it can be to competitive performance differences. But also Latency Perception itself is a variable! (A few cannot speak for everyone and everyone cannot speak for the few.)

From this article, there are two paragraphs that are very interesting to explain a lot of the craziness at hand in our tasks here:

phpBB [video]

Eryk essentially hypothesizes that the same re-stitching effect we get from audio and visuals at different times are being synced, is the same concept to understand why lag is perceived differently by people. If a person is within the range of horizon of simultaneity (example numbers, 50-200ms) such as when a person speaks from a distance, the actual audio versus the visual aspect of the person moving their mouth are at two completely different intervals, yet we perceive it as real-time and happening at once due to "temporal ventriloquism". That is what the Horizon of simultaneity entails, as long as the audio/visuals infront of a person is within the limits/ranges of their own horizon of simultaneity, their brain will magically "sync" the audio and visual.
Now Eryk suggests that this concept can be further applied to inputlag (Tactile, Haptic / Visual / Audio) and why players notice it differently. Because there's a certain lag threshold needed for each player until they stop noticing that there is lag in whatever task they may be doing, regardless of if there actually STILL IS LAG. This is a pretty big a-ha moment, because it is the only possible way where someone successfully explained why there is a difference, without it being deemed a wild-goose hunt or mocking a person for making "excuses" for their performance at a LAN or Online event (even if that is the case!)

Imagine how many times you have heard of stories of a person being an "onliner" and can't perform at "LAN/Offline events" That they simply choke or can't handle the stress and pressure so their performance is worse, they need the comfort of their own home. The star AWPer who just could not perform when it mattered at offline events. Here's the deal, there will always be a case for this(people do not always have a consistent mental game or performance with high stakes) but I highly, highly suspect that there are more sinister causes for this. Often the players most likely have had setups that are slightly better than at the competition itself, especially in the past years. However, through my career as a person in the Esports industry (I am not special, I just have been fortunate enough to have a lot of contacts and been able to visit and live experience a lot of things for myself.) I have come to realize that the optimization level of pro players are very, very poor. They are not a strong empirical case for what is good and bad even if their skillset is supposed to be reflecting of this. Along with this, competitions themselves are often poorly optimized and consistently has ran with a "make it work for the average pro as a minimum" (yet the average pro does not completely know what is good or bad, just if THEY are feeling bad or good in the very specific moment.) I went through this in a post on here, you may go read it if you want; My experience in Esports as a bystander and competitor

There's many factors that can also make someone completely push through latency. One of them is by being completely engulfed in the moment where you force the game, your motions and everything including it to completely sync up aka; "the zone/flow state" The problem with this however is that to enter the zone or flow state, it has been hypothesized that you require a slight challenge (but not too much) while being 100% completely comfortable with your own movement/inputs/aiming without conscious thought on these aspects specifically(thinking about how you want to move, rather than just moving.) Our earlier talk about latency should essentially make that last task requirement, impossibly hard due to constantly fighting the perceived latency threshold in being able to "feel like you are one with the game." to enter flow state.

So to go back to our earlier point, this person that we have previously chalked up to being a choker essentially has now not been able to distinguish why he consistently keeps performing poorly at LAN even though he has never felt performance anxiety related symptoms in his life. One of his aspirations to even becoming good at games and participating in Esports was so that he could be on the stage more often and show people his ability! He will go back home, ridiculed. He cannot speak up, because it is simply an "excuse." He will work on himself on what the perceived problems are according to media or coaches that also perceive it to be the same. He will go back to the stage and he will try again. This time, either he has been able to calibrate to slightly worse LAN environments (through ever so slightly lowering the latency threshold for feeling latency by adaptation or calibration, by accident or as an incident.) Or he will yet again fail in the exact same way and be chalked down to not being able to perform on stage.
This has happened to me personally. This has most likely happened to several Pros in the history of Esports. However the problem is incredibly delicate, as we are in a new generation culture where we have not figured out many problems of our scenes at all, yet we constantly behave and make decisions for others as we have. Like I said, these players who are affected are not able to speak up about real problems without it being called an excuse due to the nature of competition. Even in the cases of something being an excuse, it does not necessarily mean it is not a problem!

I hope this poorly formatted and sloppy explanation that does not do Eryk justice, will however still just be enough to create a discussion around this as a reference point for the future. I would appreciate any and all comments on this as at the end of the day, I am simply just a player who wants to make the entire competitive scene in all Esports genres the best there can be for one sole reason; that everyone will be able to display their true ability/skill which in turn gives everyone else a better viewing experience and entertainment.

Lastly I will finish off with some aspects that can influence lag to push people under or above latency thresholds along with a ton of study excerpts with their respective links that both prove and disprove a few of my statements. Keep in mind that they are majorly not designed around low latency situations / for gaming purposes, which makes them faulty by nature for this.

Regarding the list itself, these are absurd amount of things to account for. I am not a person who chases unreasonable and not realistic scenarios however I do believe we have pushed that line a bit too far towards "not possible" when a lot is possible to alleviate. I personally propose a goal of attempting to lower latency and responsiveness variance to a latency threshold where the overwhelming majority will not be able to be strongly impacted by it. This would mean that even in the presence of lag, it is low enough to calibrate and push past it thanks to our incredible human brains. The goal is to simply get to the threshold where we are able to sync asynchronous actions for the overwhelming majority of players, casuals or competitors alike. The only aspect of this that is genuinely impossible is to try to do it alone or not start somewhere. I have been able to do it in my own community and game (Brawlhalla) but that is as far as I can personally change the landscape. I think 2020 and 2021 will show to be great times to finally talk about the "dark horse" or more "sinister but controllable aspects of Esports competition" and it seems like the market has been steering itself towards latency as well.

This list is not only for raw latency, but anything that can affect latency perception itself. It's pretty much just everything I can think off plus some inspiration from /R/Gamingoptimisations
Framepacing/Game Code/Game Optimization
High DPI vs low DPI (smoothing, no smoothing)
High Polling Rates vs Low Polling Rates (250hz vs 1000hz vs 8000hz vs XXXXX)
EMI / Power interference
RGB induced lag (heat & possible overhead)
Network lag, Bufferbloat, Peering, Routes, Packetloss etc
Refresh rates (60hz vs 120hz vs 240hz vs 360hz vs XXXhz
Monitor differences (Response times, Signal Processing, Cable used etc)
Computer Setup/OS Differences
Different "tweaks" and "optimizations" on software, hardware and OS
Computer Hardware
Cooling (as temperature fluctuations depending on the ranges can catalyst and cause variable lag)
Software
Computer Peripherals
Surface(Mouse pad)/Mouse feet/Mouse comfort/Cable/Sensor
Chair / Chair position
Sleep & Physical Performance State (glasses, sound)
Body Performance through Nutrients & Diet
Environment (Humidity, Lights, Stage, Noise, Oxygen levels/CO2)
Psychological/Mental state & Mentality (Meditation, Calibration attempts, Music {different genres and beats})

Disclaimer; a lot of the mentioned study links are not direct approvals nor disproval of the thread, they are merely interesting links between the concepts discussed regarding latency & latency perception. There are certain things to recognize in the temporal aspects of a human body & the possible links they have for Esports and latency.

NVIDIA's Main Research Publicization & Blog post on their studies regarding Latency & its effects

NVIDIA's blog post on reflex, their new innovation for latency, references & utilizes aforementioned research

(New post on Esports & Excercise, supplementary info)
Converging Evidence Supporting the Cognitive Link between Exercise and Esport Performance: A Dual Systematic Review


------------------------------

Realigning Thunder and Lightning: Temporal Adaptation to Spatiotemporally Distant Events


Perceptual association enhances intersensory temporal precision


Rapid recalibration to audiovisual asynchrony follows the physical—not the perceived—temporal order


Contextualizing the clock(s) : integrating cues for the perception of time and timing an action


Audiovisual temporal recalibration occurs independently at two different time scales


Selective Attention Modulates the Direction of Audio-Visual Temporal Recalibration


Smoothness perception: Investigation of beat rate effect on frame rate perception


Perceptual Limits of Visual-Haptic Simultaneity in Virtual Reality Interactions




Grouping and Segregation of Sensory Events by Actions in Temporal Audio-Visual Recalibration


Sensitivity to Audiovisual Temporal Asynchrony in Children With a History of Specific Language Impairment and Their Peers With Typical Development: A Replication and Follow-Up Study


What Causes Specific Language Impairment in Children?

[diakou]

Reserved spot for future updates

[deama]

On the topic of "the zone", here's a nice video where the guy explains it as well as explains a neat way to "get into the zone".

phpBB [video]

I've myself tried a tdcs device, it's not magic in that you instantly can go into "the zone", but it noticeably helps, I did however use more of a hobbyist device, with maybe a better device, and a better map of my brain, I could better pin-point where to stick the electric nodes.

[Chief Blur Buster]

I still need to fully digest your information; all a useful read; but I wanted to also pre-emptively remind other readers that Blur Busters isn't just about only latency alone -- there are many factors far beyond latency. Technical threads like these tend to get lots of IP addresses from universities, so...

A Note to Budding Researchers / Students
Since this research-heavy thread might be a jumping-off point for researchers / university students looking for topics for future papers (since Blur Busters / TestUFO is cited in more than two dozen research papers). And some researchers may think that latency is the reason for refresh rate to stop. So, I have to play my classic informative role here -- I still do lots of descendants of "Humans Can't See 30fps vs 60fps" mythbusting and educating -- so without further ado:

I wanted to add some links because readers diving into this advanced subject also need to be aware of additional factors (different reaction speeds to different stimuli, different benefits of the milliseconds other than latency -- such as stroboscopics and motion blur). It is common to miss the forest for the trees.

Researchers do not test for all variables, so the "1% performance improvement" may be unimportant to some of us, but something else is more important (such as motion quality):

Non-Gaming uses of 240 Hz and 360 Hz
Do you have a 120Hz iPad? Same reason -- many non-gaming readers on this website here also use 144Hz and 240Hz for motion-blur ergonomic reasons -- non-gamers who want to have web browser scrolling that is 4x clearer than 60Hz. A 240Hz monitor is twice as clear in scrolling than a 120Hz iPad, which is another non-esports benefit. A 360 Hz monitor has 1/6th the browser scrolling motionblur of a 60Hz LCD, getting close to CRT clarity without the need for strobing. With IPS taking over 240Hz and 360Hz markets, the ergonomic-use factor has recently boomed too, since they no longer have the color/viewing angle/etc compromises typically historically associated with TN.

Although a bit premium priced, 240Hz and 360Hz just isn't for esports anymore, much like 4K was a $10,000 luxury in 2001 (who needs 4K? It's beyond human resolving capability!), but is now a $299 Walmart special. Later this decade, 120Hz increasingly becomes a freebie feature as more and more TVs/smartphones include it. That too, will happen to 240Hz, too as well until we've retina'd out refresh rates for free, much like retina resolution is no longer a noticeable cost-add in many industry segments.

Likewise, the dual-use (tiny performance improvement + more noticable comfort improvement), can appeal to gamers who want to enjoy gaming more without regards to performance. I count myself in this category -- I love the enjoyment of reduced motion blur of high refresh rates.

It is really too bad 4K and 8K has less resolution than a 1080p display for moving images. Stationary images, 8K is gorgeous. But as soon as everything moves, the high-Hz 1080p or 1440p displays blow the 4K and 8K displays out of the water. Like some are picky about colors, others picky about tearing, some of us are picky about display motion blur for general-purpose enjoyment, without regards for latency.

Refresh Race Race Upgrade Differentials Are Often Too Small
I also add that 240Hz->360Hz is smaller than the usual mainstream Blur Busters recommendation of "Double Your Refresh Rate" upgrade recommendation. The natural upgrade for 144Hz is 280Hz or 360Hz, and the natural upgrade for 240Hz should really be 480Hz. One has to go geometrically up the diminishing curve of returns when upgrading refresh rates. This is not too different from the upgrade path of resolution. One has to double Hz to noticeably halve display motion blur. Small upgrades (144Hz->165Hz, 240Hz->280Hz) are way too incremental.

The refresh rate race will be fairly slow, like the NTSC to 8K journey. For a long time, we didn't need to go above 60Hz to fix motion blur, because CRTs naturally strobed (no motion blur). Now ever since we went flickerless sample-and-hold, we have to raise refresh rate to regain CRT motion clarity. 8K 1000Hz might not be an RCA Walmart special for many years yet, but there is already an engineering path to commoditization of high-Hz for ergonomic benefits, over the next several decades -- high Hz isn't necessarily expensive forever.

Until all retina benefits are milked (much like 8K or 16K is a dead end resolution-wise for household direct-view screens). Right now, we're still far away from retina refresh rates, since CRT phosphor decay dictated motion blur; and most CRT phosphor decayed at less than 1ms. To emulate that, 0.5ms requires 2000fps@2000Hz sample-and-hold for the same amount of motion blur. Achieving blurless+flickerless simultaneously is a tough engineering problem that is the rocket fuel that propels the refresh rate race for the next few decades.

So to keep things complete, adding this:

Related Blur Busters Topics

• ARTICLE: Input Lag and the Limits of Human Reflex, a 4-part series commissioned by Blur Busters about reaction times to a different kind of stimuli (including audio/movement)
• FORUMS: The Amazing Human Visible Feats Of The Millisecond (including milliseconds useful in non-esports contexts too, such as reducing web browser scrolling motion blur).
• ARTICLE: The Stroboscopic Effect of Finite Framerate Displays, which helps partially explain why displays can't yet pass the blind test of a "display" versus "looking out of a window"; this is useful for reality tests, where displays have to emulate real life more and more accurately. Those trying to build a VR headset that can pass a Holodeck Turing test (can't tell apart reality from VR, in blind test ski-goggles versus VR headset), need to understand all display weak links.
• ARTICLE: Blur Busters Law: The Amazing Journey To Future 1000 Hz Displays, which helps cover multiple angles of the benefits of higher refresh rates in a non-latency context. Most importantly, see the Vicious Cycle Effect section, where something may be unimportant until the specs are high enough (e.g. increased resolution amplifying the limitations of refresh rates).

Being Blur Busters is all about all sorts of temporals (Hz,fps,GtG,MPRT,lag,polls,frametime,etc), I cast a large net on describing the benefits of high-Hz. In other words, a good researcher correctly has to ask "What weak links remain, when latency is not important?", at least in an acknowledgements section of their paper -- rather than concluding that something has no benefits at all, because there is no lag benefits.

TL;DR: In the diminishing curve of returns, it is true that latency benefits may diminish sooner than other benefits of high-Hz -- but many other benefits can remain (some much more important than lag for some of us). Researchers need to be aware and acknowledge this to ensure that when they're focussing on testing lag-related stuff, that it does not preclude other benefits.

Latency is important. But I must remind readers & researchers that latency is not the only reason of the refresh rate race.

[diakou]

I still need to fully digest your information; all a useful read; but I wanted to also pre-emptively remind other readers that Blur Busters isn't just about only latency alone -- there are many factors far beyond latency. Technical threads like these tend to get lots of IP addresses from universities, so...

Being Blur Busters is all about all sorts of temporals (Hz,fps,GtG,MPRT,lag,polls,frametime,etc), I cast a large net on describing the benefits of high-Hz. In other words, a good researcher correctly has to ask "What weak links remain, when latency is not important?", at least in an acknowledgements section of their paper -- rather than concluding that something has no benefits at all, because there is no lag benefits.

TL;DR: In the diminishing curve of returns, it is true that latency benefits may diminish sooner than other benefits of high-Hz -- but many other benefits can remain (some much more important than lag for some of us). Researchers need to be aware and acknowledge this to ensure that when they're focussing on testing lag-related stuff, that it does not preclude other benefits.

Latency is important. But I must remind readers & researchers that latency is not the only reason of the refresh rate race.

Thank you for chiming in on that! Yes, truly blurbusters is not JUST a latency specified monitor forum at all. That is simply just one part of a many in the vicious cycle for the journey to 1000hz refresh rates (and beyond) There's so many other aspects that can cause problems similar to latency induced detriments, but show themselves as things that would quickly be dismissed / ignored and become a weak-link with an all latency lowering approach such as motion blur.

I apologize if this post came off like that (or for anyone else reading) I will spend a few hours on editing the main-post to make it look more neat and research-like rather than the exploration, ramblings and findings from a competitive player on the latency warpath the last 5 years.

I think one of the most important aspects about this entire thing is definitely Eryk's original suggestion. Because it is able to offer an alternate explanation on why latency perception is neither the person who can notice it or the person who cannot notice lag's fault. It simply is based off a few factors that have so far been hypothesized, but not confirmed as there are many variables. We however know that there is a latency threshold that both players would benefit under - regardless of not being able to notice the latency. So it puts the pressure away from someone either having a "mental barrier" or someone "not being good enough to notice lag" and instead meets both in the middle with an explanation (and somewhat guideline to solution) for both to be satisfied.

[Chief Blur Buster]

<!--UPDATE-->
Some thread distractions by other posters (including ME; whoops) formerly in this location were moved to separate posts:

For future VSYNC-is-obsolete posts, it has been moved to:
[Technical Talk] VSYNC is Obsolete: Future Framerateless Displays

For replies from member 'schizobeyondpills', it has been moved permanently to:
[Untitled Topic] Post Moved from ESPORTS: Latency Perception, Temporal Ventriloquism & Horizon of Simultaneity
Moderaton Note: Due to multiple complaints privately messaged to me, member 'schizobeyondpills' is no longer able to reply to any future threads where member 'diakou' is the original poster of.

Please carry on any ontopic discussion. Thanks!

<!--/UPDATE-->