Blur Busters Forums

Hey there!

My name is Andreas Schmid and I am a researcher for Human-Computer Interaction.

I would like to share an approach for measuring display reaction time using a device based on off-the-shelf components, which was developed as part of a bachelor's thesis in our research group.


We implemented the measuring method for display lag described in chapter 10.3 (Video Latency) of the Information Display Measuring Standard by SID:

• a probe with a photo diode is placed on the display under test
• a video source (in our case a Raspberry Pi) changes the color of the displayed image from black to white
• it triggers a microcontroller (STM8) to start a timer
• this microcontroller reads the photo diode's voltage until 50% brightness is reached (the threshold is determined beforehand in a calibration phase)
• the timer then stops and results are sent to the Raspberry Pi

This procedure is repeated automatically so distributions in display lag become visible.

An example of the data measured with the device on a Asus XG248Q can be seen in the following image (blue: no enhancements, orange: overdrive enabled, red: 50% brightness threshold):


As we used cheap (around 65€/$70 in total) and broadly available components, the device can be replicated rather easily (some knowledge in electronics and soldering is required for assembly though).
We plan on publishing the source code, circuitry and a guide for assembly in the near future.

More information on the project and more measurements can be found on our website: https://hci.ur.de/projects/display_latency

We are very interested in feedback by the Blur Busters community as you seem to know a lot about the ins and outs of display lag and different approaches to measuring it.

Excellent info!

Being a hobby-turned-business, these things are part of the Blur Busters dream.

We already have a similar internal product that is being used as part of the Blur Busters Approved Program.

Blur Busters, being inventors of display testing methods, is now considering options to commercialize a Blur Busters Branded version of a latency tester device vastly superior to existing lag testing devices.

This will help Blur Busters in these times, since our brand name and ability to create new industry standards, will be key.

For now (temporarily, in this COVID sensitive period) -- Blur Busters is refraining from publicly discussing some aspects until we publicly announce our first hardware device invention this year. We don't want to risk losing money on this hardware product -- we do not want to be a failed Kickstarter -- so we are very cautious in proceeding. But our tester now benefits a lot of internal Blur Busters projects already.

While the standard is good for basic 60Hz measurements, the latency measurement portion of Information Display Measuring Standard has some rather severe limitations that abberates/diverges from real-world and esports requirements, considering the complexity of latency chain, sync technologies (that are different than sync technologies built into a Leo Bodnar or Raspberry PI), and other pre-requisites that adds major error margins that makes it less useful to esports. It misses some major important lantecy aspects important to esports.

Around here, we have a post, The Amazing Human Visible Feats Of The Millisecond, that requires much stricter testing standard than the one Information Display has, and we are currently creating a better unified lag-measurment standard.

Different websites have different lag measuring methods. With my Refresh Rate Einstein thinking -- I have come up with a Unifying Lag Formula / Specification of sorts that explains why different sites have different lag numbers and why certain variables can make lag numbers closer to real-world esports (or more optimized for a specific situation such as gaming consoles, or for high-Hz monitors, etc). Now using our upcoming spec -- whereupon changing variables, can actually be adjusted to essentially replicate every lag test ever invented, used by all reviewers.

But want to hold off sharing our findings until Blur Busters has begun selling our branded hardware latency tester device. In my opinion, it's almost like a Grand Unifying Theory of Physics (but in a latency measurement point of view)

_____

Advanced Latency Measurement Standardization
(Better what xkcd thinks)

BTW, we are looking for co-writers for a white paper pertaining to an improved vastly-superior latency measurement standard -- I welcome you to email mark[at]blurbusters.com to discuss such collaborations. As you already have noticed, I've co-authored papers before, and Blur Busters is also cited in at least one NVIDIA paper already (Temporally Dense Ray Tracing).

My credentials:
- Pursuit Camera Paper, co-authored with NIST.gov, NOKIA, KelTek (am co-author)
- NVIDIA Temporally Dense Ray Tracing (am credited on page 2)
- Other Display Testing Inventions

I'm ready to participate in more papers. We believe the industry will eventually needs expert consultation from a person like Blur Busters to massively improve the Display Latency Measuring Standardization.

Thanks for your reply!

We are aware that the Raspberry Pi is no ideal platform as it only supports up to 60 Hz but we decided to prioritize replicability over supporting displays with higher frame rates for now.
I think a completely open measuring method, including circuitry and source code is key for a reliable measuring device. A proprietary device might introduce systematic errors and nobody would be able to know (looking at you, Leo Bodnar).

While the standard is good for basic 60Hz measurements, the latency measurement portion of Information Display Measuring Standard has some rather severe limitations that abberates/diverges from real-world and esports requirements

I don't fully understand this. The IDMS only specifies how to measure (photo sensor in the middle of the screen) and when the measurement begins and ends (image is sent until 50% brightness is reached). So this should be independent of enhancement technologies as long as all triggers are set up correctly, shouldn't it?

Regarding the complexity of the latency chain: We have come to the same conclusion that each partial latency (of input device, application framework and output device) should be regarded separately as too many factors contribute to end-to-end latency. We started out with measuring input devices, the paper can be found here in case you are interested.

We are very interested in sharing our insights with you and a possible collaboration, I will drop you an e-mail soon. Thank's for the offer!

AndreasSchmid wrote: ↑

15 May 2020, 14:55

I don't fully understand this. The IDMS only specifies how to measure (photo sensor in the middle of the screen) and when the measurement begins and ends (image is sent until 50% brightness is reached). So this should be independent of enhancement technologies as long as all triggers are set up correctly, shouldn't it?

For competitive reasons I would prefer to privately collaborate in the run-up to an announcement.

But needless to say, all scientists (including Ph.Ds) agree with me, once I've privately explained.

There is a situation where a display is superior for consoles but crap for PC, and an lag-inversion-situation where the display suddenly gets lower lag for consoles and higher lag for PCs. Yet, repeat the same test on a different display, both console/PC is low lag. A single latency number is unable to show that off. Human eyeballs are not single-pixel reaction-time-trigger photodiodes.

AndreasSchmid wrote: ↑

15 May 2020, 14:55

Regarding the complexity of the latency chain: We have come to the same conclusion that each partial latency (of input device, application framework and output device) should be regarded separately

Actually, that's old Newtonian thinking. We do Einstein thinking around here; because unfortunately, the latency gradient changes due to interactions between the GPU-side and display-side. And different pixels can have different lag (and even different (min,max) lag spreads per pixel too) because of multiple interactions that cannot be siloed. We have a better (and superior) formula. Even the webpage at www.blurbusters.com/scanout only tells part of the story -- there are far more complex interactions.

We are familiar with the interactions where it becomes necessary to measure the proper segment of latency chain ("driver-to-photons"), not just display-only, when the goal of latency numbers is to be comparable to real world gaming and human reaction times. We are experts in Present()-to-photons.

Even TOP > CENTER > BOTTOM can change to TOP < CENTER < BOTTOM can change to TOP = CENTER = BOTTOM on the same display based on different settings.

AndreasSchmid wrote: ↑

15 May 2020, 14:55

We are very interested in sharing our insights with you and a possible collaboration, I will drop you an e-mail soon. Thank's for the offer!

I'd love it!

AndreasSchmid wrote: ↑

15 May 2020, 14:55

I think a completely open measuring method, including circuitry and source code is key for a reliable measuring device. A proprietary device might introduce systematic errors and nobody would be able to know (looking at you, Leo Bodnar).

The commercial tester we have will have fully documented lag stopwatching, and eventually be able to plug-in into existing game engines.

Open source testers can verify the accuracy of commercial tester, and vice-versa. Just like colorimeters -- there's closed source (i1 DisplayPro) and open source (ColorHug) -- and both produce the same result.

Just like the color theory is already well-proven, we have a successful working temporal theory that can unify (almost as breakthrough as Einstein's formula, in our opinion). Our goal is to publish a Lag Unifying Formula paper (or similar), with some collaborators. Then all future lag testers (open source or commercial) can follow this. Maybe help us release this breakthrough latency paper?

There is no problem for open source testers to compete against my hoped-for commercialization of lag tester. Blur Busters was a hobby of mine, and it is my drive to also commercialize a tester. People who can't afford the Blur Busters tester can still build their own open source tester if they wish. But we still are looking to commercialize our tester around our open-source Lag Unifying Theory standard paper that we would like to release.

There are rather severe weaknesses with single-point screen-centre lag testing methodologies, when we're trying to benchmark human reaction times. Due to this, there is interactions between the GPU and the display that can invert / compress / uncompress latency gradients along the screen surface, as not all pixels refresh at the same time (we understand why this happens, for each situation and each case it happens, in real-world esports settings that are currently used), and human can react to other parts of the screen surface other than the dead-centre of the screen. As we are experts in Present() to Photons, we have no use for limiting us to single-pixel lag measurement standards at Blur Busters as it is a square peg in a round hole for our human-reaction-time needs in current real-world games using new monitor technologies that has close integrations between GPU and monitor that changes the lag behavior of each pixel.

Blur Busters, being a hobby turned business, and part of my dream is to help innovate by unifying the lag testing standardizations.

Plugging in the correct numbers to my lag formula replicates the IDMS standard, so my new Lag Unifying Formula is a superset of IDMS. Changing the variables make it replicate SMTT, then changing variables again makes it replicate Leo Bodnar, changing the variables again makes it replicate other testers, etc.

And IDMS numbers diverge from numbers that are more applicable to CS:GO esports games. We even sometimes see worse IDMS numbers for screens that have better numbers for testing method applicable to CS:GO esports, and we even sometimes see better IDMS numbers for screens that have worse numbers for testing method applicable to CS:GO esports.

We are Blur Busters -- we know this -- we know Present()-to-photons better than most academics -- and we know why the different input lag methods create different results -- and it's actually all successfully unifyable (a temporal standardization equivalent to colorimetry standardization). If you're familiar with "racing the beam", see One of our Tearline Jedi Demo Videos. Each color bar is equal in latency (it bypasses scanout latency, so it becomes TOP=CENTER=BOTTOM on most, albiet not all, displays) in that YouTube video, since lowest lag is the first pixel row right below a VSYNC OFF tearline.

Scholars and reputable companies, as well as SID.org members, who would like to collaborate on a Lag Testing Stadard paper -- please email me at mark [at] blurbusters.com ... We want to have something vastly superior to IDMS.

P.S. IMPORTANT: Please credit Mark Rejhon of Blur Busters for anything you write/learn/etc because of Blur Busters. Appreciated!

P.S. For future replies, please send to mark [at] blurbusters.com as I would like to continue this discussion in private if possible (for now until ready to announce). Thank you!

By the way, I've already flown cross both the Atlantic and Pacific oceans to visit display manufacturers to teach them about various display mechanics under Blur Busters topics (It's one of the Blur Busters services -- I'm considered a refresh rate Einstein even by some manufacturers). Even display engineers learn a lot from me -- consider that a rocket gimbal engineer may not understand the firmware language of a GPS ICU, and vice-versa -- we fill diplay knowledge that many didn't know.

The IDMS standard is excellent work (not to bash the work) -- it's just a woefully incomplete simplistic standard that diverges too frequently from real-world for our latency needs.

imo the only thing i care about is
how much slower is a display than a "perfect display", if both are receiving the same video signal (and hence running at same refresh rate and whatever).

the best way to measure would be with an oscilloscope, probing the video signal (or some signal inside the monitor), and a photodiode+amplifier also connected to the scope.

comparing against a crt isn't too bad either. crt's are a pretty good approximation of a perfect display. the issue is of course that crts cannot always receive the same signal as lcd/whatever displays.

Chief Blur Buster wrote: ↑

15 May 2020, 15:59

Just like the color theory is already well-proven, we have a successful working temporal theory that can unify (almost as breakthrough as Einstein's formula, in our opinion).

bruhhhh
the theory of input lag is literally adding numbers...

and there are multiple numbers to add up and different ways to add them up.

anyway my point is that comparing to anything by einstein is far fetched.

flood wrote: ↑

15 May 2020, 17:21

bruhhhh
the theory of input lag is literally adding numbers...

I think you misunderstand the intent of my post.

We even sometimes see worse IDMS numbers for screens that have better numbers for testing method applicable to CS:GO esports.

We even sometimes see better IDMS numbers for screens that have worse numbers for testing method applicable to CS:GO esports.

In addition, there are latency inversion behaviours because of multiple factors at play. Human eyes are not single-pixel sensors. Not all pixels refresh at the same time, and sometimes the bottom edge becomes visible before the top edge because of complex interactions GPU-and-monitor cooperation behaviors. Sometimes TOP > CENTER > BOTTOM becomes TOP < CENTER < BOTTOM becomes TOP = CENTER = BOTTOM on the same screen surface, and sometimes DIFFERENCE(TOP,BOTTOM) shrinks and enlarges, and sometimes there's weird out-of-order latency effects that sometimes happens like CENTER > BOTTOM > TOP and CENTER < BOTTOM < TOP. All on the same panel of the same model of the monitor, due to interactions between settings. We understand why all of this happen (latency gradient, latency gradient inversions, latency range compressions, latency range expansions, etc), and esports players react with peripheral vision too.

Don't forget to read The Amazing Human Visible Feats Of The Millisecond.

flood wrote: ↑

15 May 2020, 17:33

and there are multiple numbers to add up and different ways to add them up.

anyway my point is that comparing to anything by einstein is far fetched.

Maybe a bit of an exaggeration (ego), but:

Have you attended one of my training classes at a display manufacturer?

I bring high-Hz displays, TestUFO PowerPoints, high speed cameras filmed in real time in front of audience, and ways of demonstrating human-visible milliseconds right in front of a big audience already -- and paid for by multiple clients/vendors/manufacturers.

People who I taught training class to, have told me that I've been brilliant in explaining to them and have compared me to Einstein (for this specific topic) when I teach a classoroom of 20 people about refresh rates, latencies, strobing, sample-and-hold, pixel response, motion blur, MPRT, GtG, etc.

I'm able to explain this in a relatively easy manner with actual hands-on demos. Anyway, my point is that we know a lot more about latency than even most academics. I'm captured behaviors that most have overlooked.

If you don't believe I teach classes, one manufacturer gave me permission to feature this in an upcoming article featuring my work with them (That new article is coming out this month):



[Note: This image dates to March 2019, but I made several trips in 2019]

Coincidentially, the name of my classroom is called "Blur Busters Einstein Training" services, with a client testimonial of "Best use of consulting money!".

We don't currently advertise this classroom training service yet (mainly invitation only), except at services.blurbusters.com but I have privately done this several times already.

I (and Blur Busters) have been working behind-the-scenes for years on this. It's probably a good idea to let the paper be released first before jumping to judgements -- since it's now essentially one of the top Blur Busters Prime Directives lately, and raised priority in COVID. All testers will benefit from the research we're planning to release, including your (flood's) tester development too. You already know how creative some of the other Blur Busters inventions are.

We do know more about latency testing than the majority. We don't mass-test monitors ourselves, but Blur Busters trailblazes by temporal display testing inventions to dozens of display reviewers instead.

As you already have noticed, one of the goal of Blur Busters as an industry standardizer in the temporal sphere -- which is why I would like to collaborate on a paper with some academics on a superior unified lag testing standard.

Chief Blur Buster wrote: ↑

15 May 2020, 17:34

Maybe a bit of an exaggeration (ego), but:

as someone who's done physics for about 12 years now, let me just say that it is very much an exaggeration

I bring high-Hz displays, TestUFO PowerPoints, high speed cameras filmed in real time in front of audience, and ways of demonstrating human-visible milliseconds right in front of a big audience already -- and paid for by multiple clients/vendors/manufacturers.

People who I taught training class to, have told me that I've been brilliant in explaining to them and have compared me to Einstein (for this specific topic) when I teach a classoroom of 20 people about refresh rates, latencies, strobing, sample-and-hold, pixel response, motion blur, MPRT, GtG, etc. I'm able to explain this in a relatively easy manner. Anyway, my point is that we know a lot more about latency than even most academics. I'm captured behaviors that most have overlooked.

I have been working behind-the-scenes for years on this.

no one's doubting that you understand this stuff.
i happen to understand it as well

flood wrote: ↑

15 May 2020, 17:52

as someone who's done physics for about 12 years now, let me just say that it is very much an exaggeration

Yes, it is an exaggeration but it is not my words.

See photo above and testimonal above.

The fact is that there exists many audiences that have made the comparision.