Click To Photon/Nvidia Reflex: Shortcomings You Should Be Aware Of

[MaxTendency]

First of all I'd like to say that I am extremely happy with the fact that the industry seems to be moving toward latency reduction with Nvidia Reflex, 360hz monitors, 8k hz mouse etc. With reflex analyser monitors people can now measure their click to photon delay with easy, which is good because it allows people to actually measure the effect of their optimization/tweak instead of just having to rely on feeling.

However, there's a huge problem with this method of measurement.

Click to photon doesn't CONTINUOUSLY measure latency, it it only capable of measuring the latency of a certain event (eg, click.) This is a problem because latency is not stable, latency spikes exists. Which may go COMPLETELY unnoticed by this method of testing. You can slave alot of trials (like 100+) to minimize this, but its still entirely possible that all latency spikes would dodge the capture.



In the left the the latency is quite stable while on the right the latency is spiking, which will have a very bad impact on players experience. However since we cannot measure continuous latency its entirely possible that the "clicks" will miss the spikes. This will result in both of the tests above showing an avg of somewhere around 15ms, which would indicate that both of these scenarios have similar latency. However that's not true at all.

This is somewhat similar to the problem with avg fps. You could have massive stutters making the game downright unplayable however the avg fps would remain high. This is why including 0.1% lows are a great improvement, although a pure frametime graph is still superior. However fps capture tools can still capture the spikes, with click to photon testing 0.1% wouldn't make sense since as demonstrated above, the spikes may completely dodge the capture.

What needs to be done:

In order to solve this problem we need to be able to measure the latency continuously. With Nvidia Reflex we can automatically measure the latency of each click, however that's not enough like I explained above. We need to be able to measure the latency of each poll instead. This would greatly reduce if not eliminate the possibility of latency spikes going under the radar and would show results that are more accurate to the player's experience.

So until that's possible we need to be aware of the shortcomings of click to photon testing when measuring latency ourselves or looking at measurements done by others.


__________ __________ __________



As a sidenote, if this type of measurement becomes a reality this could massively improve the accuracy of research done on latency vs aiming. Since you will be not only be able to measure the player's accuracy on a per click/shot basis, you will also be able to measure the impact of latency variance as well.



For example with just click to photon (Nvidia Reflex for example) both figure A and B would show similar latency. But if we can measure continuous latency we would be able to see the latency spikes and quantify its affect on aiming far more precisely and on a per shot basis.

[Chief Blur Buster]

Yes! Latency consistency is also an additional important metric.

(IMHO for the topic title -- click to photon isn't necessarily a flawed latency measurement -- just simply incomplete, as all lag tests are necessarily incomplete. The correct way to phrase it is "Click To Photon should NOT be the ONLY latency measurement methodology" and describe the limtiations of current lag measurement tests, etc.)

All the system jitter that varies, is variable latency that can happy at any point in the chain, from system microfreezes to dropped polls, driver jitter, power management jitter, game engine performance jitter, etc.

Sometimes more important (for some contexts) than button-to-photons latency, is gametime-to-photon latency consistency -- this is a capital mandatory requirement of ultrasilk operation (e.g. virtual reality). The world's smoothest game motion prioritizes gametime-to-photon latency above all else (as well as head tracking sync), even more important than a few milliseconds random variance in click-to-photon latency.

Gametime-to-photon latency fixes was how I helped the the developer of the Cloudpunk game solve stutter issues, and made the game G-SYNC/FreeSync compatible. Now, clicktime and gametime are pretty much usually in sync, but gametime jitters will jitter the button latency. So a game developer targetting gametime precision can do a huge step towards correct consistent latency, and the rest (e.g. high pollrates) will automatically fall in place if the drivers are well-written and the game engine is reasonably efficient in handling the polls, etc.

Prioritizing latency optimization is quite difficult; but this is the easiest low-lying apple target for a software developer of a game to "future proof their game" to all future refresh rates & sync technologies.

[Meowchan]

One thing I like about Fortnight's implementation of NVidia Reflex is that they added the optional white square on the side of the screen for each mouse click to help with latency testing. If we take this a step further, the game or app itself can have an implementation of some established benchmark of either clicks or motion or both, and on the other end the mouse (or even monitor in the case of the new 360 hz monitors with their built in mouse hookup measurement) can perform a series of actions that can be programmable via software giving us a continuous latency measurement.

For example: user goes into his mouse software and programs an auto script left 30 dots + click, and on next (second divided by polling rate) right 30 dots + click. The game / app has a way to track these actions and display them on a latency graph. This captures most of the chain only skipping the mouse + polling. If we want to capture the mouse / usb aspect as well we need the same solution, only with hardware support which gets pricier and more rigid. In general I am quite happy measuring the system itself without the ends. Just like the 360 hz monitors fail to capture themselves properly because they are only aware of when the buffers change to display the muzzle flash, but they are not capable of adequately measuring their own pixel transitions (at least as far as I'm aware). If you want to test a monitor don't go to a monitor, and if you want to test a mouse don't go to a mouse. The OS / game however are quite good at measuring themselves.

[MaxTendency]

Tweet thread from Josef, researcher at nvidia, explaining why even 2000 samples (click to photon) may still not be enough to draw conclusions.

"While 200 samples sounds like a lot, I highly recommend going to at least 2000 to get a better distribution of latencies."

"Early on when I took 1000-2000 samples (automated clicking), I still found significant sample correlation from the automation bias. If a person clicks individually the 1000-2000 times, then you might get good enough results, but at 20-200 clicks, there's still too much error IMO."
-Source

While click to photon might reveal big changes quite easily (going from 144hz to 240hz for example), its still not really good enough for detecting smaller changes even with a lot of samples. I believe this is worth sharing because with newer monitors equipped with reflex analyser a lot of people are gonna be able to measure their system latency, so its important to be aware of the short comings of the method being used and be aware of its flaws.

[jorimt]

Tweet thread from Josef, researcher at nvidia, explaining why even 2000 samples (click to photon) may still not be enough to draw conclusions.

[...]

While click to photon might reveal big changes quite easily (going from 144hz to 240hz for example), its still not really good enough for detecting smaller changes even with a lot of samples. I believe this is worth sharing because with newer monitors equipped with reflex analyser a lot of people are gonna be able to measure their system latency, so its important to be aware of the short comings of the method being used and be aware of its flaws.

In a certain context, correct, but a smaller sample rate still gives you adequate data for determining the min/max/avg button-to-pixel latency. E.g. it establishes the range of latency on the given setup you're testing.

While this may not fully reveal the distribution of said min/max/avg over a longer period of time, it still allows you to determine, say, optimal user-configurable settings, such as V-SYNC, pre-rendered frames, GPU limit graphical settings, etc, but may or may not be as helpful for detecting smaller variances caused by RAM timings, system timers, OS bloat/processes, etc.

What your thread seems to be getting at, is 20-200 samples vs 2000+ samples with button-to-pixel testing methods is like framerate vs. frametime graphs; one gives you a basic average, the other gives you moment to moment. It just really depends on what you're trying to test for.

That said, I do agree the more samples the better (though this isn't always possible, or even always relevant to certain test scenario goals; diminishing returns), and I also agree that once Reflex testing is available mainstream, result interpretation and determination will be a potential mess among novices.