Potential HD Mouse Extensions API Specification [ETA by 2025]

Talk to software developers and aspiring geeks. Programming tips. Improve motion fluidity. Reduce input lag. Come Present() yourself!
User avatar
Chief Blur Buster
Site Admin
Posts: 9059
Joined: 05 Dec 2013, 15:44
Location: Toronto / Hamilton, Ontario, Canada
Contact:

Re: Potential HD Mouse Extensions API Specification [ETA by 2025]

Post by Chief Blur Buster » 19 Dec 2020, 16:49

griffin1987 wrote:
18 Dec 2020, 07:48
Would there be a way to reverse everything to use push instead of poll? Like freesync/g-sync kinda does it for monitors.
This could also lead to power savings which could be relevant for mobile users. The mouse would just send movement or clicks when they happen, up to x hz.
Thanks to mandatory timestamps for all mouse coordinates (with a strong recommendation that timestamps be sensor-side) -- the proposed HD Mouse Extensions API doesn't care if it's push or poll or even asynchronous (analog-like).

I've intentionally made HD Mouse API proposal agnostic on poll/push -- and sensor-side timestamping will help benchmark whether or not push/poll is lagged or not (and force vendors to improve that).

HD Mouse API can use non-USB methods. Imagine connecting a Thunderbolt mouse with DMA permission granted, if it can be done in a secure way, with a sufficiently flexible cable! It's a bit overkill for majority, but that would be sort of the Holy Grail to run at direct dedicated PCI-X performance specs, if the cable is not TOO thick.

Another theoretical idea is GPIO based methods, like turning the mouse cable into a dedicated SPI bus to a special PCIe card in the computer. But we likely won't do that, because it's too specialized and would cost several hundred dollars per retail unit just to pay off the engineering/NRE costs. However, any hobbyist designer could theoretically try doing that, if they're familiar enough with creating a custom mouse driver that presents standard APIs (legacy mouse API and future HD Mouse API) to apps.

However, behind-the-scenes, USB is the standard for mice, and high-rate USB is better done as a poll method for lower lag, since push-based USB is laggier. Polls can be continuous (for stationary mice) or only intermittent (movement + button). The problem with intermittent mode is the lag before the first movement/press when a mouse is dormant. Because of behind-the-scene USB sheninigians such as power management and such. Which is why some mice have a force always-poll (continuous 1000 Hz even with stationary mice).

My view is we're stuck with USB for now, and HD Mouse API can help punch through USB performance issues with sensor-side timestamps and poll aggregation (e.g. 20 KHz raw mouse deltas on just 2KHz poll, at 10 deltas per poll). Overkill for 60 Hz days, but important for future retina refresh rates on ultrafast-GtG displays.

And you can still do some USB workarounds like performance tweaks + dedicated USB card + dedicated PCIe lane to CPU (and Razer just mentioned they're discussing with motherboard vendors about guaranteed-dedicated-chipped USB ports with zero contention with any other buses and PCIe lanes, etc).

Regardless, latency purists can skip that and do 1:1 operation with the best mouse connection available, while smoothness purists can reduce poll rate + add about 0.5ms tapedelay (for multiple-deltas-per-poll aggregation overheads) to get massively better mouse smoothness with a lower pollrate. The aim is to let the user choose, while still being plug-n-play.
Head of Blur Busters - BlurBusters.com | TestUFO.com | Follow @BlurBusters on Twitter

       To support Blur Busters:
       • Official List of Best Gaming Monitors
       • List of G-SYNC Monitors
       • List of FreeSync Monitors
       • List of Ultrawide Monitors

Brainlet
Posts: 70
Joined: 30 May 2020, 12:39
Contact:

Re: Potential HD Mouse Extensions API Specification [ETA by 2025]

Post by Brainlet » 26 Dec 2020, 13:53

Related to this topic, I also think game devs should adopt sensitivity settings like Diabotical did. When you click on "Mouse Details" you can type in your DPI and the desired cm / 360° (with multiple decimals as well) and it will automatically set the sensitivity multiplier to match it (I don't have the game installed and couldn't find a screenshot of that section).
Starting point for beginners: PC Optimization Hub

User avatar
Chief Blur Buster
Site Admin
Posts: 9059
Joined: 05 Dec 2013, 15:44
Location: Toronto / Hamilton, Ontario, Canada
Contact:

Re: Potential HD Mouse Extensions API Specification [ETA by 2025]

Post by Chief Blur Buster » 02 Jan 2021, 00:37

Brainlet wrote:
26 Dec 2020, 13:53
Related to this topic, I also think game devs should adopt sensitivity settings like Diabotical did. When you click on "Mouse Details" you can type in your DPI and the desired cm / 360° (with multiple decimals as well) and it will automatically set the sensitivity multiplier to match it (I don't have the game installed and couldn't find a screenshot of that section).
A possibly great crosspost for the HOWTO: Please Future Proof your Game Engine

___________________

On another topic, thought of: Also, poll rate and sensor rate is sometimes a brute-force hammer for the inability to synchronize polltime perfectly to gametime.

Now, the HD Mouse API is to have mandatory timestamps with all mouse deltas (with a strong recommendation that they be sensor-side timestamps). This means the mouse can send timestamps independently of any granularity or interval -- that is now possible with HD Mouse API.

Asynchronous / Analog-like = A mouse that sends deltas that is not time-aligned with a poll Hz. Since timestamps must be no worse than microsecond accurate where possible. Perhaps a time-resolution query is a good idea, so that the time resolution is known -- i.e. like a QueryPerformanceFrequency equivalent.

But the HD Mouse API could provide an additional API call where the game asks for a specific delta nearest a specific timestamp (or current time), and it returns that delta, if it is yet available, and if so, how old/long ago the nearest available delta was. If current time, the internal implementation of the mouse programming could actually do 2-way communication to read that sensor data right there and then, and return the delta -- perfectly synchronized to the game time.

Now, there are potential major problems with that approach:
- Variable USB lag (The Internet atomic time-sync problem, but on a local scale)
- Deltas not yet available (requesting too early = delta not available)
- Delta too old (requesting too late = adds lag)

Another internal implementation vector/curve data transmitted from the mouse sensor to the HD Mouse API, and the HD Mouse API would compute the appropriate delta (along the line of the movement vector/curve) from a timestamp request. So many internal driver workflows can be unlocked by a HD Mouse API internally -- this would be the internal responsibility of the mouse driver. As long as the timestamps-vs-delta are accurate, it could be a legitimate innovative workflow.

However, almost anything in this territory pretty much digitally sampled anyway, albiet at high frequencies, so sometimes the brute is much easier. What we need is to reduce the system load/requirements of such high rate of deltas.

In this sense, it is the lesser of evil to just have a compact USB packet of compressing 20 deltas into one poll (20KHz sensor read in 1KHz poll), all with their timestamps and relay it to the mouse driver containing HD Mouse API. The game engine would just cherrypick the closest timestamp. You would eliminate the need to monitor 20,000 mouse events per second, yet you easily cherrypick the 500 mouse deltas closest to your 500 gametimes for 500 frames per second on a 500Hz+ monitor.

Low workload; doesn't overload CPU or USB chips, you can still configure pollrate to your system's limit but it would not limit the precision of deltas (just latency of a lower poll rate -- but 2000Hz pollrate is only 0.125ms laggier than 4000Hz pollrate from a pure mathematical latency-difference calculation -- half of ((1/2000sec) - (1/4000sec)).

So I think a "delta nearest requested timestamp" API call, is likely a sensible addition. We just need to think this through -- carefully.
- Game calls HD Mouse API asking for mouse delta closest to now (or a specific historical timestamp not too long ago)
- HD Mouse API returns the delta.

This can be done just once per frametime.

(How the mouse driver handles this internally is up to the mouse driver. Most of the time it would be just a small trailing buffer of conventional historical sensor reads, but it unlocks optional internal 2-way workflows with mice connected via consistent ultralowlatency buses: Actually asking for the instantaneous sensor data at the instant)

Actually giving poll responsibility directly to the game software to actually poll the mouse once per gametime -- would eliminate mouse-event load on the software (e.g. 20,000 mouse events per second from a 20,000 Hz mouse), gobbling lots of CPU.

This would not be the only workflow, but it would unlock a hell lot of mouse precision, and also be far more jitter-immune, and permit a more asynchronous mouse workflow -- literally it unlocks an infinite-Hz analog mouse in theory, without the problem of an infinite pollrate.

So 8000 Hz mice would not overload a game engine with 8000 mouse movement events per second. Instead, the game engine would actually call the mouse driver to ask for the current delta.

Many existing game engines also wrappers a mouse and hides the event implementations away from the developer, only returning coordinates once per frame. In that sense, only the core engine needs modification -- the user of the game engine would not need to know the HD Mouse API existed! e.g. Unreal Engine, Unity Engine, etc. The engine vendors would just be responsible for HD Mouse API and everything would continue backwards compatible. Except you get 20,000Hz mouse precision without the CPU load of a 20,000Hz mouse!

The ability to query HD Mouse API only on-demand would do a better service to the Vicious Cycle Effect (higher Hz and higher resolutions amplifying tinier imperfections).
Head of Blur Busters - BlurBusters.com | TestUFO.com | Follow @BlurBusters on Twitter

       To support Blur Busters:
       • Official List of Best Gaming Monitors
       • List of G-SYNC Monitors
       • List of FreeSync Monitors
       • List of Ultrawide Monitors

User avatar
lyrill
Posts: 194
Joined: 06 Oct 2020, 10:37

Re: Potential HD Mouse Extensions API Specification [ETA by 2025]

Post by lyrill » 12 Feb 2021, 10:08

I noticed that you mentioned 20khz sensor refresh rate, did Razer tell you that 3399 has max 20k fps? thanks Chief, hope it's ok in a less populated thread.

User avatar
Chief Blur Buster
Site Admin
Posts: 9059
Joined: 05 Dec 2013, 15:44
Location: Toronto / Hamilton, Ontario, Canada
Contact:

Re: Potential HD Mouse Extensions API Specification [ETA by 2025]

Post by Chief Blur Buster » 13 Feb 2021, 14:37

lyrill wrote:
12 Feb 2021, 10:08
I noticed that you mentioned 20khz sensor refresh rate, did Razer tell you that 3399 has max 20k fps? thanks Chief, hope it's ok in a less populated thread.
I can confirm that the sensor does 20KHz. "sensor refresh rate" is the same thing as "sensor frames per second".

Mouse sensors are essentially low-resolution high-speed video cameras. Often crazy low numbers 32 by 32 pixels or 64 by 64 pixels, but enough to watch movement of a mouse surface.

With hacking, you can get the pictures from a mouse sensor: Hacking an Optical Mouse To Get Images From Mouse Sensor.

And, yes, the Razer 8KHz Viper is essentially a 20,000fps video camera.
Head of Blur Busters - BlurBusters.com | TestUFO.com | Follow @BlurBusters on Twitter

       To support Blur Busters:
       • Official List of Best Gaming Monitors
       • List of G-SYNC Monitors
       • List of FreeSync Monitors
       • List of Ultrawide Monitors

User avatar
lyrill
Posts: 194
Joined: 06 Oct 2020, 10:37

Re: Potential HD Mouse Extensions API Specification [ETA by 2025]

Post by lyrill » 13 Feb 2021, 15:23

Chief Blur Buster wrote:
13 Feb 2021, 14:37
lyrill wrote:
12 Feb 2021, 10:08
I noticed that you mentioned 20khz sensor refresh rate, did Razer tell you that 3399 has max 20k fps? thanks Chief, hope it's ok in a less populated thread.
I can confirm that the sensor does 20KHz. "sensor refresh rate" is the same thing as "sensor frames per second".

Mouse sensors are essentially low-resolution high-speed video cameras. Often crazy low numbers 32 by 32 pixels or 64 by 64 pixels, but enough to watch movement of a mouse surface.

With hacking, you can get the pictures from a mouse sensor: Hacking an Optical Mouse To Get Images From Mouse Sensor.

And, yes, the Razer 8KHz Viper is essentially a 20,000fps video camera.
Nice. Although this means that when using 20K dpi choice and 1 ips move speed (pretty low by today's standards lol) the sensor refresh rate is already saturated. any higher speed and it can't report every physical data/dots otherwise capable of analyzing/identifying.

This is curious because the 3360 also decided to pair the same number. any significance to that? because sure as hell it's got nothing to do with the 1 ips movement speed....? And on top of that there's the phenomenon that most pro gamers used <1k dpi and hz as if to pair them. Well now we can at least pair 8khz with 8k dpi. I'm using 20k even if the mouse doesn't have 20k from mcu to pc. That is unless theFiend has findings that there would be actual accuracy drawbacks of any kind.
Last edited by lyrill on 13 Feb 2021, 15:28, edited 1 time in total.

User avatar
Chief Blur Buster
Site Admin
Posts: 9059
Joined: 05 Dec 2013, 15:44
Location: Toronto / Hamilton, Ontario, Canada
Contact:

Re: Potential HD Mouse Extensions API Specification [ETA by 2025]

Post by Chief Blur Buster » 13 Feb 2021, 15:27

lyrill wrote:
13 Feb 2021, 15:23
Nice. Although this means that when using 20K dpi choice and 1 ips move speed (pretty low by today's standards lol) the sensor refresh rate is already saturated. any higher speed and it can't report every physical data/dots otherwise capable of analyzing/identifying.
Can you give me a link to the test result of 1 ips saturating 20 KHz? The saturation point is quite a fair bit higher. That means the sensor would have a super-narrow FOV

Remember... It depends on the FOV of the sensor. The bigger FOV a mouse sensor is, the faster the mouse can move without a higher frame rate. The 3399 is capable of moving really fast without saturating the sensor.

However, processing low resolution images accurately of many kinds of mouse pads can be quite difficult, so you need a good mouse pad that doesn't have a repeating pattern that confuses the mouse sensor. A mouse movement of half a video frame width can be correctly detected if the mousepad texture is appropriately random enough. However, the bigger % of a video frame movement between two adjacent video frames (1/20000sec apart), the less accurate the tracking is, since there's so little image data to correctly pattern-match.
Head of Blur Busters - BlurBusters.com | TestUFO.com | Follow @BlurBusters on Twitter

       To support Blur Busters:
       • Official List of Best Gaming Monitors
       • List of G-SYNC Monitors
       • List of FreeSync Monitors
       • List of Ultrawide Monitors

User avatar
lyrill
Posts: 194
Joined: 06 Oct 2020, 10:37

Re: Potential HD Mouse Extensions API Specification [ETA by 2025]

Post by lyrill » 13 Feb 2021, 15:30

Chief Blur Buster wrote:
13 Feb 2021, 15:27
lyrill wrote:
13 Feb 2021, 15:23
Nice. Although this means that when using 20K dpi choice and 1 ips move speed (pretty low by today's standards lol) the sensor refresh rate is already saturated. any higher speed and it can't report every physical data/dots otherwise capable of analyzing/identifying.
Can you give me a link to the test result of 1 ips saturating 20 KHz? The saturation point is quite a fair bit higher. That means the sensor would have a super-narrow FOV

Remember... It depends on the FOV of the sensor. The bigger FOV a mouse sensor is, the faster the mouse can move without a higher frame rate. The 3399 is capable of moving really fast without saturating the sensor.

However, processing low resolution images accurately of many kinds of mouse pads can be quite difficult, so you need a good mouse pad that doesn't have a repeating pattern that confuses the mouse sensor. A mouse movement of half a video frame width can be correctly detected if the mousepad texture is appropriately random enough.
ah ok. so it is speed related and independent of dpi? are you sure? I was just self translating the different mice terminologies and connecting the dots (literally lol)

User avatar
lyrill
Posts: 194
Joined: 06 Oct 2020, 10:37

Re: Potential HD Mouse Extensions API Specification [ETA by 2025]

Post by lyrill » 13 Feb 2021, 15:44

Chief Blur Buster wrote:
13 Feb 2021, 15:27
lyrill wrote:
13 Feb 2021, 15:23
Nice. Although this means that when using 20K dpi choice and 1 ips move speed (pretty low by today's standards lol) the sensor refresh rate is already saturated. any higher speed and it can't report every physical data/dots otherwise capable of analyzing/identifying.
Can you give me a link to the test result of 1 ips saturating 20 KHz? The saturation point is quite a fair bit higher. That means the sensor would have a super-narrow FOV

Remember... It depends on the FOV of the sensor. The bigger FOV a mouse sensor is, the faster the mouse can move without a higher frame rate. The 3399 is capable of moving really fast without saturating the sensor.

However, processing low resolution images accurately of many kinds of mouse pads can be quite difficult, so you need a good mouse pad that doesn't have a repeating pattern that confuses the mouse sensor. A mouse movement of half a video frame width can be correctly detected if the mousepad texture is appropriately random enough. However, the bigger % of a video frame movement between two adjacent video frames (1/20000sec apart), the less accurate the tracking is, since there's so little image data to correctly pattern-match.

hmm. So it is static on 20K fps right now? even though the ips is rated like 650, how fast of an ips would you say it starts to incorrectly pattern match for the typical shiden-kai or gsr?( ratmat had SQUAL ratings for those since like 2 years ago last i checked it still wasn't changed). I get that Razer always claims on the Master Guides that their sensors are tuned to work on their pads best, but they used to make metal pads, and metal pads are supposed to be more accurate that plastic ones, but the Acari is supposed to be especially tuned for optical, so maybe it beats the Invicta/Manticor
Last edited by lyrill on 17 Feb 2021, 03:55, edited 1 time in total.

User avatar
Chief Blur Buster
Site Admin
Posts: 9059
Joined: 05 Dec 2013, 15:44
Location: Toronto / Hamilton, Ontario, Canada
Contact:

Re: Potential HD Mouse Extensions API Specification [ETA by 2025]

Post by Chief Blur Buster » 13 Feb 2021, 15:48

lyrill wrote:
13 Feb 2021, 15:30
ah ok. so it is speed related and independent of dpi? are you sure? I was just self translating the different mice terminologies and connecting the dots (literally lol)
NO.... Not independent of DPI.

They can be linked. But remember, sensors are NOT 1 pixel!!! ;)

Mouse sensor only needs partial overlap between frames. For example, half a video overlap. Imagine a photo montage (two photos) where only half overlap perfectly. Let's make that assumption, for simplicity's sake, that means a ~16x multiplier for a 32x32 sensor, and a ~32x multiplier for a 64x64 sensor.

But that's not even the only factor.

You've also got the FOV -- a wider-FOV mouse sensor can amplify the ability to move faster without needing a higher frame rate. You can get more adjacent-frames overlap with a wider FOV camera but you get less detail of the mousepad surface. So you can have faster movement but less reliable tracking of smooth surfaces (glass-like surfaces).

While algorithms for detecting video overlaps can be complex, the mathematics of how many ips can work, has many variables:
- The FOV of sensor (double FOV = you can move twice the speed)
- The minimum amount of sensor video-frames overlap (A half-overlap means you can move twice the speed of quarter-overlap)

And these factors can be additional influences on whether it's easy/hard to adjust above variables.
- The resolution of the sensor, since higher resolution can track marginal surfaces better.
- The noise of the sensor, since less video noise can track marginal surfaces better.
- And don't forget the ability to accurately track marginal surfaces can be improved/compromised by tweaking the above
- And don't forget colordepth / dynamic range of sensor can help compensate (e.g. 8-bit of greyscale versus 4-bit of greyscale) on marginal surface tracking ability, which can give sensor a bit more luxury to increase FOV without losing ability to track some smooth mousepads (or almost-smooth desks such as melamine surfaces).

P.S. I'm going to have to split this thread into "Basic Mouse Sensor Mathematics 101" thread and let other experts reply on my behalf. Some of these are easy high school thought exercises.
lyrill wrote:
13 Feb 2021, 15:44
hmm. So it is static on 20K fps right now? even though the ips is rated like 650, how fast of an ips would you say it starts to incorrectly pattern match for the typical shiden-kai or gsr?( ratmat had SQUAL ratings for those since like 2 years ago last i checked it still wasn't changed). I get that Razer always claims on the Master Guides that their sensors are tuned to work on their pads best, but they used to make metal pads, and metal pads are supposed to be more accurate that plastic ones, but the Acari is supposed to be especially tuned for optical, so maybe it beats the Invicta.
I don't know the answer to the question.

However, understanding the variables can help you study & research this. Design your own tests. Etc. Usually it is a continuum of slow degradation where you may have 1% deviation then 2% deviation then 5% deviation and so on, a curve that falls off in accuracy.

Also, flex/vibration of the surface is a problem sometimes for certain kinds of mouse pads, with your furious mouse movements and the stiction of the mousefeet stretching around/etc a mousepad, and all, the inaccuracies can build up during faster movements

For power management purposes, the 20 Kfps does slow down when the mouse is stationary.
Head of Blur Busters - BlurBusters.com | TestUFO.com | Follow @BlurBusters on Twitter

       To support Blur Busters:
       • Official List of Best Gaming Monitors
       • List of G-SYNC Monitors
       • List of FreeSync Monitors
       • List of Ultrawide Monitors

Post Reply