spacediver and ad8e public collaboration [reaction research]

Talk to software developers and aspiring geeks. Programming tips. Improve motion fluidity. Reduce input lag. Come Present() yourself!
ad8e
Posts: 68
Joined: 18 Sep 2018, 00:29

spacediver and ad8e public collaboration [reaction research]

Post by ad8e » 12 Jul 2019, 00:13

I decided that for useful stuff that everyone would learn from, I'll post it publically rather than in our PMs.

Here's a specific description of the model method. The model says, "as long as the mouse continues to move downward, I predict the next movement point will be within a 99.99% confidence interval that is determined by the typical characteristics of a downward sweep." Then you try to bound this confidence interval as tightly as possible, using what you know of downward sweeps. Once you receive a leftward movement point that isn't in this confidence interval, you return the flag "behavior is different from expected! the leftward jerk has started!"

Model bounds are generally holistic in some way, so when considering a new timepoint, you can't necessarily assume that all the previous timepoints were part of a downward sweep, even though you haven't thrown the "jerk started" flag yet. Some of those timepoints may be under suspicion already, and the last timepoint is just the last straw that triggers the flag. You can see where things can get complicated and sophisticated real fast.

These models also usually let you tune the 99.99% number to other values like 75%, but I don't particularly recommend that for a beginner, just make something that works well on your own hand.

Here's a picture.
Image

This is an illustration of one possible example input. 1 2 3 4 5 are the previous timepoints. 4 and 5 are already starting to look suspiciously like a left jerk, but the code isn't sure yet. The green region is the region where the code thinks, "if the next movement point is here, it's plausible that the person is still moving downward, and I'll return the 'not yet' value". The orange region is the region where the code thinks, "if the next movement point is here, I no longer have any expectation that the person is still moving downward, he has clearly started a left jerk, so I'll return the 'it's started' value". The green and orange regions together should cover all, or nearly all inputs. There might be some leftover space outside the green and orange regions which the mouse rarely achieves. When it's sufficiently unlikely that the mouse ends up in that leftover space, it doesn't matter what is returned. (For example, if the next movement point teleports back to 1 again in an extremely unlikely scenario, then the most likely explanation is that the mouse is bugged, and the code can return whatever it wants.)

Here's a picture of another possible situation.
Image

In this example, 4 and 5 are not suspicious. Since there isn't suspicion yet, the acceptable region is larger.

The regions you eventually settle on may look very different from my MS Paint pictures; they're only illustrative of a model technique, not what I think the correct model will be.

User avatar
Chief Blur Buster
Site Admin
Posts: 11647
Joined: 05 Dec 2013, 15:44
Location: Toronto / Hamilton, Ontario, Canada
Contact:

Re: spacediver and ad8e public collaboration [reaction resea

Post by Chief Blur Buster » 12 Jul 2019, 17:56

For readers (which sometimes includes academics and researchers) who google and come directly to an unfamiliar thread (it happens more often than one thinks)....

To newly arriving readers: The links below provide helpful pre-requisite context, plus also supplementary esports "reaction time" discussion, to understand some of the background of this thread. This is just for thread-completeness sake for other readers arriving to this thread.

Interesting project about mouse/ gamepad latency

Page 2 | Page 3 | Page 4 | Page 5 | Page 6 | Page 7 | Page 8 | Page 9 | Page 10

(carry on)
Head of Blur Busters - BlurBusters.com | TestUFO.com | Follow @BlurBusters on Twitter

Image
Forum Rules wrote:  1. Rule #1: Be Nice. This is published forum rule #1. Even To Newbies & People You Disagree With!
  2. Please report rule violations If you see a post that violates forum rules, then report the post.
  3. ALWAYS respect indie testers here. See how indies are bootstrapping Blur Busters research!

spacediver
Posts: 505
Joined: 18 Dec 2013, 23:51

Re: spacediver and ad8e public collaboration [reaction resea

Post by spacediver » 12 Jul 2019, 18:40

Thanks for taking the time here.
ad8e wrote:
Model bounds are generally holistic in some way, so when considering a new timepoint, you can't necessarily assume that all the previous timepoints were part of a downward sweep, even though you haven't thrown the "jerk started" flag yet. Some of those timepoints may be under suspicion already, and the last timepoint is just the last straw that triggers the flag. You can see where things can get complicated and sophisticated real fast.
I think I understand (at least abstractly) all of your post except this part.

What about a simpler approach? Take this trajectory, which is raw data from a trial I just ran based on the binary you gave me. Admittedly, this is a very clean trial, so it's relatively easy to determine when the jerk occured.

This is x plotted against y, so this just shows the shape of the trajectory, unordered by time. I start moving by moving mouse downward, and then suddenly jerk it left.

Image

Here is a plot of the angle that I was moving mouse (ignore the absolute values of the angles, just look at the pattern). Superimposed in red is the velocity in x direction (it's been scaled to fit the same axis). Data are low pass filtered.

Image

I think it's not too difficult to design an algorithm (in linear time) that, using a combination of peak detection, and some thresholds for velocity and angle, could reliably determine the jerk onset time, though not sure what the associated error would be.

ad8e
Posts: 68
Joined: 18 Sep 2018, 00:29

Re: spacediver and ad8e public collaboration [reaction resea

Post by ad8e » 12 Jul 2019, 19:57

spacediver wrote:Thanks for taking the time here.
ad8e wrote:
Model bounds are generally holistic in some way, so when considering a new timepoint, you can't necessarily assume that all the previous timepoints were part of a downward sweep, even though you haven't thrown the "jerk started" flag yet. Some of those timepoints may be under suspicion already, and the last timepoint is just the last straw that triggers the flag. You can see where things can get complicated and sophisticated real fast.
I think I understand (at least abstractly) all of your post except this part.
What it means is that the shape of the reject and accept regions will depend on the previous timepoints in strange and non-linear ways. With the info I gave, an approach that uses the velocity direction as an axis is almost certainly the optimal one, and will remove most of the strangeness and non-linearity.

Your proposed method will also work, but the less features it captures from the mouse model, the more conservative its results will be. (i.e. it will take longer to detect the jerk.)

Your mouse's CPI is indeed too low, as I suspected. The stuttering is in full effect; check the graphed positions around the -100 to -200 region. That will make your life harder.

As an illustration of why high CPI should be so important, let me give some physics background.

The first rule is, the mouse's position will be near its previous position; this is inertia.
The second rule is, the mouse's velocity also won't change too fast, because of conservation of momentum.
The third rule is, the mouse's acceleration also won't change too fast, because your hand and the mouse form a spring system.
The fourth rule is, the derivative of the mouse's acceleration also won't change too fast, because of conservation of momentum plus the third rule.
Of these four rules, the first rules are more true than the later rules. Detection systems with finer time resolution can see more rules, producing fifth, sixth, and seventh rules. My guess is that the 1000 Hz mouse can only see up to four.

These four rules, plus the characteristics of human mouse movement which I described as "inconsistent speed and semi-consistent direction within a single movement", mean the optimal model most likely differentiates the XY position graph to produce 3 new graphs, for velocity, acceleration, and acceleration-derivative. Then, it splits the acceleration and acceleration-derivative graphs into an along-velocity component and perpendicular-to-velocity component. Note that velocity may need smoothing and error bounds.

The end region should mainly depend on the split acceleration and acceleration-derivative graphs. But with a low CPI, higher derivatives become noisy, and the smoothing needed to fix the noisiness will delay them out to uselessness. That is, they'll have no predictive power on the final region, with the acceleration-derivative graph losing its power first.

But you should build the system you're able to build. If the system I described looks too hard, you can ignore it.

For error, you likely won't be able to produce a specific number with your current skillset, and that's not a problem. "Reliable enough" is all we need.

spacediver
Posts: 505
Joined: 18 Dec 2013, 23:51

Re: spacediver and ad8e public collaboration [reaction resea

Post by spacediver » 12 Jul 2019, 21:32

ad8e wrote: What it means is that the shape of the reject and accept regions will depend on the previous timepoints in strange and non-linear ways. With the info I gave, an approach that uses the velocity direction as an axis is almost certainly the optimal one, and will remove most of the strangeness and non-linearity.
Ok, so the model's input is the history of the trajectory (perhaps up to a certain point in the past), and the output (which changes from moment to moment) is a two dimensional probability distribution that describes how probable it is that a particular point in space is the future trajectory assuming that the downward sweep is still continuing. Is that a correct description?
ad8e wrote: Your proposed method will also work, but the less features it captures from the mouse model, the more conservative its results will be. (i.e. it will take longer to detect the jerk.)
Yep, that makes sense.
ad8e wrote: Your mouse's CPI is indeed too low, as I suspected. The stuttering is in full effect; check the graphed positions around the -100 to -200 region. That will make your life harder.
Yea this was just a quick test, I'll be using my other mouse (which hopefully has a higher native DPI) for proper testing.
ad8e wrote:As an illustration of why high CPI should be so important, let me give some physics background.
.

Yes, the smaller the timescales are that actually contain diagnostic information, the higher the temporal resolution needs to be to capture these patterns - btw, perhaps we should at this point rename "jerk" (used to describe when the person jerks their hand after downward movement) to something else, since we may be using the word "jerk" to refer to the derivative of acceleration down the road. Maybe "snap" or something...
ad8e wrote: The third rule is, the mouse's acceleration also won't change too fast, because your hand and the mouse form a spring system.
Interestingly, last night, was doing some testing with human benchmark, and when I squeezed the mouse hard between my thumb and pinky, my reaction times went down immediately by ~30 ms. Might have been fluke, might not have been, but it made me wonder about whether it was due to priming my muscles, or perhaps it simply just stabilized the mouse so that the force of the button press was more efficiently transferred to pressing the button rather than moving the mouse. Either way, squeezing the mouse hard might reduce latency of a snap movement, since there will be less mass to deform in the finger pads when initiating the movement, although doing so might also worsen performance due to 'tension' or something.
ad8e wrote:The fourth rule is, the derivative of the mouse's acceleration also won't change too fast, because of conservation of momentum plus the
These four rules, plus the characteristics of human mouse movement which I described as "inconsistent speed and semi-consistent direction within a single movement", mean the optimal model most likely differentiates the XY position graph to produce 3 new graphs, for velocity, acceleration, and acceleration-derivative. Then, it splits the acceleration and acceleration-derivative graphs into an along-velocity component and perpendicular-to-velocity component. Note that velocity may need smoothing and error bounds.
Is this as simple as taking the dot product of acceleration with velocity to get the along component (and repeating this for accel derivative)?

How would you get the perpendicular component?

The dot product of acceleration and velocity is proportional to power. Is this relevant? What's the reasoning behind obtaining these components. Is the idea that somehow they are a more suitable space to look for snap detection?

ad8e wrote: But you should build the system you're able to build. If the system I described looks too hard, you can ignore it.
It might be too hard, but I'm not ready to give up on what could be a really valuable learning experience either!

Depending on my knowledge gap, speed of learning, and your patience, we'll see what happens :p

ad8e
Posts: 68
Joined: 18 Sep 2018, 00:29

Re: spacediver and ad8e public collaboration [reaction resea

Post by ad8e » 12 Jul 2019, 22:18

spacediver wrote:
ad8e wrote: What it means is that the shape of the reject and accept regions will depend on the previous timepoints in strange and non-linear ways. With the info I gave, an approach that uses the velocity direction as an axis is almost certainly the optimal one, and will remove most of the strangeness and non-linearity.
Ok, so the model's input is the history of the trajectory (perhaps up to a certain point in the past), and the output (which changes from moment to moment) is a two dimensional probability distribution that describes how probable it is that a particular point in space is the future trajectory assuming that the downward sweep is still continuing. Is that a correct description?
Excellent. That's how the model is usually built. Your version can be simpler (yes/no rather than a % chance), but that's the core of the idea.
spacediver wrote:Maybe "snap" or something...
Let's go with snap then.
spacediver wrote:Interestingly, last night, was doing some testing with human benchmark, and when I squeezed the mouse hard between my thumb and pinky, my reaction times went down immediately by ~30 ms. Might have been fluke, might not have been, but it made me wonder about whether it was due to priming my muscles, or perhaps it simply just stabilized the mouse so that the force of the button press was more efficiently transferred to pressing the button rather than moving the mouse. Either way, squeezing the mouse hard might reduce latency of a snap movement, since there will be less mass to deform in the finger pads when initiating the movement, although doing so might also worsen performance due to 'tension' or something.
Tight grip will both increase sideways noise in the downward sweep and increase responsiveness in the leftward snap.
spacediver wrote:Is this as simple as taking the dot product of acceleration with velocity to get the along component (and repeating this for accel derivative)?
Yes, after normalizing velocity to 1. Considering that velocity is not perfectly accurate, there will be some error when doing this, but this is the right method. For example, if velocity is 0, then direction is meaningless and then error becomes really important.
spacediver wrote:How would you get the perpendicular component?
Subtract the parallel component from the original vector.
spacediver wrote:The dot product of acceleration and velocity is proportional to power. Is this relevant? What's the reasoning behind obtaining these components. Is the idea that somehow they are a more suitable space to look for snap detection?
Power is not relevant. The idea is that each component has predictive power on the next position, and components capture conceptually distinct parts of the behavior. "Suitable space" sounds right. As an analogy, if the system was linear (it's not), then the components would be the eigenvalues from PCA.
The components are: XY position, XY velocity, parallel-accel, perpendicular-accel, parallel-accel-derivative, and perpendicular-accel-derivative.

The last four components have some velocity-uncertainty thing going on. The direction is also much less consistent when the velocity is low. I recommend as a first step, taking all the data you have of downward sweeps fast and slow, then doing a scatter plot. The plot will have velocity magnitude as its X axis (magnitude is sqrt(x^2 + y^2)), and the other components on the Y axis. This will give a very good idea of what the error tolerances will look like. My description of "inconsistent speed and semi-consistent direction" is like saying that the parallel and perpendicular components will have different plots on this graph. If you can make 3D rotatable scatter plots, that will make any interactions easier to see.

My guess of why direction and speed behave differently is that the human mind controls its hand by streaming out pulses of force. A steady movement is approximated by a sequence of pulses, one after another. I think it might be possible to build a model that detects these underlying pulses, but that's some top tier research, definitely not suitable for us to do in a small reaction test.

spacediver
Posts: 505
Joined: 18 Dec 2013, 23:51

Re: spacediver and ad8e public collaboration [reaction resea

Post by spacediver » 12 Jul 2019, 22:39

Thanks, this is some good stuff to munch on. Will be back at this next week.

And building a model to detect those pulses sounds like a very cool line of research.

spacediver
Posts: 505
Joined: 18 Dec 2013, 23:51

Re: spacediver and ad8e public collaboration [reaction resea

Post by spacediver » 15 Jul 2019, 12:24

ad8e wrote:The idea is that each component has predictive power on the next position, and components capture conceptually distinct parts of the behavior. "Suitable space" sounds right. As an analogy, if the system was linear (it's not), then the components would be the eigenvalues from PCA.
The components are: XY position, XY velocity, parallel-accel, perpendicular-accel, parallel-accel-derivative, and perpendicular-accel-derivative.
If I'm understanding correctly, you're saying that XY velocity magnitude is likely to contain information that accounts for most of the variance in the data associated with a downward sweep.

But wouldn't Y-position have the most variance?
ad8e wrote: I recommend as a first step, taking all the data you have of downward sweeps fast and slow, then doing a scatter plot. The plot will have velocity magnitude as its X axis (magnitude is sqrt(x^2 + y^2)), and the other components on the Y axis. This will give a very good idea of what the error tolerances will look like. My description of "inconsistent speed and semi-consistent direction" is like saying that the parallel and perpendicular components will have different plots on this graph. If you can make 3D rotatable scatter plots, that will make any interactions easier to see.
Just to be clear, do you mean doing a scatter plot for each downward sweep trial? Or do you mean that each point on the scatter plot should represent a single trial?

ad8e
Posts: 68
Joined: 18 Sep 2018, 00:29

Re: spacediver and ad8e public collaboration [reaction resea

Post by ad8e » 15 Jul 2019, 13:45

The variance I'm speaking of is the noise away from the prediction. As an example, if you toss a ball in vacuum, your model will have zero variance because you can calculate its future position perfectly, even though its position variance will be high.

Y-position will have the most variance but I don't see why it matters. Sweeps are translation-invariant. The only value of XY-position is to determine the various derivatives; after that, XY-position can be discarded.

Each point on the scatter plot will be one timepoint. One scatter plot for all the sweeps together. So if you have 20 trials and 50 timepoints per trial, your scatter plot will have 1000 points. And you'll have a few different scatter plots, one for accel-perpendicular, one for accel-parallel, etc. With 3D plots, you can cut down on the number of plots plus see more per plot, because rotating the 3D plot is like doing orthogonal projections in arbitrary directions.

spacediver
Posts: 505
Joined: 18 Dec 2013, 23:51

Re: spacediver and ad8e public collaboration [reaction resea

Post by spacediver » 15 Jul 2019, 16:36

I've spent too much time today trying to overclock my other mouse (the one with higher cpi) to 1000 hz, as it's currently stuck on 125hz. No success. I can pick up this mouse for about $15 down the street:

https://www.redragonzone.com/products/r ... ming-mouse

Hopefully that one will do the trick.
ad8e wrote:The variance I'm speaking of is the noise away from the prediction. As an example, if you toss a ball in vacuum, your model will have zero variance because you can calculate its future position perfectly, even though its position variance will be high.
I *think* I'm starting to get it. By collecting a bunch of trials, we can see how these components relate to each other, and build a model based on that relationship.

You've explained why acceleration and jerk are likely to contain diagnostically useful information (the general idea being that they shouldn't change too fast during a sweep).

I'm still not clear why the parallel and orthogonal "subcomponents" (with respect to XYVelocity) will show the "conceptually distinct parts of this [sweep] behaviour".

In my mind, the onset of the snap is associated with an abrupt change in regime of incoming force pulses.

Is the idea that if when this snap occurs, that the component-of-accel-and-jerk-that-show-the-greatest-change would be the component that is orthogonal to XYVelocity? And if so, is this primarily due to the rapid change in intensity/frequency of these pulses? Or the rapid change in direction of mouse movement?
ad8e wrote: Y-position will have the most variance but I don't see why it matters. Sweeps are translation-invariant. The only value of XY-position is to determine the various derivatives; after that, XY-position can be discarded.
Ok, so we're solely concerned with the kinematics of the movement, rather than the direction (i.e. a valid sweep could be a horizontal one).

Post Reply