Could dlss ai upscaling improve motion resolution in the future?

[tygeezy]

Nvidia has made a lot of headway with dlss with the 2.0 release. Future releases say they are looking at implementing it at the driver level in any game that support TAA.

My question is, would it be possible to use ai upscaling to improve motion resolution on camera panning in shooter games?

That seems like a holy grail type of technology since sample and hold isn’t going away and we are probably far away from 1000 hz displays and hardware that can render that fast.

[deama]

Probably not as the motion blur problem comes from the monitor itself.
There are games that force motion blur, then a dlss-type thing or some filter would work, but I think that's just a hand full of games.

[Chief Blur Buster]

My question is, would it be possible to use ai upscaling to improve motion resolution on camera panning in shooter games?

Short answer: YES

For a theoretical DLSS 3.0 I'd love to see them combine the current spatial FRAT algorithm with Oculus-style ASW temporal FRAT algorithm, for non-VR games, to improve motion resolution of panning/turning/scrolling, for 240Hz+ and 360Hz+ displays.

Frame Rate Amplification Technology (FRAT)


If done properly, the input-to-photons (1000Hz mousemoves) can be an input into the frame rate amplification technology, allowing reprojection to lower input lag, unlike old-fashioned television interpolation that increases input lag, generates a Soap Opera Effect (SOE) artifacts, and parallax edge-movement artifacts. None of that exists with my Oculus Rift's amplifying 45fps into 90fps, since it's reprojecting in 3D with the help of the Z-Buffer, using ASW 2.0

Amplifying 100fps into 1000fps (for future retina refresh rate displays -- blurless sample-and-hold, like ULMB but without strobing), can be less lag than 100fps, despite the extra processing overhead of reprojection.

Old-fashioned interpolation was an ugly black box, but the frame rate amplification of the future, that is already happening, uses things like Z-buffer and input (head trackers, mouse, etc) at higher refresh rate than the original GPU full-render framerate, so theoretically adding more "fake frames" can reduce input lag, instead of increase input lag, since no lookahead buffering is needed once you're including high-Hz motion vectors from high-Hz input (and game variables) into a frame rate amplification technology that is less black box.

It's easy to predict that NVIDIA is already probably working on this stuff, most NVIDIA researchers likes the "Frame Rate Amplification Technology" article. Adding more "fake frames" isn't bad if they're virtually flawless frames without input lag...

(Netflix video already does something similar in 2D -- I-Frames, B-Frames, and P-Frames -- so this is essentially rearchitecturing the GPU motion pipeline in all 3 dimensions, to reduce the number of original GPU full-renders to achieve an ultra high frame rate, and using AI + FRAT + reprojection + non-blackbox stuff (high-Hz controller input) to skip the need for full GPU renders for every single frame. New techniques now exist to make these intermediate frames virtually flawless.

I would not be shocked that a future version of DLSS within five years, will also eventually add optional temporal frame rate amplification algorithms (similar to Oculus ASW).

Imagine, Unreal 5 graphics quality at 1000fps...

[Chief Blur Buster]

Probably not as the motion blur problem comes from the monitor itself.

This is correct.

However, most new games run at framerates far below max Hz of highest Hz monitors (240Hz, 280Hz, 360Hz).

Unreal 5 probably won't support 360fps with current graphics detail, without the help of frame rate amplification.

Motion resolution is maximized at framerate=Hz, and that's where DLSS AI upscaling can improve motion resolution by bringing framerates up to Hz, while keeping ULTRA detail.

This is how 1000fps becomes cheaper for future planned 1000hz monitors
(Did you know? ASUS has long-term roadmapped 1000Hz already -- PC Magazine)

[tygeezy]

Probably not as the motion blur problem comes from the monitor itself.

This is correct.

However, most new games run at framerates far below max Hz of highest Hz monitors (240Hz, 280Hz, 360Hz).

Unreal 5 probably won't support 360fps with current graphics detail, without the help of frame rate amplification.

Motion resolution is maximized at framerate=Hz, and that's where DLSS AI upscaling can improve motion resolution by bringing framerates up to Hz, while keeping ULTRA detail.

This is how 1000fps becomes cheaper for future planned 1000hz monitors
(Did you know? ASUS has long-term roadmapped 1000Hz already -- PC Magazine)

I see, so the idea would be to maximize your displays potential. So if you’re running a game at say 60 FPS in your 120 hz display, dlss Cohen theoretically increase the displays motion resolution to the maximum potential of your display? That actually sounds really good. This type of tech would be awesome for movies as well.

Standard 23.976 FPS films look just terrible whenever there is panning shots. This tech on newer nvidia shields would probably be a game changer. If you could increase motion resolution without messing with the film cadence and without adding flicker and picture dimming.

Movies and games stand to benefit a ton from this type of stuff.

[Chief Blur Buster]

I see, so the idea would be to maximize your displays potential. So if you’re running a game at say 60 FPS in your 120 hz display, dlss Cohen theoretically increase the displays motion resolution to the maximum potential of your display? That actually sounds really good. This type of tech would be awesome for movies as well.

Yes, DLSS 2.0 does roughly a 2:1 frame rate amplification so you can already double your motion resolution today! This assumes your frame rates are well below Hz, and you want to use DLSS 2.0 to double your frame rates to effectively double your motion resolution.

What I'm REALLY looking forward to is 5:1 and 10:1 frame rate amplification ratios -- those ratios are needed for 1000Hz displays of the future, without sacrificing graphics detail.

Standard 23.976 FPS films look just terrible whenever there is panning shots. This tech on newer nvidia shields would probably be a game changer. If you could increase motion resolution without messing with the film cadence and without adding flicker and picture dimming.

Standard 24fps (23.976fps) looks fine if you need frame rates to be low enough for story telling (movies). Not everyone can handle ultra high frame rates, so there's a market for standard cinema frame rates (and this is where GPU motion blur benefits, since that can make low frame rates less painful).

We're actually fans of the Hollywood FIlmmaker Mode in TVs.

Now -- however, other peole get headaches from motion blur, so an ultra HFR option is needed. This is why we're now also big advocates of ultra HFR video too -- www.blurbusters.com/ultrahfr

There's currently a sort of uncanny valley between low frame rates (i.e. 24fps) and ultra high frame rates (i.e. 1000fps) if you're playing to a sample-and-holde display -- standard "HFR" at 120fps still produces motion blur (1/120sec) for a sample-and-hold display.



120fps HFR is very smooth but it still has motionblur from display persistence of sample-and-hold displays. This is not always enough to eliminate nausea at the cinema (headaches from 48fps and 120fps HFR at cinema is very similar in some ways to VR headaches from virtual reality motion blur), since there's still motionblur from display persistence.

(A) Lower the frame rate back to 24fps since it's below the smooth motion threshold
Pro: Frame too low to make most people motionsick
Con: Extremely blurry and stuttery

(B) Or Impulse the display (flicker the display, via strobe backlight or black frame insertion, etc), 120Hz at low MPRTs
Pro: Very smooth. Eliminates motion blur of fast-shutter video/film material
Con: Flicker from the impulse-driven display

(C) Or go even more brute frame rates. To match motion clarity of a 1/1000sec stationary photograph, via video on a sample-and-hold display, requires 1000fps at 1000Hz.
Pro: No flicker, no motion blur from fast-shutter video/film material
Con: Few. Some people will still prefer 24fps, but fewer people will get motionsick than 120fps HFR

TL;DR: 24fps/24Hz and 1000fps/1000Hz is more comfortable than 120fps/120Hz on sample-and-hold displays
(surprisingly so!)