Guffman wrote: ↑09 Sep 2022, 23:31
If interpolation really is the only solution then I suppose that's a different avenue I'd have to go down... Pretty unfortunate though given most interpolation I've seen introduces significant processing lag -- totally fine for video content but mostly unplayable for gaming.
Good news.
Interpolation is not the only option, at least if we’re handling modern engines.
If you’ve ever played VR, they use a virtually lagless “add extra frames” technology called “reprojection”. Such as Oculus Asynchronous Warp (ASW).
PC games don’t generally do this, but VR games do it quite a lot. So the technology already exists, but requires access to additional data from the game engine (like high-Hz controller inputs and Z-buffer) to make it lagless and more artifact free.
You can read more about this in the
Frame Rate Amplification Technology: Getting 1000fps For Cheap In Future.
As AI-based frame rate amplifiers become higher performance, they will eventually be capable of 10:1 frame rate amplification ratios, generating 1000fps output from 100fps feedstock in realtime.
We’re actually seeing elements of frame rate amplifiers borne itself out of research as we speak — in a left field area. DLSS is using neural network AI, as you already know — and AI is rapidly improving.
You’ve seen those AI art-engines (DALL-E, Midjourney, et al). DLSS 5.0 on steroids will probably be realtime inpainting those in-between frames. I heard a rumor that the top end of the RTX 4090 is capable of generating AI art in just 1-2 seconds, and DLSS is a much simpler operation than AI art. Soon, we can generate “AI-art-movies” at realtime framerates in a couple or more GPU generations later — I would predict.
Virtual-reality reprojection algorithms converts 45fps feedstock into 90fps view, eliminating stutters without adding interpolation latency. So the technology is here today, if you own a recent VR headset.
Now in the future …
GPU vendors can redirect some of the new ‘AI skills’ from AI-art-processing capability called “inpainting” (and similar algorithms) as a turbocharged DLSS with a 5:1 or 10:1 frame rate amplification ratio, possibly combined with temporally-dense raytracing algorithms too (NVIDIA cited my TestUFO in their research paper). This is all exciting developments.
Essentially, a merger between NVIDIA DLSS + Oculus ASW + subvariants of AI-art techniques like AI-based inpainting algorithms will make the 1000fps frame rate amplifier a reality within a decade or so — on a single commodity GPU.
That said, the silicon manufacturing shortage and geopoliticals (TSMC…) may set frame rate amplification back a few years. But UE5-quality 1000fps should become practical eventually.
A frame rate amplifying co-processor (or dedicated silicon) may be required, to keep the GPU workload down — given the power requirements of GPUs it may need to be done as a chiplet approach (bandwidth…), but as long as the AI engine has access to 1000Hz controller inputs as well as the Z-buffer to generate parallax-correct (artifact free) AI based frame rate amplification.
Guffman wrote: ↑09 Sep 2022, 23:31
Is there something possibly that can be done with interlacing video? In theory, each field of an original 30fps source is only on-screen for 16.66ms, right? So even though the original frames are still duplicated to reach 60hz output, only the odd or even lines of each frame will be displayed at a time.
Obviously this would require a video processor to interlace the content and then a second video processor to de-interlace back into 60hz progressive, but combining this with a framedoubler for 120hz and adding BFI (please correct me if this terminology is wrong btw, since I mean the signal itself is 120hz but only 60 of those frames are black... is that 60hz BFI?) seems like an interesting idea... I would consider just leaving the signal interlaced, but the flicker drives me up a wall, and progressive displays generally deinterlace anyway (often lagfully), not to mention bob deinterlacing being about as flickery, if not more-so, than the original interlaced video was to begin with...
A smart programmer would need to combine the video processor steps into a simultaneous step to avoid the serial latency-adding steps.
Perhaps even beam-race the video processing as much as possible (at least to keep it only 1/60sec latency, since you can only beam-race the second field with a resulting beam-raced progressive scan image.
Raster beam raced video processors/scalers are very rare (I think HDFury is one of them) because they use subframe buffering (rolling window of number of scanlines) spewing output concurrently with input, with only a few scanlines lagbehind.
Be noted that BFI is not necessarily a 50%:50% step, and sometimes BFI is a hardware operation (strobe backlight). BFI is often used as a synonym to strobing.
However, BFI at the software level, depends on the output refresh rate, so if you have 360Hz of output refresh rate, you can do BFI in 1/360sec increments. You can see variable-persistence BFI at
www.testufo.com/blackframes#count=4&bonusufo=1 …. This version uses a 4-frame BFI per frame cycle, so best played at 240Hz for 60fps variable-persistence BFI (25% blur reduction, 50% blur reduction and 75% blur reduction). If you try to play this at 60Hz, it will flicker a lot at 15Hz, but it can still demonstrate the educational concept.
RetroArch is one of the only emulators that can do BFI at higher refresh rates (e.g. 180Hz, 240Hz, 300Hz, 360Hz) to produce less motion blur than 60Hz BFI.
The most lag-free scalers (e.g. HDFury) does not have built in BFI, but in theory software BFI won’t add more lag on top of the lag that a video processor generates, provided the visible frame is the first frame of the BFI sequence.
It’s more ideal if you use a progressive scan source, but a good beam-raced video processor should keep 30fps deinterlacing to only 1/60sec input lag (buffering the first field, and doing beam-raced deinterlacing during the second field). I don’t know if any video processors are that low latency in deinterlacing 30fps material.
This can be problematic for cadence-change events (e.g. when you’ve got a 1/60sec framedrop, so you’ve got one frame that has even-odd-even or odd-even-odd and subsequent frames having swapped odd-even versus even-odd cadence. All this stuff ideally needs lookahead, so buffering a few fields is more common.
I would expect that a theoretical future AI-assisted beam raced deinterlacer could keep latency pretty low, without generating artifacts, simply by the AI knowing what to expect from the engine during frame drops, etc.
If you are doing an emulator, beam racing is already possible (synchronization of real raster to emulator raster), I helped a few emulator authors achieve this (most particularly, WinUAE, but also is in CLK and an early-test version of GroovyMAME by Calamity).
. Unfortunately companies create technologies for 60fps or more and forget about 30fps. 
I don't care about the motion blur, the problem is that at 30fps it has a lot of flickering in several games, especially in very bright environments, I think that the light gray BFI would already solve this and with less flickering at 60hz monitor. I plan to do 60hz = 30fps.