Hyperlapse: Realtime convert video to 3D scenery! Holodeck?

[Chief Blur Buster]

An astoundingly impressive technology, coming from Microsoft Research (of all things), is Hyperlapse.
See Microsoft Research Article.

It is able to produce astoundingly smooth timelapse video, from existing videos such as GoPro.

phpBB [video]

It essentially is converting 2D video to true 3D scenery (your GoPro camera becomes a 3D scanner!) and creating a new shake-free virtual camera path through it all, and rendering the video as 3D scenery using the new virtual camera path.



See the video below. The computing power that makes this possible today, is astounding.

phpBB [video]

Looking for into the crystal ball (for technology geeks), this is the Wright Brothers of a fundamental technology innovation that may lead to holographic video playback.
-- this could be a possible root of H.266 or H.267 compression algorithm: Vector video playback as true 3D scenery. A very first generation of a viable vectorized, framerateless 3D video compression (no upper limit to framerate in playback) where there is no resolution limit or framerate limit to video, because the video is made out of vectors, possibly SuperResolution'd on a continuous basis with whatever resolution the intermediate frames provides.
-- Framerate interpolation with no upper limit to framerate.
-- Interpolation can be improved greatly over time
-- Eventually, be able to enhance the resolution of existing video via realtime SuperResolutioning (and similiar algorithms) between adjacent frames
-- You could do two virtual camera paths, and do 3D!
-- You could do holographic 3D (to a certain extent) where you're allowed to tilt your head around, since there's lots of flexibility in the virtual camera paths.
-- Possibly in the distant future; maybe converted to Holodeck format so we can "walk around in our 100 year old videos" virtually. (Eventually, thousands of cameras will have passed over the same spot on Earth, and it could all be merged into one ultra-high-def Holodeck experience with few missing interpolated spots that can be artificail-intelligently & artistically auto-completed.)

Obviously, the Hyperlapse result is only as good as the source camera allows -- resolution limits and framerate limits in the source material will limit how good the Hyperlapse result. But, still, imagine a 4K 120fps camera recording in 2015 would easily be able to reliably be near-flawlessly converted to 8K 1000fps 30 years from now, via a future distant descendant of this Hyperlapse. One may be dreaming at this stage, but this Hyperlapse glimpse is impressive!

[flood]

not bad

this will definitely be interesting as vr headsets become mainstream

about superresolution: i haven't followed the recent developments about it, but honestly i don't think it's that useful. it definitely works for improving the SNR of a video frame but it's not going to do anything close to how it's portrayed in hollywood films. like i think you can get 80% of the result of "superresolution" by using edge-directed interpolation upscaling algorithms on a single frame. (unless your source video is strongly aliased like from a 3d video game without antialiasing)

[Chief Blur Buster]

There could be a lot more intelligent interpolation being invented. Future, as-of-not-yet-invented multiframe upscaling algorithms (scaling that uses data from multiple frames) once sufficient computing power becomes avaialble. Layers upon layers of improvements could theoretically be done with this sort of stuff.

[RealNC]

This sounds more interesting when used to create assets in video games.

[Chief Blur Buster]

Today, this tech is amazing from a computing and programmer perspective.

Yet, this is a very wright brothers hint of the future, a very primitive attempt right now. They only choose a few frames, process over a limited time period, and only stitch from a few frames. And it severely taxes the most powerful systems. That we can do this at all though, on mere teraflops of computing power, is quite impressive. Very impressive optimizing work done!

Tomorrow's algorithms on more powerful computers, could be able to interpolate more perfectly from all available intermediate frames over far longer time periods, to produce more accurate 3D representations off the video, at much higher resolutions with much fewer interpolation artifacts.

There is enough info in the videos for a smarter supercomputer to do a virtually artifact-free interpolation, if the computer smartly interprets the video and self-recognizes interpolation artifacts like a human brain could, and intelligently artistically paint and stitch even higher-def graphics out of low-def material, based on knowing what things look like in real life, recognizing plants, cars, humans, and knowing what are the correct interpolations for things are, based on proper curves and shapes of known real life items, etc. And also re computing all possibilities until it "looks interpretatively flawless" to a smart program. There could be goofs such as missing leaves or blades of grass, but would be interpolatable to ultra-high-def Holodeck quality from just mere 1080p video. See a blurry maple tree, but computer knows what a typical maple tree leaf, branch structure and bark looks like, and interpolates a natural ultrahighdef tree in place of the blurry tree to an exact match with an adobe-blur filter (from all angles revealed in the video), while making sure the ultrahighdef interpolation looks natural and non-CGI to human eyes. Imagine the extra computing power (many orders of magnitude more than Microsoft Hyperlapse) and computing intelligence required for that, maybe in a couple decades from now.

Pie in sky? Yes, now. Tomorrow, no.

"Computer, please create a real-life-quality Holodeck world based on this old low-def video file."

In year 1989, I read a Scientific American article about the theory of 3D video compression, where real life scenes can be compressed into geometric objects, the outlandish idea that a supercomputer converts video into a 3D scene. That was pie-in-the-sky then, pigs-can-fly dreaming. Today, this is finally (sort of) beginning to happen, in a "Wright Brothers" way. Even the people working on H.265 (H.EVC) video codecs suggested that future compression algorithms (e.g. What could become H.266) would need to begin using 3D in order to gain further compression efficiencies! This could finally happen in the 2020s.

[flood]

the real amazing thing is that our brains are able to do pretty much this exact thing, and more, in real time with quite low power

[RealNC]

the real amazing thing is that our brains are able to do pretty much this exact thing, and more, in real time with quite low power

That's because brains don't use floating point math in a binary representation

[flood]

yea so probably it's just an analog computer


I'm curious to see how well this hyperlapse thing works for video game footage.

[RLBURNSIDE]

the real amazing thing is that our brains are able to do pretty much this exact thing, and more, in real time with quite low power

That's because brains don't use floating point math in a binary representation

That's not the only reason, no. Our brains do amazing things, with quantum processing.

Superb article though.

One of the first things we should be doing with multi-frame interpolation as a pre-processing step, is to remove motion blur that's baked into the video. Most video is captured at 24 / 25 / 30 / 50 / 60hz at best, and at the lower end of that, there is a ton of blur. You don't want to interpolate between blurry source frames, but rather extra motion vectors and metadata as you remove the blur itself, so that when you re-imagine / re-render those objects, they are blur free at whatever frame rate you want (or you can re-add extra blur as a postprocessing step, for effect).

Accurate motion blur removal isn't something for the distant future, it has direct bearing on the haloing / shimmering artifacts that you see around moving foreground objects when you enable frame interpolation on your TV or projector (or use something like SVP). If you fed SVP perfectly blur-free 24hz video, it would be able to interpolate intermediate frames with much less visual artifacts.

You can make a blur removal filter as a shader and add it as a pre-processing step to MPC-HC to try it our yourself. The better the filter, the less post-interpolation artifacts you see.

For each motion vector, you'd need a velocity and acceleration vector map too, which should be computed over several frames, or a rolling window. Then with all that metadata, you use a deblur filter with the motion estimate parameters at whatever time fraction your new frame wants to be between the previous ones. Of course your estimation algorithm should reduce itself to a very small error compared to the reference frame when estimating the frame value back again, at the reference time.

There's tons of research in the area of motion blur removal. It's not a simple / trivial task and is not in general a solved problem.

[Chief Blur Buster]

Something even more advanced now -- the conversion of 360-degree videos into explorable virtual worlds:

Videos can now be converted into explorable 3D-rendered worlds

Check it out!