Modify Samsung Gaming Display's PiP Mode

[fliegenklatsche]

First of all I own a Samsung Odyssey G7 32" LC32G75TQSUXEN Display. I was looking for a video mixer, but I haven't found any that can do 1440p240 (probably because of DSC), most consumer models only go up to 1080p60. My idea was to use 2 inputs of my Display and mix the two into one, using some sort of Chroma Key or Luma Key. The Display already has a great PiP/PBP Mode and Onscreen Crosshairs. Is it possible to modify the display/scaler (using hardware or custom firmware for example) to do that?

[Chief Blur Buster]

First of all I own a Samsung Odyssey G7 32" LC32G75TQSUXEN Display. I was looking for a video mixer, but I haven't found any that can do 1440p240 (probably because of DSC), most consumer models only go up to 1080p60. My idea was to use 2 inputs of my Display and mix the two into one, using some sort of Chroma Key or Luma Key. The Display already has a great PiP/PBP Mode and Onscreen Crosshairs. Is it possible to modify the display/scaler (using hardware or custom firmware for example) to do that?

Can you describe your use case more specifically? Twitch-style streaming? Prerecorded video? Entertainment? Monitoring?

I was thinking that you could use a 2nd PC, using two video capture cards, and use the 2nd PC to display the two video windows wherever you want.

Some gamers in esports often use a 2nd PC with a high end HDMI capture card for zero-overhead video recording and streaming (Twitch!). The preview window also displays (essentially a mirrored image). Could use the 2nd PC as your 2nd video source (two video windows that you can manipulate as well, and maybe configure into edgeless, to create a PIP computer)

Unfortunately the high end HDMI capture cards I know of only can do up to 1080p240Hz or 4K60Hz -- not enough bandwidth to do 1440p240Hz. Ouch.

However, my new idea is to make the video mixer computer 1440p240Hz native and run whatever you need at 1440p240Hz. Your PIP window in your 1440p240Hz would be limited (1080p240 or 4K60, scaled to whatever window size you need) but the primary window would still be 1440p 240Hz.

The Elgato 4K60 Pro MK.2 can do 1080p240, 1440p144, and 1080p60.
You can simply install this card on a 1440p240 PC.
- Use the PC to do your 1440p240 deed, it just can't be an external input.
- Use the capture card to PiP or colorkey something else on top of your 1440p240 (the second source just can't be 1440p240, but fortunately this video source doesn't have to be a computer -- could be a video capture of yourself for a talking head on a game stream)

Now you've PiP-mixed two "inputs" (one native computer, one input) that is now being mixed at 1440p240. I do recommend sticking to multiples (if you Elgato capture, stick to 240Hz, 120Hz or 60Hz external sources, so it doesn't stutter when mixed)

_____

This may not be your use case, but if you are a streamer (like Twitch), let's try a potential unconventional use case:

Reportedly (don't vouch me, though) -- the Elgato uses fairly low % of CPU when you're just live previewing (not recording). It's a reverse solution obviously since the Elgato 4K60 is usually used for streaming. But in your case, you're using Elgato to capture your talking head for your stream (in order to solve your 1440p240 problem). So, yes, you're playing on the Elgato 4K60 PCIe-equipped PC connected to a 1440p240 monitor. Might be overkill (could use a cheap USB video capture if your needs are simply 1080p60 talking head, but Elgato hardware does appear to save a lot of CPU/GPU so you can play on the 1440p240Hz PC equipped with the Elgato 4K60 card...). So if you're trying to create an ultra-high-quality stream (game graphics and talking head), playing directly on the PC that contains a high end capture card is an unconventional option that can solve some workarounds such as the need to do a 1440p240fps stream with high quality colorkeyed graphics overlaid in realtime, without much sacrifice to game framerate...

You'd need to use software (existing or write new) that does a frameless live video preview window that you can resize and place anywhere, or dock at any screen edge, while passing through inputs to the window behind, creating a software-based PiP simulator while you play the game. This isn't particular hard (especially if you have the skills to write a barebones DirectShow API style app). Well, more than three orders of magnitude easier than modifying a scaler.

Yes, you can add colorkey and whatnot (for things like talking heads in esports videos) -- no problem. Simple colorkey programming logic here.

This would run concurrently with any 1440p240 software you want to run on that 1440p240 PC, for your 1440p240 needs. Like running a game, of course.

Obviously, you will have to play with all the PiP in the way (e.g. silhoetted talking head of yourself captured by video and colorkeyed on top of your video game, like a Twitch streamer). So you're streaming WYSIWYG, what viewers you see is what you see on your primary monitor too, so you're playing a game through all that clutter. But if that workflow is OK for you, there you go -- your solution. At least it's "honest" -- everything going on your screen is revealed for the world to see.

That being said, if you're a streamer, and doing it that way, you'll probably need to stream 1440p240 via ShadowPlay or another internal streaming mechanism compatible with 1440p240. If you're a streamer, that's a fairly beefy graphics card and PC you will need for a PiP-mixed 1440p240! And you will still lose a few frames per second -- but that's fine for a streamer that prioritizes streaming over max esports performance... And talking heads (yourself on a Twitch style game stream) don't need to be video-captured at 240fps anyway, you can overlay 60fps on top of 240fps.

Other streamers may have come up with something better for a 2nd video source being PiP'd (e.g. using a Logitech 4K webcam or converting a 4K smartphone into a webcam that's routed to a DirectShow API), but this is the best solution I can come up without a 1440p240 capture card.

Mind you, I haven't thought through all possible gotchas, but this is the workflow that I envision you could do. It's not Blur Busters specialty to do streaming rig suggestions -- but offer unconventional workarounds -- there are other forums that may be able to help you figure out things more fully, on missing pieces of the puzzle (e.g. missing software).

________

Paraphasing/rephrasing (if above does not make sense):
Basically, your 1440p 240Hz PiP PC would be holding the high end capture card.
So you run
1 -- whatever content you need at 1440p 240Hz is running
2 -- that computer has a high end video capture card
3 -- a second video source (can be computer or non-computer) which can be 1080p (up to 240Hz), 1440p (up to 144Hz), or 4K (up to 60Hz) and input video capture card.
4 -- now you can use existing software (or create new software) to PiP-mix the two computer streams (one natively running on the 1440p 240Hz PC, one from the second computer that is being live-previewed via the video capture)

This is fine if you're okay with this workflow.
_________

This is much easier than trying to modify a scaler with FPGA VLIW programming in Xilinx IDE...without its original source code! And some scalers are very ASIC (not flexible enough to modify PiP feature).

I cannot think of an easy method that allows arbitrary external-video-source mixing of two 1440p240 sources... yet.

[fliegenklatsche]

Thank you very much for the in-depth response.

Can you describe your use case more specifically? Twitch-style streaming? Prerecorded video? Entertainment? Monitoring?

I want to show additional information like performance overlays, FPS counters or system time without it being visible in a display capture or on a capture card, so my only option is mixing the video signals coming from the GPU.

Mind you, I haven't thought through all possible gotchas, but this is the workflow that I envision you could do. It's not Blur Busters specialty to do streaming rig suggestions -- but offer unconventional workarounds -- there are other forums that may be able to help you figure out things more fully, on missing pieces of the puzzle (e.g. missing software).

Well it's not really related to streaming, I'm not a streamer and I don't plan on being one in the near future.

Basically, your 1440p 240Hz PiP PC would be holding the high end capture card.
So you run
1 -- whatever content you need at 1440p 240Hz is running
2 -- that computer has a high end video capture card
3 -- a second video source (can be computer or non-computer) which can be 1080p (up to 240Hz), 1440p (up to 144Hz), or 4K (up to 60Hz) and input video capture card.
4 -- now you can use existing software (or create new software) to PiP-mix the two computer streams (one natively running on the 1440p 240Hz PC, one from the second computer that is being live-previewed via the video capture)

This is fine if you're okay with this workflow.

Yes I've thought about this too and it would be the simplest solution, however it would be tempering with the 1440p240 system in terms of software(chromakey) & hardware(capture card) which is not possible for my use case. That's why I need a video mixer, but as none can do 1440p240 (yet) I was hoping for the display to be capable of mixing signals.

This is much easier than trying to modify a scaler with FPGA VLIW programming in Xilinx IDE...without its original source code! And some scalers are very ASIC (not flexible enough to modify PiP feature).

Yes, a hardware mod would be the most work and would be pretty overkill, however I would go with it if it's the only solution. I was hoping it would be possible to create a custom firmware based on the newest Samsung firmware (which is public) to add Chromakey functionality.

I cannot think of an easy method that allows arbitrary external-video-source mixing of two 1440p240 sources... yet.

Me neither

[Chief Blur Buster]

Yes, a hardware mod would be the most work and would be pretty overkill, however I would go with it if it's the only solution. I was ho<ping it would be possible to create a custom firmware based on the newest Samsung firmware (which is public) to add Chromakey functionality.

There's a monitor hacking community here, e.g. adding DIY strobed VRR was successfully achieved by a forum member here! But what you are asking for is much harder than that.

It's possible if you have the skills of someone like Zis -- of Zisworks, the creator of the world's first 480 Hz display, www.blurbusters.com/480hz

If you are willing to pay several hundreds of dollars (or a few thousand) plus send a sample display, you could hire someone like him to at least investigate and give a price quote and the requirements --
If you have the experience with hacking monitor firmware (<disclaimer>warning: it will void warranty, and manufacturers don't like it</disclaimer>) as well as resurrecting bricked monitor firmwares -- you could start a journey, but if you are new to it despite having electronics experience, be prepared to spend months on experimentation that might or might not produce useful results.

This stuff deserves, however, to become easier in the future. If you try to pull this off, try to opensource (as much as you legally can) whatever you're doing so that others can become better at hacking monitors! .

Also, especially if it takes you a year or so to finally crack, you might be in a race against time to someone who successfully does 1440p240 video capture -- video capture tech isn't staying static either. Or it might take longer.

Another idea is it might be engineering-easier to try to find a way to video-capture non-HDCP 1440p240. You would just simply create a dummy display with no panel, capturing the 1440p240 bitstream and using your chip logic (FPGA) to do whatever processing you dream. You can buy sample HDMI transceivers that has enough bandwidth. Since you're doing it live, you don't have to worry about compressors (e.g. H.264 or H.EVC) and just reprocess the colorkey on the fly, to create a mirrored display with an overlay. I envision you would create a box that has two HDMI inputs (a colorkeyed input and a overlay input), with some Xilinx FPGA in it running a super simple merge-program, to a single HDMI output. You can HDMI splitter your original output, to split off your unmodified output. Then you have your unmodified output, and your overlaid output. I estimate that this electronics engineering task to be at minimum about 10x simpler than trying to hack a monitor -- but I could be wrong.

HDMI bitstream hacking with your custom electronics circuit is not for the faint of heart mind you, but it currently has a higher likelihood of success than trying to hack a monitor. However, brute skills can compensate (if you already know what is going on inside many monitors, as well as being familiar with decompiling firmwares) so you might actually find it easier. However if you have skills in neither, then you should keep both routes open, and/or work concurrently on them.

[fliegenklatsche]

Yes, a hardware mod would be the most work and would be pretty overkill, however I would go with it if it's the only solution. I was ho<ping it would be possible to create a custom firmware based on the newest Samsung firmware (which is public) to add Chromakey functionality.

There's a monitor hacking community here, e.g. adding DIY strobed VRR was successfully achieved by a forum member here! But what you are asking for is much harder than that.

It's possible if you have the skills of someone like Zis -- of Zisworks, the creator of the world's first 480 Hz display, www.blurbusters.com/480hz

If you are willing to pay several hundreds of dollars (or a few thousand) plus send a sample display, you could hire someone like him to at least investigate and give a price quote and the requirements --
If you have the experience with hacking monitor firmware (<disclaimer>warning: it will void warranty, and manufacturers don't like it</disclaimer>) as well as resurrecting bricked monitor firmwares -- you could start a journey, but if you are new to it despite having electronics experience, be prepared to spend months on experimentation that might or might not produce useful results.

This stuff deserves, however, to become easier in the future. If you try to pull this off, try to opensource (as much as you legally can) whatever you're doing so that others can become better at hacking monitors! .

Also, especially if it takes you a year or so to finally crack, you might be in a race against time to someone who successfully does 1440p240 video capture -- video capture tech isn't staying static either. Or it might take longer.

Another idea is it might be engineering-easier to try to find a way to video-capture non-HDCP 1440p240. You would just simply create a dummy display with no panel, capturing the 1440p240 bitstream and using your chip logic (FPGA) to do whatever processing you dream. You can buy sample HDMI transceivers that has enough bandwidth. Since you're doing it live, you don't have to worry about compressors (e.g. H.264 or H.EVC) and just reprocess the colorkey on the fly, to create a mirrored display with an overlay. I envision you would create a box that has two HDMI inputs (a colorkeyed input and a overlay input), with some Xilinx FPGA in it running a super simple merge-program, to a single HDMI output. You can HDMI splitter your original output, to split off your unmodified output. Then you have your unmodified output, and your overlaid output. I estimate that this electronics engineering task to be at minimum about 10x simpler than trying to hack a monitor -- but I could be wrong.

HDMI bitstream hacking with your custom electronics circuit is not for the faint of heart mind you, but it currently has a higher likelihood of success than trying to hack a monitor. However, brute skills can compensate (if you already know what is going on inside many monitors, as well as being familiar with decompiling firmwares) so you might actually find it easier. However if you have skills in neither, then you should keep both routes open, and/or work concurrently on them.

It's funny, I came to the same conclusion. It's just too much work when there is a much simpler solution, a DisplayPort 1.4 video mixer. As there are none I will create one. I'm currently looking for a PCB Designer. Unfortunately DisplayPort is licensed nowadays

I may update this thread on how things worked out or didn't. Not sure though yet.