Did you use the pixel clock patcher?
DVI requires this. If you exceed 330 MHz pixel clock it will ignore or revert to defaults. Sometimes if you go too far the entire refresh rate gets ignored. Over Displayport you don't need the patcher, but firmware must be V4 or V5 for displayport to work correctly on the XL2720Z and XL2420Z.
Another way to test if VT tweaks are working is to look at http://www.testufo.com/#test=photo&phot ... &height=-1
And set strobe phase to 000 and see how much crosstalk there is at the bottom of the screen. If there is only a few pixels of crosstalk then it's working. If there's several rows of very bad crosstalk then the custom resolution is not working.
Another easy (probably easiest!) way to check if your VT tweak is working is to go into the OSD->system information and see if it says 60hz. if it says 60hz it's working!