Excessive Banding and Noise After Compressing

[Melichan923]

Hey all,

I ripped all my footage, cleaned it up in Avisynth, and my footage looks great. I sent it to a few conventions and gave them either lossless copies or converted to MPEG-2 and even though they were large files, they looked good.

But when I compress my video to x264 MP4 and XviD AVI I see excessive banding and noise and discolorations in especially the dark places. I spent a long time cleaning it up before editing and have even tried gradfun2db() and various denoisers such as deen before I compress to XviD/x264 so I'm not sure what's causing it. Is it the way XviD and mp4 compresses footage?

Here are some example pictures. But please beware of SERIES SPOILERS for the first season of Clannad and Kanon 2006 which provide the best examples for darker areas:

Spoiler :

The before pictures are the Huffy and the after pictures are the compressed XviD.

Before:

After:


Before:

After:


Before:


After:


Before:

After:


Before:

After:

Maybe those scenes just pop out to me because I've looked at the HuffYUV version for so long. Some scenes are worse than others and some scenes even look practically as good as the original, but I'd love to make the darker scenes above look much better if I can. I am using VDub to convert to XviD: BVOPs set to 1, Quantizer ratio set to 1.00 to boost quality slightly, and on the quantization tab I alternate 1 and 31. For x264 I am using Zarx264GUI latest version: The preset is set to insane, the Quantizer is set all the way down to 16, and the mode is constant quality. I've tried it compressing for the ORG with and without gradfun2db and gradfun2dbmod() as well as with various denoisers. The best results I got were when I had gradfun2b() and that only. the x264 compresses better and looks a little better than the XviD, but not quite as good as I want it to be.

Anything I can try to do next, like any settings I can mess with or new filters I can try? Thanks.

[Qyot27]

I forget what the settings for Insane in Zarx264gui are concerning deblocking/in-loop, but if you set them to 1:1 (if they're not already), that might help reduce the noise in the H.264 copy.

With Xvid, several things that spring to mind: enable Trellis (on the Quantization tab), set the maximum I-frame interval to 50, rather than 300, and if possible, use 2-pass encoding. I typically have Xvid set to use 4000kbps, which in conjunction with the other settings, reaches saturation well below that level and virtually never gets close to it. The B-frames may also be contributing to it - you could try disabling them, if you don't mind the increase in filesize.


I had to crank my monitor's brightness all the way up to see some of the issues really clearly (the exception being the shot of Mai in the corridor; I could see the noise there). Then again, I prefer 'dark' displays in the first place (Contrast=100%, Brightness=60%, using the 9300 color setting - I'm also not using an LCD).

[Kariudo]

I can't see anything wrong with your caps there...I think it may be just Vdub's output color depth setting causing the gradients in the preview window

[Melichan923]

Wow, you guys are fast.

I forget what the settings for Insane in Zarx264gui are concerning deblocking/in-loop, but if you set them to 1:1 (if they're not already), that might help reduce the noise in the H.264 copy.

With Xvid, several things that spring to mind: enable Trellis (on the Quantization tab), set the maximum I-frame interval to 50, rather than 300, and if possible, use 2-pass encoding. I typically have Xvid set to use 4000kbps, which in conjunction with the other settings, reaches saturation well below that level and virtually never gets close to it. The B-frames may also be contributing to it - you could try disabling them, if you don't mind the increase in filesize.

I believe the settings for Zarx264gui are already set to 1:1. I checked the XviD settings and I changed I-frame interval to 50 from 300 and turned off B-frames. I am encoding the first pass now. I will let you know how it goes. I couldn't find the option to set XviD to use 4000kbps though.

I had to crank my monitor's brightness all the way up to see some of the issues really clearly (the exception being the shot of Mai in the corridor; I could see the noise there). Then again, I prefer 'dark' displays in the first place (Contrast=100%, Brightness=60%, using the 9300 color setting - I'm also not using an LCD).

My monitor is an LCD, so maybe some of the noise might be a bit more accentuated. The worst ones on my screen are the ones of Nagisa standing on the stage (the walls behind her show a lot of banding) and the picture of Yuuichi looking at Shiori (discoloration and banding on both their faces along with noise). I've never actually adjusted the brightness and other settings that much on my monitor. I'll try it out.

I can't see anything wrong with your caps there...I think it may be just Vdub's output color depth setting causing the gradients in the preview window

You got me very happy there for a minute because I thought that might be the case, but I watched them in my video players (I use Windows Media Classic and VLC) and the issues are still there. Awww.

First pass is encoding now with the new settings and I might try out the second pass as well. I'll let you both know of the results. *Crosses fingers* Thanks. I just hope it isn't too big of a file size. I always have a bit of a file size problem with XviD.

[Melichan923]

Well, as I expected the first pass of the video was too big for the org, so I did a second pass and set the target to about 85% and it's now under 100MB.

The result after the second pass is about the same size and quality as the first XviD. I'm not sure what I am doing wrong. I did discover this though:

Huffy:


New XviD:


See the floor? The detail in the floor is almost completely gone in that scene! It seems as if XviD is transforming some of the detail into blocking and banding. The dark scenes is where it really shows up. Any more suggestions will be greatly appreciated. I'm willing to make a few clips to put up too, to see if it really is my monitor.

[Kionon]

See the floor? The detail in the floor is almost completely gone in that scene! It seems as if XviD is transforming some of the detail into blocking and banding. The dark scenes is where it really shows up. Any more suggestions will be greatly appreciated. I'm willing to make a few clips to put up too, to see if it really is my monitor.

Wow. You really did lose the tatami weave. Is there any reason why XviD would be failing to account for the tatami weave? Can you post a screen cap of the first pass, I would be interested in seeing how we get from Picture 1 to Picture 2, because that is, indeed, a huge difference.

[Melichan923]

See the floor? The detail in the floor is almost completely gone in that scene! It seems as if XviD is transforming some of the detail into blocking and banding. The dark scenes is where it really shows up. Any more suggestions will be greatly appreciated. I'm willing to make a few clips to put up too, to see if it really is my monitor.

Wow. You really did lose the tatami weave. Is there any reason why XviD would be failing to account for the tatami weave? Can you post a screen cap of the first pass, I would be interested in seeing how we get from Picture 1 to Picture 2, because that is, indeed, a huge difference.

Sure I can. Here's a pic of the first pass:



I don't see a very big difference between the first and second pass. A tad bit more detail is lost during the second pass, but I noticed it only after I looked a few times.

[Kionon]

Yep. That sucks. Also, here:

[Melichan923]

Oh thanks, Kionon. Is that what Qyot27 meant by using 4000kbps? My target was a lot higher. It can't hurt trying that setting out though. If it helps, here are screenshots of most of my settings for the last encode:

Spoiler :

If I'm lucky, maybe someone can point out something that I missed.

[Scintilla]

Oh thanks, Kionon. Is that what Qyot27 meant by using 4000kbps? My target was a lot higher.

No it wasn't; "target size" != "target bitrate".