Sane aq settings for h264 encoding

[BasharOfTheAges]

So, I'm removing some minor banding, made more obvious with an upscale from 480p to 720p (not in the mood to debate the need of it). I've followed advice from a previous encode where I had horribly strong banding, and added grain and whatnot, but when it came to the second part of the advice, adjusting the aq setting for the encode, I found I didn't really have a grasp on the scale that was being used and where the benefit/drawback tipping points were... or even what the drawbacks to high values might be. Anyone have insight?

[mirkosp]

Values lower than 1 help compression, but if too low introduce obvious banding. Values greater than one will preserve fine detail, at the cost of even huge bitrate increases. We're talking more or less of actual bitrate doubling from 0.7 to 1.4 (which makes sense, I guess). Generally a 0.8 value is very safe, won't inflate the filesize and overall the banding should be absent. The default 0.6 for the animation preset is more meant for western animation, which has much less gradients and details than eastern animation, as it uses more paint bucket-like fillings.

[BasharOfTheAges]

Exactly what I was looking for, thanks.

[Mister Hatt]

If you have heavy handing (FMA:B, Cobra, etc) then AQ of up to about 1.2-1.3 is necessary. Anything more than that and you want to look at dithering and 10bit encoding tbqh. The general way it works is as mirko said, where bits from flat areas are relocated to areas of higher detail. The strength is a scale of 0-3 on how aggressive that is. Note that you can use zones in x264 to apply different strengths to different sections, IIRC anyway. Once you get to high AQ strengths however, you tend to make flat areas blocky, and at lower strengths you'll have the opposite effect of detailed areas getting squandered for useless chroma gradients.

[mirkosp]

Note that you can use zones in x264 to apply different strengths to different sections, IIRC anyway.

No, you can't. As D_S stated himself, the way aq works is incompatible --zones and would require its own --aq-zones command if anything.
Related: http://forum.doom9.org/showthread.php?t=162268

[Mister Hatt]

Oh right. When I wrote that I was like "now AQ either works with zones or is the one thing that doesn't… well nobody would zone crf, must be aq" and then I remembered a few hours later w. Also I totally wrote IIRC, first "I" of which is "if" :3

[mirkosp]

Heh. Also you CAN zone crf anyway... >_>

[Mister Hatt]

Yes I know. What I meant was the main reason for zoning was either CRF or AQ but I forgot which.