Ugly edges

[Mastamind]

This has bugged me for awhile. Basically when I mask out a shape in vegas and add certain effects to it, an ugly outline appears:

Here's the same bit of the hair masked, without the brightness change:

Spoiler :

This happens with pretty much every color correction tool except for saturation adjust... how can I get rid of it?

[Enigma]

Seems fine to me...

[NS]

You need to work with color correction and try feathering a little

[mirkosp]

Work the opposite way. Instead of masking what you need to tweak the brightness for, mask the rest and color correct the hidden layer with what you need. That way it should work, although it feels like a pretty dumb and just temporary fix.
If you can specify an order to do things (eg: color correct first, and THEN mask) it could still work, although I think the way vegas handles masking doesn't help much without a project nesting or a render (maybe).

[Mastamind]

You need to work with color correction and try feathering a little

Feathering only makes it thicker

Work the opposite way. Instead of masking what you need to tweak the brightness for, mask the rest and color correct the hidden layer with what you need. That way it should work, although it feels like a pretty dumb and just temporary fix.
If you can specify an order to do things (eg: color correct first, and THEN mask) it could still work, although I think the way vegas handles masking doesn't help much without a project nesting or a render (maybe).

For some reason, even that doesn't work. And AFAIK you can't change the order.
Here's another example:

Here, the normal layer is on top with a negative mask, and the darkened eyes are on the bottom. The eye on the right has a slight feather, and you can see how that turned out

[mirkosp]

Yeah, but you didn't get the edge like before. The thing there is that by darkening too much you made it stand out, and that's why you got the issue.

[Mastamind]

The edge is slim, but it's still there. I did the same thing in AE and it came out clean.
I could always just render a layer with the color correction and mask that out, but it would be inconvenient and won't exactly be solving the problem.

[mirkosp]

You're best off just doing it in ae at this point... vegas isn't precise enough, apparently.

[Mastamind]

Problem is I only have the trial for AE
Guess I'll have to do it the long way unless I end up buying after effects. Thanks for the suggestions though