Photoshop technique for masking with action scripts

[DriftRoot]

PLEASE NOTE that I no longer endorse the Photoshop masking method referenced below. Although it still works, more recent versions of Photoshop offer different and, IMO, far better ways to create masks for your videos, ways which don't necessarily have anything to do with action scripts. The basic use of an Action script in Photoshop remains a valuable tool, however, so I hope people still find this guide useful. (PM me if you want detailed instructions for my current tried and true Photoshop masking method). I am always happy to answer any questions about Photoshop and help with projects or techniques people are having trouble with, so please don't hesitate to PM me for assistance anytime!

In the (locked) thread on masking in this forum there is a very old, but very thorough, step-by-step guide to creating masks in Photoshop. There is a better way, one that alleviates about 80% of the work inherent in that method: you let Photoshop make a lot of the mask for you by creating an Action script which completes much of the process via automation. The only work required on your part, once things are properly set up, is defining the transparency in each frame.

Creating custom action scripts in Photoshop is not at all difficult. Basically, you just need to have a tried and true method for achieving the results you want, then record that method and play it back whenever you need those results. The masking method suggested in the thread mentioned above works nicely as an action script if you alter the progression of steps just a tad and take some extra precautions preparing your files for processing.

I'm going to assume people know their way around Photoshop for this:

1. Bring an image you wish to mask into Photoshop, be it a frame from a video editing program or something from somewhere else.

2. With the pen tool, create a workpath around the area to be masked, then turn it into a selection. As described in the masking thread, at this point "marching ants should be marching around the borders of your cutout." PEN TOOL? you shriek in horror…yeah, pen tool. Immerse yourself in my easy-to-follow, overly thorough guide to getting over pen tool fear and hatred and then go on to step 3.

3. Save this selection with some generic, easy-to-remember name that will never be used for another selection.

4. Save the file as a Photoshop (PSD) document, making darn sure to label it appropriately so that, for example, if it's the first in a series of 52 frames you intend to mask, you know it's frame one.

5. Make your action script (this can actually be done ahead of time, it doesn't matter as long as it gets done before you go any further than assigning a selection area to a whole bunch of files. I also recommend you do this before creating selections for 51 more frames only to find out that your selection-creating method isn’t going to cut it).

To make an action script, again, all you need to do is record the steps you take to achieve certain results. There's no Visual Basic or Java involved - most graphic designers abhor math, after all. Photoshop comes with a fantastic script-making application just for this reason, which is actually loaded up already with a lot of useful stuff (more details can be found in Photoshop Help, btw).

How to make an action script:
1. Practice what you plan on doing to create a mask; in other words, make sure your method works. I’m harping on this because there can be unforeseen glitches, wrong choices and oversights that come out of nowhere and ruin everything, no matter how many times it's worked before. Believe me, this is one reason I LOVE scripted masks, I never ever EVER will screw it up again. Of course, also be sure that your mask produces the results you want in your video editing program.

2. When you’re ready to record your script, open up whatever that file with the selection was that you saved as a PSD.

3. I’m assuming you have a default Photoshop workspace on deck, so your Action palette will be under your History palette. Bring it forward and take a look at what’s there – lots of scripts.

4. At the bottom of the Action palette, click on the “create new action” icon (looks like the new layer icon). I personally go the further step to create a special folder/set for custom actions, but that’s just because I’m hyper-organized – or pretend to be.

5. A new window will open, asking you to name your action and listing a few other options. Name this action “AMV Mask” or some such thing that will be easy to identify and then hit “Record.” The little circle on the Action palette will turn red. Don’t freak out, this isn’t like a screen capture program that’s going to stalk your every movement. Nothing’s getting recorded until you give Photoshop a command – you can look under menus, scroll around your image and wander away from your computer for hours if you want.

6. If you’re using the masking guide from the other thread, you’re going to be starting the script from Step 6 with one major difference. Instead of using the magic wand tool to assign a selection area, you’re going to use that selection you saved. So, for the first action in your script, load the selection, making sure “New Selection” is checked off in the dialogue box.

7. Continue with your mask-making plan, substituting your saved selection whenever necessary for any magic wanding it mentions. (Magic wand bad…BAD). By the end of the process, you should have a nice mask sitting in front of you.

8. When you’re done, save the file in whatever format you intend on bringing into your video editing program – JPEG, PSD, etc.

9. Close your JPEG, PSD, or whatever file your mask resides in.

10. Stop recording the script by hitting the little square on the bottom of the Action palette.

11. Assuming everything went ok and you were happy with the mask-making process you just went through, test your script out on that original PSD file you just used to make a mask. For beginners, the most obvious way to do this is to just revert/reopen that original PSD, go to the Action palette, select your mask script and then hit the little triangle at the bottom of the palette. Before your stunned and grateful eyes the entire masking process should play out in warp speed.

Now, back to the grunt work. Chances are pretty good your mask script didn’t turn out right because it’s quite a fussy process and there’s all kinds of things that can go wrong. Identifying where the problems lie and repairing the damage can be tedious, but so is masking frames by hand and at least this way there’s an end in sight! Fixing a script can be done either by completely recording the process all over again or by going in and selectively deleting the bad steps and inserting/recording new ones. See the Photoshop Help section for more on this.

6. If you’re really sure your mask script works, that it gives you exactly the results you want, go ahead and create selection areas for all the other frames you need to mask following steps 1-4 above. Here’s the trick that makes this all work, though: you must save your selection under the same name each time. Why? Because your action script has been programmed to load a selection with a certain name, so as long as that selection exists in your numerous PSD files, it’s going to be able to work with it.

7. Now for some really nice stuff that will save you even MORE time! Instead of opening each of your 52 selection-inclusive PSD files one after another and running the script on them one after another – or even opening them all at once and doing so – let Photoshop handle this.

a. Close all your PSD files.

b. If you’ve not already done so, move those PSD files into their own folder somewhere.

b. In Photoshop, go under File>Automate>Batch

c. Select your script from the drop down menu and under Source choose “Folder.” A new menu will activate, and “Choose…” will let you point Photoshop towards that folder where all your PSD files are waiting.

d. Mess around with the four options under “Choose…” if you want.

e. Assuming your action script included a Save As and Close command at its end, you can leave the Destination field set to “None” or assign a folder into which all these masked files should be put. If your script didn’t end that way, you can explore the various options here for outputting your masked file. Just be careful NEVER to save over your original PSD file.

f. When all your options are set the way you want them, just hit “OK.” And as Emeril would say, BAM! Photoshop will open, process, save and close all your target files for you. You’re done!

This scripting process can be customized for any other mask-making method that can rely on a constant (in this case, your saved selection) across multiple files.

Now, why did I emphasize saving your PSD files for a rainy day? Because mass production tactics like this do not allow you to double check on the conveyor belt whether everything is coming out pretty.

When the script is done processing your files, you need to take a good hard look at each one to be sure the masks appear the way you intended. If some don’t, go back to your original PSD files and make the necessary changes, toss out your saved selections in favor of the new mask-to-be area and then run the script again. I generally do not run the mask script on more than 10 files at a time unless I’m 100% sure what it will produce is exactly what I’m after (like each frame has the exact same selection area and I know that area is perfect). Fixing the selection areas on 10 files can take awhile, depending on the extent of work to be done, and since you’re going to have to recheck the masks on those AGAIN…well, I’d rather deal with 10 files at a time than 52.

Once again - sorry but it’s true - the pen tool is the only thing you should ever be using to create masks. It is extremely fast and easy to fix bad masks if you’ve used the pen tool to create a selection area (and did other things you were supposed to, see my tutorial for the nitty gritty) – if you didn’t and went the route of the magic wand or lasso, you’ve made a lot of unnecessary work for yourself and scripting masks is not going to be as painless as it should be.

It's also worth mentioning that you may need to make a new mask script every time you tackle files from a different source - there are some variables in the process that sometimes throw things out of whack. What works on one kind of image may not work for another.

[Bauzi]

8. When you’re done, save the file in whatever format you intend on bringing into your video editing program – JPEG, PSD, etc.

1.If the program can work. You should ALWAYS (in my eyes) use the psd layers directly. I remeber that I had for example a psd plug in for Vegas.
2.If there is no psd use png. Why? On one hand... psd is looseless and on the other, it saves the whole transparency and alpha channel.
png took the best out of other stuff like jpeg or gif. It´s an awesome mix.

If you have to key out some colour on the prepared jpeg files, it is likely to get poor results. It´s powerful to blur a bit the edges, but this is only possible with a format that supports RGBA. What png does and jpeg not.

3. Save this selection with some generic, easy-to-remember name that will never be used for another selection.

Good that somebody talks about this trick in a guide
Very good for some moving action ^^

How to make an action script:

cool. Thanks. Good to know.

[Scintilla]

8. When you’re done, save the file in whatever format you intend on bringing into your video editing program – JPEG, PSD, etc.

1.If the program can work. You should ALWAYS (in my eyes) use the psd layers directly. I remeber that I had for example a psd plug in for Vegas.
2.If there is no psd use png. Why? On one hand... psd is looseless and on the other, it saves the whole transparency and alpha channel.
png took the best out of other stuff like jpeg or gif. It´s an awesome mix.

I feel compelled to put in a good word for the TIFF format, as it also can retain the alpha channel, it also can encode losslessly, and it's got wide compatibility. Works like a charm going from The GIMP to Adobe Premiere Pro.

[DriftRoot]

I didn't want to start listing all the various file formats for all the various scenarios out there...people should know what they need to use/want to use if they're going to be tampering with transpency anyways. My little disclaimer at the beginning about assuming folks know their way around Photoshop was meant to cover this, as well as more general things like "How do I save a selection?"

A generic statement saying do NOT get rid of the files which are capable of retaining workpaths and selections would have prompted questions about what kind those are, leading into more questions about why you would choose one over the other, what reaction various video editing programs might have to them, etc. You guys are picking up the slack, though, so it's ok.

I apologize if I was aiming more for the lowest common denominator by pushing PSD files, but they can do it all, whereas a lot of the others can't. If you REALLY don't want to/can't use a PSD file format for retaining your selections and workpaths, then use TIFF. Just don't try to do so in a JPEG, because it is absolutely not going to work.

And if you're bringing PSD files into your video program...well, you're either making a mistake or up to something above and beyond a simple masking method like this.

[DriftRoot]

2.If there is no psd use png. Why? On one hand... psd is looseless and on the other, it saves the whole transparency and alpha channel.
png took the best out of other stuff like jpeg or gif. It´s an awesome mix.

Forgot to point this out specifically, good thing I am now:

No do NOT use PNG if you're going to try and save your selections and your workpaths, which is what this entire mask-scripting method hinges on. PNG is not a replacement for PSD. PNG file formats do contain transparency information, but (in Photoshop, at least, I don't use anything else but I would suspect it's the same across the board) they cannot retain selection or path data.

[josefina]

On Photoshop elements 7 is it possible to colour in a hand drawn picture? I've recently got photoshop elements and I want to try colouring in one of my hand drawn anime pictures. I know i haven't got photoshop Cs4 which is recommended but i couldn't afford it. So would someone give me some tips on how to colour it in so it becomes like a professional picture?

[Neverend]

So are you saying that one masked frame (only 1?!?!) takes around 10 min?

[DriftRoot]

So are you saying that one masked frame (only 1?!?!) takes around 10 min?

I am saying that a 10-minute mask in Photoshop utilizing this old masking technique plus an action script is not out of the question. Everyone works at different speeds, everyone has different expectations for the end result and not all scenes are masked the same way.

For the record - thanks to advances in Photoshop - my preferred method of masking has changed significantly and does not rely on a complex action script to drastically speed up the process. The masking technique found in the old forum is exactly that, an old technique: slow, cumbersome, imprecise and inflexible. What's nice about Photoshop (and other Adobe programs) these days is there are so many ways to accomplish the same task that something like mask creation is largely a matter of personal preference. There is no right or wrong way to go about it, necessarily, just what works best for you. If that old method works for you, keep at it. If you want to try something different, go for it.

I would hope, however, that people can still see the value of an Action script for all kinds of purposes and be able to use that knowledge to their advantage should it ever become appropriate.

[Castor Troy]

So are you saying that one masked frame (only 1?!?!) takes around 10 min?

My record for masking one complicated frame is about 50 seconds. Sometimes even less (if it's a closeup of a foot or something).

[DriftRoot]

So are you saying that one masked frame (only 1?!?!) takes around 10 min?

My record for masking one complicated frame is about 50 seconds. Sometimes even less (if it's a closeup of a foot or something).

Tool? Technique? Results? Don't go flinging out numbers like 50 seconds without explanation!