Professional Quality Redeye Removal
Using Layers

Step 1




This fine gentleman is Gwai-loh, or simply Gwai, as we call him around here. Gwai's got himself a serious case of red-eye. This tutorial will show you how to use layer keying to remove the red-eye in such a way as to leave all the other eye detail intact. We've provided both the original image and the layered result of the tutorial for your convenience. For the purposes of this tutorial, the following should be in effect:

  • You are using WinImages FX R7.50[f] (or later)
  • Area->Pixel Transfer Mode is Matte
  • Area->Opaque Fills is checked
  • Area->Mask with Target Alpha is checked
  • Area->Feathering is set to zero
  • Area->Antialias Area Selections is not checked
  • Area->Autofill is checked
  • Settings->Affect Alpha in View is checked
  • Area Toolbox Complement, Union, Intersection, Exclude are not selected
  • Area Toolbox Edit Area, Color Key, Splatter are not selected
  • Area Toolbox Hold Square, Hold to Aspect, Hold to Source are not selected
Note that you can identify the area toolbox buttons by hovering the mouse pointer over them and observing the name in the status bar at the bottom of the WinImages main window.

Step 2


The first thing to do is convert the atomic (non-layered) image into a layered image. You can do that by clicking once on the image's titlebar and them pressing the L key, or you can use the context menu at the upper left of the image: Layers L. When you do this, you'll get this dialog (you may also get a small dialog that asks if you are sure you want to do this -- if so, agree that you are sure.)

Step 3


Now we're going to create a keying layer. Simply press the Duplicate button in the layers dialog, and you'll have two layers called "gwai.jpg". Then select the topmost layer (at the top of the dialog) and then press Rename Layer. In the rename dialog, rename the layer to "key" and press OK. The results should be as shown above. Now close the Layers dialog using the Done button.


What we need to do at this point is determine the approximate color of the redeye. To do this, go to the standard group and select the Sample tool. In the area toolbox, select the Freehand tool (at the upper left.) Then click once at a midpoint in the red region of the pupil, between the white highlight and the darker area that begins the blue iris portion of the eye. We're trying to get a single color that is decently close to the redeye color -- redeye has the agreeable characteristic that it is usually fairly uniform in color, so this doesn't have to be too precise. Still, try to sample a color that looks in the sample dialog at least somewhat like the color of the redeye in the image. We've overlaid an arrow on the image above so you can see approximately where we chose to take our sample.

Step 4


At this point, switch to the Color Fill tool, also located in the Standard group. You'll see the color we sampled at the lower right hand corner -- sample always sets this color in the fill dialog. Left-click on it to set the fill color, then select Entire Image in the Area Toolbox and left-click on the key layer image. It will fill with the selected color as shown above.

Step 5


At this point, click on the title bar of any layer or the master image and press L to open the layers dialog. Left-click on the key layer, and using the Mode drop-down list, set the key layer to Inverted RGB Key (soft) mode. At this point, if the layer dialog isn't covering the master image, you'll observe Gwai's pupils (and various other areas of the image) become transparent to one degree or another.

The key layer's color is being compared to the color in each pixel, and where the color matches exactly, the image is made 100% transparent. Where it is close to matching, the image is partially transparent, and where it doesn't match well at all, the image is opaque.

To put it another way, alpha transparency is generated that corresponds to the color match between the key layer pixels and the image layer pixels. At the moment, that transparency extends from top to bottom, so the master image actually becomes transparent. We'll be changing that shortly.

Step 6


Select the Make Alpha tool from the Alpha group. Set it up as shown above. Select the Ellipse tool from the Area Toolbox. Pull an ellipse around the pupil area in Gwai's right eye (on the left) as shown in the Master Image.

Important to note: The ellipse should lay approximately along the center of the iris, well into into the area where Gwais' blue iris color shows. You are not trying to precisely select the red pupil, you are trying to select beginning in the blue iris outside the pupil.

When you let go, there will be no change. That's what we want -- Master images can't be changed. But as a side-effect, WinImages FX will remember the location of the selection you just made. That will turn out to be very useful!

Step 7


In the Area Toolbox, select the ReDo tool. This tool uses the location of the previous area selection to place the next operation. Now click on the key layer; all of the color key will be alpha-masked to off except the region around Gwai's right eye, as a consequence of the area selection you made previously on the Master Image.

This is a very powerful general technique -- you can use one image to locate something precisely on another, even if the target image has no detail of its own, as is the case with our key layer.

Step 8


Now we're going to do the same process again, except with the other eye. Select the Ellipse tool and in the Master Image, select just outside Gwai's left pupil (on the right.) Release the mouse, and nothing will happen, as before, but the area selection location and fill details are now set to the new area.

Step 9


Now, in the Make Alpha dialog, change the mode to Add to existing alpha. This will allow you to create new alpha information without disturbing the already existing alpha outside the selected region. Select the ReDo tool, and left-click on the key layer.

At this point, you should see that only Gwai's eyes remain transparent. That's because we used the key layer's alpha channel to turn off the keying effect where we're not interested in using it. This in turn restricts the keying to just the area where the key is designed to work -- the eyes.

Step 10


The keying effect ends abruptly at the edge of the two elliptical area selections, because the alpha channel of the key image turns off abruptly. Above we show how to use the Soften Alpha tool (also from the Alpha group) to change the hard edges to a gentle edge in one operation. use the Entire Image tool in the Area Toolbox to apply Soften Alpha. We used a setting of three; you might want to experiment with this.

Step 11


Now we want to create a layer to hold the black that we expect to see in Gwai's pupils. Left-click on the titlebar of any layer or the master image, and press L to open the Layers dialog. Left-click on the gwai.jpg layer, and then left-click the Duplicate button. A new layer, also called gwai.jpg, will appear at the bottom of the layer stack. Left click on it, and use the Rename command at the left of the layer dialog to change the name of the bottom layer to black. Close the Layer dialog using the Done button.

Step 12


From the Standard group, select the Color Fill operation. If black (RGB 0,0,0) is not an available color, create it now. Select black. Select Entire Image from the Area Toolbox. Click on the black layer; it will fill with the black color.

You'll note that Gwai's eyes will still remain transparent even though we have created a black layer; that is because the key layer's effect extends all the way to the bottom of the layer stack, which means it also extends through the black layer.

Step 13


The last thing to do is to open the Layer dialog by clicking on a titlebar and pressing L, then left-click on the key layer and lastly left-click on the Localstack box so that it becomes checked.

This tells the key layer that we only want it to affect the layers immediately below it, and so the key no longer extends down through the black layer, which now is opaque due to its own alpha channel, which is entirely opaque.

Intermission: The Result Thus Far


Not too bad. In fact, this is already better than you can do using most other approaches. It's not as good as it could be, though. Taking a close look, we can see that Gwai's eyes don't quite have the same shade of redeye. This means that the optimum key for one eye should be a bit different than the other eye. If you look at the adjusted result of his left eye (picture on your right, eye on the right of the picture) you'll see a hint of deep red still remains. Let's get that handled.

Step 14


Again, we need to determine the approximate color of the redeye, this time for the left eye. To do this, go to the standard group and select the Sample tool. In the area toolbox, select the Freehand tool (at the upper left.) Then click once at a midpoint in the red region of the pupil, between the white highlight and the darker area that begins the blue iris portion of the eye. We are still trying to get a single color that is decently close to the redeye color. Try to sample a color that looks in the sample dialog at least somewhat like the color of the redeye in the image. We've overlaid an arrow on the image above so you can see approximately where we chose to take our sample.

Step 15


Go to the Area Menu and un-check Opaque Fills as shown above. This will cause the Color Fill tool, which we're about to use, to not modify the existing alpha channel when the fill is created.

Step 16


Go to the Area Toolbox and select the Ellipse area tool. Go to the Standard group and select the Color Fill operation.

Ellipse fill generously around the left eye key region. This changes the key color, and hence the key itself. Watch what happens to the left eye in the master image when you do this if you have enough display real estate to expose the master while you do this step. Note that the key layer's alpha channel does not change as a result of the color fill; that's because of the current Opaque Fills setting.

Step 17


Above, we show the change to the key layer after having been adjusted to the left eye color.

Be certain to turn the Opaque Fills checkmark back on at this point, or you're going to be pretty confused the next time you try to work with an alpha channel!

Keying Complete!


Here we have placed the gwai.jpg layer, which contains the original, unmodifed image, next to the master image, which contains the desired effect, that of redeye removal.

Observe how the reflective detail remains in the pupil; and how the variations of those details keep the eye looking realistic, alive and registered to the original illumination environment. This result is quite a bit better than you can achieve using a simple area selection and color fill or RGB re- balance.

Aside from the very high quality of the redeye removal itself, there are several important things to note about this process.

  1. Because the original image is not modifed in any way (as you see above, it's still in layer 2, untouched), and because the modifications are consequences of layer effects and settings, you can adjust the keying effect or start over completely at any time.

  2. You will never permanently lose any detail in the eye, because the source image is actually unmodified. That allows you to intelligently recover from any errors you might make, and it allows you to modifiy things like the alpha mask of the key and so on without risking the image. To demonstrate this, try playing with the layer settings; in particular, adjust the Factor for the key layer. 100 is the default (it was set when the layer was created.) Try 75. Try 125. Try 50 and 150. Use the TAB key to force the image to update after you've typed the number into the Factor field. See what happens to the redeye removal? Is one setting better than another or than the default setting of 100?

  3. You can adjust the color key itself independently of the rest of the process once you have the alpha mask built. Steps 14 through 17 demonstrate this specific advantage. You can go further, though. You can sample the pupil in multiple places and make a key that varies as does the redeye component using range fills. It's quite a bit of effort to go to in order to get just a slightly better result, but the point is, because determining the color portion of the key is entirely separate from the rest of the process, there is no limit to what you can do with it.

  4. You can do additional mask work by using the Make Alpha, Add to existing Alpha and Subtract from alpha channel modes. If you played with the color key as described in the previous paragraph, you probably noticed that the ellipse selections extend beyond the top of the eye and keying can visibly affect Gwai's brow, which is undesirable. A fun exercise would be to use Subtract from alpha channel to adjust the alpha mask of the key layer to eliminate that effect. Hint: Make the selection on the master, then use ReDo to apply it to the key, just as we showed you in the tutorial for the elliptical selections. It is also worth noting that if we had used a more sophisticated area selection such as the Polyarc tool, this issue would not exist at all.