NeoTA's* Antialiasing Tutorial

* (formerly av3.)

This tutorial covers some advanced antialiasing techniques. Nemesis's AA tutorial covers the basic ideas. Read this tutorial when familiar with the basic idea of AA. Throughout this tutorial, you can zoom into any image by clicking it (ctrl-click to zoom out).

You must understand this one term to understand the rest of the tutorial..
Opacity: how solid a pixel plotted is. ie the mixing factor of the pixel you plotted and background color. for example:

50% opacity == 50% pixel, 50% bg
25% opacity == 25% pixel, 75% bg

To measure the apparent thickness of an antialiased line,

  1. pick the axis it travels slowest in. in the example, it traverses the y axis slowly
  2. consider each (column or row) along that axis -- how much total opacity? (for example 1.a shows a constant 100% opacity (either 50% + 50% or 100%))
  3. pick the smallest of the totals

Antialiasing improves the approximation of a shape with pixels by simulating sub-pixel precision, so it creates notions of eg 'half a blue pixel and half a yellow one'. The opacity of antialiasing pixels corresponds to how much of the (infinitely-precise) shape falls into a pixel. hence 1b is 50% thicker than 1a.

If monitors displayed intensities in a linear fashion, we could rely on eg 1a. looking perfect just because it is mathematically correct. Monitors respond to intensity in a non-linear way -- giving 50% intensity as input may result in a display that is more or less than 50% intensity, depending on the way the monitor responds to intensity. gamma describes the shape of the intensity curve of the monitor.

usually the gamma value is > 1.0, which results in more precision on the high end (bright) and less on the low end (dark).. so the AA may look better if biased towards the brighter color (this depends on the color though.) This effectively alters the thickness of your lines. light lines will look thicker and dark lines thinner (but because gamma varies a lot between monitors, you will find it difficult to predict exactly how much.)

Smoothing modifies the softness of the line (rather than its perceived thickness. I assume the line segment i'm AAing is 1 pixel or more* longer than it actually is. I discard the most distant AA pixel because of it's near-invisibility, and offset the aa one pixel into the line.
Smoothing tends to eliminate the necessity for gamma-correction. Try to keep some full-opacity pixels as in 3.a.a to maintain clarity
* more == softer.

Staggering compensates for the jaggedness of unevenly patterned lines by introducing a more obvious and harmonious pattern. I place strips of antialiasing at the same angle as the line where long line segments meet short line segments.

Exaggeration: Best for humans to perform AA, as they can prioritize. This section covers some examples of situations where exaggeration improves the clarity of the lines.

Tapering: You may use AA for more than mere smoothing.

Mathematical basis: Text version:

L == length of AA segment (this is the distance between the outer corner
of the rectangle created by this line segment and the outer corner of
the next line-seg-rect)

 You can modify or offset L to achieve smoothing.
S = step size.
calculate S like this:
    S = 100% / (L + 1)
O == list of opacities.
calculate O by:
    current_opacity = 100% - S
    while current_opacity > 0%
	add current_opacity to the end of the list.
	subtract S from current_opacity
for an AA segment 3 pixels long:

L = 3
S = 100% / (L+1) == 25%
O = (75%, 50%, 25%)

I like to soften most AA by adding 1 to L. for this example that works out as:

L = 4
S = 100% / (L+1) == 20%
O = (80%, 60%, 40%, 20%)

Example picture sequence (flat-colors -> shaded -> finished).
if you use a color-restriction you usually must approximate the correct colors for AA using the nearest color in your palette. This picture uses 16 colors.

Written by David Gowers.
Zoomer script by Jeiel Aranal (licensed under the creative commons license)