One of the key advantages of working with images on the computer is that various different sorts of visual material can be combined and used as part of the one medium. Video can be combined with hand-drawn animation and synthetic 3D shapes. The term 'motion graphics' is used to describe all these under one name. Conventionally motion graphics has been portrayed as something requiring expensive, high-end computers and specialised software. With the growth of Free Open Source Software (FOSS), however, this has all changed, and many of the techniques and tools used in professional studios can be acquired and used by everyone. The emphasis upon programming in Free Software, and designing things to enable people to change and alter code as they feel necessary, also brings with it greater artistic possibilities. Through learning to create your own motion graphics programs, you also acquire the ability to shape your own personal artistic style within those, rather than being trapped in the aesthetics and styles of someone else's software or plug-in effects package.

Motion graphics programming can be a lot easier than people first assume. This tutorial will lead you through the basics of creating your own motion graphics programs in a series of hands-on examples that can be altered and played with. The best way to learn is by creating your own projects, and the examples given here can all be adapted to become the basis of experiments, sketches and full-blown animations or videos.

Python

The programs are all written in Python. Python is a relatively simple-to-learn programming language which is extremely popular in the Free Open Source Software community. Lots of software libraries and applications support the ability to work with them in Python. This opens up a huge range of possible tools and resources that can be combined in ways conventional self-contained desktop applications cannot.

There are also plenty of online tutorials and guides for Python, many of which are listed on the main Python website:

All films, animations and videos are made from series of still images that are run together to produce the sensation of movement and change. We can therefore use tools designed for creating and manipulating still images to produce moving ones. This is how early animation and computer graphics developed, by generating sets of still image files. Even though faster computers have made "real-time rendering" more popular, most high-end animation, films like "Shrek" and "Toy Story", are created from building such sets of still frames. This approach gives us the greatest control over content and image quality, and is the approach we will be using in this tutorial.

Before we start looking at the code, lets run one of the examples. Enter the command:

./example1a.py

You will see the message in your terminal:

and then:

until a second or so later:

This tells us the animation frames have been created.

If you type:

ls renderframes

You will see a list of the all the individual frames the program has created:

listing of files created by example1a.py

If you want to view these as a set of stills, you can do this in a tool such as "Eye of Gnome", a simple image viewer:

eog renderframes/example1a_*.png

viewing frames in Eye of Gnome

Now enter the command:

ffmpeg -i renderframes/example1a_%3d.png -b 98000 video/example1a.avi

You will see something like this in your terminal:

This is ffmpeg generating a video from the still frames. The finished video file, 'example1b.avi', will be stored in the 'video' directory.

If you have already generated a version of the video, and run the command a second time, ffmpeg will ask if you definitely want to overwrite the existing file:

Type 'y' for yes, or 'n' for no. If you want to avoid this, and always write a new version, add '-y' to the ffmpeg command:

ffmpeg -y -i renderframes/example1a_%3d.png -b 98000 video/example1a.avi

To view the video with MPlayer, enter:

mplayer video/example1a.avi

This will open the video in a window and play it through once, then close the window. You should see a short animation of randomly changing rectangles on a white background:

If you want to loop the video enter:

mplayer -loop 0 video/example1a.avi

The number after '-loop' tells MPlayer how many times it should loop, if this is set to 0, it will loop continually until you manually stop it. To stop the loop press the escape key. You can also run the video in fullscreen mode by adding the '-fs' option:

mplayer -fs -loop 0 video/example1a.avi

MPlayer can be run with a playback controller by calling 'gmplayer' rather than just 'mplayer':

gmplayer video/example1a.avi

The look of the controller may vary depending on the skin-style used:

When we start working with the examples, the main commands you will be using are:

run program:

./example1a.py

compile video:

ffmpeg -i renderframes/example1a%3d.png -b 98000 video/example1a.avi

view video:

mplayer video/example1a.avi

As mentioned earlier, you can repeat a previous command by pressing the up arrow key, which will cut down on the typing you need to make. Each of the example files also contains versions of these commands that you can paste into the terminal.

Working with the code

Now open the code for example1a.py:

scite example1a.py &

Include '&' sign after the filename to tell the terminal to run Scite as a separate process from the terminal itself, so we can continue to use the terminal to do other things whilst the text editor is open. If you were able to see the entire file on one screen you should see something like this:

Starting from the top of the page, scroll through the file to see the different sections of the code matching this image. These sections are:

A - this tells your computer to run the following program in Python:

#!/usr/bin/env python

B - a GNU General Pubic License (GPL) declaration, to show that this is Free Software,

C - information about the program, who wrote it, what it does, and how to use it.

D - these lines tell Python to use the specified external modules and libraries, 'Image' and 'ImageDraw' are part of PIL, 'random' allows us to create random numbers, and 'sys' gives us access to parts of the operating system.

E - constants, these are some fixed variables that are used in various places in the program, such as the width and height of the animation images, their background colour and the name used to store the animation files.

F - functions, these do the actual work of creating the animation. We are just using two simple functions: 'drawFrame' and 'writeFrame', these are described later.

G - the 'main' section of the program, this is the part that is activated first when the program runs, and controls a loop that is used to generate the animation. This is described below.

The animation loop

When you execute the program, you have the option of defining how many frames it will run for by adding a number to the command:

./example1a.py 240

This will create an animation of 240 frames. The first few lines of the main section use the 'sys' module to read that number from the terminal. If no number is specified, it sets the number of frames to a default value of 60:

# set number of frames
try:
   # get number from command line
   frames = int(sys.argv[1])
except:
   # or set default value
   frames = 60

The program then prints a message on the terminal telling us how many frames are going to be rendered in the animation:

# tell the user how many frames are going to be rendered print "rendering %d frames ..." % frames

Then we start up a loop that repeats for the number of frames specified. On each loop it prints a message telling us which frame is being created, creates a fresh image for the frame, draws it by calling the 'drawFrame' function from our program, and then saves it to disc by calling the 'writeFrame' function:

# create a loop and draw the animation frames
for frameNumber in range(frames):
   # tell user the frame number
   print "frame %d" % frameNumber
   # create image
   image = Image.new(imageFormat, (width, height), bgColour)
   # draw frame
   drawFrame(image)
   # save frame
   writeFrame(image, frameNumber)

You can see that the imageFormat (whether the image is greyscale, "L", or colour, "RGB"), width, height and bgColour (background colour) variables from the constants section are used when creating each new image. The image is created by calling a function in the PIL Image module, see the documentation for this module for more information. The image we create in this example is just a plain white rectangle 320 pixels wide by 240 pixels high.

After it has gone through all the frames it prints another message telling us it has finished and the program then ends:

# tell the user we have finished
print "%d frames rendered." % frames

All the examples we will look at in this tutorial use the same basic structure with minor changes.

Drawing individual frames

The real work of creating the animation is done in the 'drawFrame' function. On every step through the animation loop, the new image is passed to this function which we have defined earlier in the program file:

def drawFrame(image):
   """
   Draws a single frame within the animation.
   """
   # create drawing surface
   drawImage = ImageDraw.Draw(image)
   # define a random colour
   red = random.randint(0, 255)
   green = random.randint(0, 255)
   blue = random.randint(0, 255)
   # define random coordinates for rectangle
   # top left
   x1 = random.randint(0, width/2)
   y1 = random.randint(0, height/2)
   # bottom right
   x2 = random.randint(width/2, width)
   y2 = random.randint(height/2, height)
   # draw rectangle
   drawImage.rectangle((x1, y1, x2, y2), fill=(red, green, blue))

This draws a single rectangle onto the plain white image. PIL does not allow us to draw directly onto an image. To do this we first have to create a drawing surface from the image using PIL's ImageDraw module. This module contains commands for creating a variety of different shapes, some of which we will look at in the next example. Here we are just drawing a single rectangle with a randomly set colour, size and position.

different frames from example1a

Before drawing the rectangle we define its colour. Colours are defined by three values: red, green and blue. Each can have a range between 0 (darkest) and 255 (brightest). We use the 'randint' command from the Python 'random' module to get a set of three values between 0 and 255. When calling randint, you specify two numbers, the minimum (0) and the maximum (255), and it will return a random value between these two limits. 'randint' is short for "random integer" meaning we will get a whole number (0, 1, 2, 3, etc) , rather than a decimal, or floating point, number (0.0, 0.175, 5.893, etc).

A similar method is used to define the position and size of the rectangle. When you draw a rectangle in PIL, you define its position and size by giving two sets of coordinates, the first for its top left corner, and the second for its bottom right corner:

coordinates for a rectangle

In order to make sure our rectangles always fit within the frame, the top left corner is defined by random values from the top left area of the whole image, and the bottom right corner by values from the bottom right area:

selection areas for random coordinates

Once these values have all been set they are passed to the 'rectangle' command on the drawing surface, and a rectangle is drawn on the original image.

Try experimenting with this example by changing the values given to the randint function. Giving a range between 0 and 80 for defining the colours, for example, will give you a set of dark rectangles:

   red = random.randint(0, 80)
   green = random.randint(0, 80)
   blue = random.randint(0, 80)

Setting the range between something like 200 and 255, will produce bright coloured rectangles:

   red = random.randint(200, 255)
   green = random.randint(200, 255)
   blue = random.randint(200, 255)

Try using different ranges on each of the red, green and blue values, so you get predominantly greenish rectangles, or yellowish ones (yellow is created by having equal values of red and green, pure yellow being red = 255, green = 255, and blue = 0).

Varying the range of numbers used for determining the rectangle coordinates will also produce different kinds of animation, such as ones with only small rectangles, or only squares:

   # creating a random square
   # top left
   x1 = random.randint(0, width/2)
   y1 = random.randint(0, height/2)
   # bottom right
   squareSide = random(10, 200)
   x2 = x1 +    squareSide
   y2 = y1 +    squareSide

After making changes to the code run the program again, and compile and view the video:

run program:

./example1a.py

compile video:

ffmpeg -i renderframes/example1a%3d.png -b 98000 video/example1a.avi

view video:

mplayer video/example1a.avi

Saving frames

After each frame has been drawn, the image is stored in a file by the writeFrame function. This takes the image and current frame number and stores the image as a file with the name of our animation, in this case "example1a", and the number of the frame:

def writeFrame(image, frameNumber):
   """
   Writes frame image to file.
   """
   # format number with zero padding: 001, 002, etc
   fileNumber = str(frameNumber).zfill(3)
   # create filename for frame
   frameFilename = '%s/%s_%s.%s' % (renderFiles, animName, fileNumber, fileFormat.lower())
   # write file
   image.save(frameFilename, fileFormat)

The images are stored in the order they were created with names such as:

example1a_000.png
example1a_001.png
example1a_002.png
.
.
.
example1a_058.png
example1a_059.png

Even though we have created 60 frames they are numbered from 0 to 59 as Python loops always count from zero to one less than the total number. By naming the frames in this way, ffmpeg is able to put them into the correct order when it compiles a video from them.

When we launch ffmpeg with the command:

ffmpeg -i renderframes/example1a%3d.png -b 98000 video/example1a.avi

the '-i' in the command tells ffmpeg that the following file(s) are to be used as input. The 'renderframes/example1a_%3d.png' tells it to use all PNG files in the renderframe directory whose names begin with "example1a_" followed by a three figure number, such as 000 or 059. It is important that there are no breaks in the numbering of the frames as this will cause ffmpeg to stop where it finds a break. The value after '-b' tells it the bitrate of the video frames, which determines the quality of the video image, a higher bitrate giving a better quality image. The last part of the command is the name of the video that it will create. See the ffmpeg documentation for more information about these and other options, such as changing the size of the video or adding an audio track.

Drawing different shapes

Once you are comfortable with changing things in example1a.py have a look at example1b.py. This follows the same basic structure but demonstrates some of the other shapes that can be created from PIL's ImageDraw module.

frames with different drawing shapes

In example1b we still call drawFrame to create the image, but this time rather than drawing the shapes as part of that function, we have created a set of different functions, one for each type of shape such as a rectangle, an arc, or a polygon, and several variations of drawing lines. It also shows how to draw text onto the image, so instead of printing the frame number in the terminal, as we did before, this time the frame number is printed onto the frame itself. Because of this, we not only pass the frame image to drawFrame, but also the frame number:

def drawFrame(image, frameNumber):
   """
   This draws a frame by calling all the other draw functions.
   """
   # create drawing surface
   drawImage = ImageDraw.Draw(image)
   drawRectangle(drawImage)
   drawPolygon(drawImage)
   #drawLines(drawImage)
   #drawParallelLines(drawImage)
   drawParallelStraightLines(drawImage)
   for i in range(12):
      drawArc(drawImage)
   drawPoints(drawImage)
   drawText(drawImage, 'hello')
   drawText(drawImage, 'frame: %d' % frameNumber, position=(10,10))

In this example, two of the drawing commands are commented out with a '#' symbol: drawLines and drawParallelLines. This means these won't get called when the function executes. To include these drawing commands, delete the '#' symbols. You can turn off other commands by putting the '#' in front of them, and in this way experiment with different combinations of shapes. You can have a shape drawn several times either by duplicating the particular command on multiple lines:

def drawFrame(image, frameNumber):
   """
   This draws a frame by calling all the other draw functions.
   """
   # create drawing surface
   drawImage = ImageDraw.Draw(image)
   # draw four rectangles
   drawRectangle(drawImage)
   drawRectangle(drawImage)
   drawRectangle(drawImage)
   drawRectangle(drawImage)

Or by using a loop as the example does with the drawArc function:

for i in range(12):
      drawArc(drawImage)

Remember to run the program again and compile and view the video each time you make a change and wish to see the result. Each time you generate a new animation the old frames are written over, but if you produce a long animation, say with 300 frames, and then a short one, with 60 frames, the additional older frames will still be in the renderframe directory and get included in the video. To prevent this, delete the old frames:

rm renderframes/example1b_*.png

and then run the program again to produce a fresh set.

If you look at each of the drawing functions, drawRectangle, drawPolygon, etc, you will see that they all follow a similar structure. The example also has a randomColour function as this gets used a lot and saves you having to copy the same code over and over. It also checks that you don't try to use numbers that are outside the range of those defining colours, ie: nothing below 0 or greater than 255. The drawRectangle function shows you how to define a grey colour, by just setting one colour value and then using the same value for the red, green and blue components of the colour:

   c = random.randint(0, 255)
   drawImage.rectangle((x1, y1, x2, y2), fill=(c, c, c))

For more information about each of the different drawing shapes, see the ImageDraw module in the PIL documentation.

Smoother forms of movement

So far we have been using random numbers to control shapes and their positions in the frame which results in quite jumpy, irregular animations. If we want to create smoother animations we need to use a different method of calculating positions. Interpolation is one way of doing this. Interpolation is ideal for animation as it allows us to create a smooth path between two locations simply by defining the start and end points and the number of steps in between. The rest is done by maths. In traditional animation this is known as 'tweening', and before computers existed had to be done by hand, often by some unfortunate trainee or junior animator.

Example1c.py demonstrates three different forms of interpolation. All use the same basic principles but different ways of calculating the in-between steps. It also shows that interpolation can be used to control other aspects of an animation, such as creating colour fades and transitions. Run the program and view the video to see what it does:

./example1c.py

ffmpeg -i renderframes/example1c_%3d.png -b 9800 video/example1c.avi

mplayer video/example1c.avi

You will see three squares move from the top left of the screen to the bottom right. All three start and end at the same positions but follow different types of paths. The blue square moves in a straight steady line between the two points. The green square speeds up towards the middle and then slows down as it comes to rest, and the red square moves in a curve. The differences in the paths are due to the different calculations used to the determine the in-between points.

frames from interpolation example

When you open the code for example1c you will see the three functions that generate the different paths: lineBetween - used by the blue square, easeBetween - used by the green square, and curveBetween - used by the red square. All the functions follow a similar structure. Each one receives the start and end points, the total number of steps and the current step which, in this case, is given by the frame number. The current step is divided into the total number of steps giving a decimal fraction, called 't' in the code, between 0.0 at the start of a path and 1.0 at the end. The start position is multiplied by 1.0 - t, and the end position by t, and these two new values are added together to give the current position. Over time, the value of the start position multiplied by 1.0 - t gets smaller, whilst the value of the end position multiplied by t becomes larger. In this way a smooth path between the two points is created:

linear interpolation

This is exactly how the calculation in lineBetween is done. It produces a straight, evenly spaced path known as a linear interpolation:

def lineBetween(start, end, steps, currentStep):
   """
   Creates a straight path between two points,
   with a given number of steps.
   
   This uses a linear interpolation.
   """
   if currentStep > steps:return end
   x1 = start[0]
   y1 = start[1]
   x2 = end[0]
   y2 = end[1]
   t = currentStep / float(steps)
   x = int((x1 * (1.0 - t)) + (x2 * t))
   y = int((y1 * (1.0 - t)) + (y2 * t))
   return (x, y)

The other two functions differ by introducing other factors. For easeBetween, the calculations are multiplied by cosine and sine values, creating a cushioning effect. This calculation is known as trigonometric interpolation as it uses the kind of maths normally used in calculating angles within triangles and circles:

trigonometric interpolation

By including an additional point away from the straight path between the start and end, curveBetween produces a path that curves away and then back to the straight path that the other two functions create. This is known as a Bezier curve, and the additional point is called a 'control point':

bezier curve

It's not absolutely essential to understand the maths behind these functions, what matters is knowing how to use the different paths that they create. Try experimenting with sending the squares to different locations on the screen, and by altering the control point for curveBetween.

Behind the moving square, the background of the animation is split into two halves. During the animation the top half starts black and slowly lightens to white, whilst the bottom half starts as a red colour that transforms into a light blue. These show how interpolation can be used to control other aspects of an animation such as colour. Almost anything which can be varied by a number changing over time can be controlled by using interpolation methods. Have a go at writing one which changes the size of a rectangle, making it grow or shrink during the animation.

Regular rhythms

Another type of calculation which is useful for animators is known as modulus. In Python a modulus calculation is written in the form c = a % b. A modulus is a kind of division that returns the remainder of the division rather than the normal result. If we apply the modulus of a number to a changing series of other numbers we get a repeating range of results between zero and one less than the modulus. This can be used to create regular rhythms. The modulus of four, for example, gives us:

0 % 4 = 0
1 % 4 = 1
2 % 4 = 2
3 % 4 = 3
4 % 4 = 0
5 % 4 = 1
6 % 4 = 2
7 % 4 = 3
8 % 4 = 0
.
.
.
1593 % 4 = 1
1594 % 4 = 2
1595 % 4 = 3
1596 % 4 = 0
etc ...

Two different uses of this are given in example1d.py. Run the program and watch the video to see what it does.

On every frame, the current frame number is applied to a set of modulus calculations. The results of these are then applied to create different animation effects which you can see in the drawFrame function:

def drawFrame(drawImage, frameNumber):
   """
   This draws a frame by calling all the other draw functions.
   """
   red = (255,0,0)
   green = (0,255,0)
   # right rectangle
   m = frameNumber % 8
   if m == 0:
      drawRectangle(drawImage, (width/2, height/2), (0, 0), red)
   # left rectangle
   m = frameNumber % 8
   if m < 4:
      drawRectangle(drawImage, (width/2, height/2), (width/2, 0), green)
   # moving rectangles
   m = frameNumber % (width/2)
   drawRectangle(drawImage, (20, height/4), (m, height/2), red)
   m = (frameNumber * 3) % (width/2)
   drawRectangle(drawImage, (20, height/4), (m, int(height * 0.75)), green)

In the first of these, every time the frame number modulated by eight returns zero, it draws a red rectangle in the top left corner of the frame. The rectangle is therefore drawn once every eight frames. In the second, the frame is drawn is the result if the modulus is less than four. This rectangle is also drawn every eight frames, therefore, but also again for the next three frames after. Visually this creates the effect of the rectangle appearing for four frames and then disappearing for four frames:

rhythms in top half rectangles

In the lower half of the screen are two narrow rectangles which repeatedly move from the left to the centre. Here the modulus result is used to determine the position of the rectangles by applying the frame number to the modulus of the frame-width-divided-by-two, and using this to set the x coordinate. The first, red, rectangle moves one pixel every frame. By multiplying the frame number by three, the second, green, rectangle moves faster.

cyclic motion in bottom half rectangles

Try changing the numbers used for the modulus and see the different kinds of rhythmic and cyclic effects that can be created.

The examples in lesson 1 all dealt with moving shapes around to create simple abstract animations. If we want to work with video material we also need to know how to work with pixels, the basic building blocks of all video images. On the computer, a video image is simply a grid of differently coloured squares, known as pixels. If we zoom in close on a section of a video image we can see these:

enlarged view of pixels in an image

These pixels can be processed by the computer as though there were simply a big list of numbers. The examples in this section show different effects that can be created by treating video images in this way.

Turning a video into still frames

As we are working with each frame as an individual image, before we can process a video we need to break it down into its separate frames. By basically reversing the ffmpeg command used to combine a set of still images into a single video, we can break apart a video into its individual frames. Ffmpeg will automatically number these in the same way as we have been numbering our animation frames.

Take a short video clip and copy it into the 'videosource' subdirectory of 'mgpy' with a command such as:

cp myvideo.avi mgpy/videosource

The video clips should ideally be either in AVI or MPEG format. If you want you can download a video of the web and use that. You may have to try a few different clips however, as a lot of videos that are online nowadays use proprietry codecs, such as Windows Media or Sorenson, that ffmpeg cannot read. Alternatively you can film and digitise your own material, see the Kino tutorial for instructions on how to grab video and export it as an AVI.

Processing pixels can be a time-consuming activity so we will mostly work with less than fullscreen video clips. The following ffmpeg command will take a video clip in the videosources directory, scale it down to 320 by 240 pixels, and export it as a series of numbered frames into the videoframes directory:

ffmpeg -i videosource/input.mpg -s 320x240 videoframes/example2a_%3d.png

The frames should be named to match the example programs. There is a version of this command with the correct names in the notes of each example that you can copy into the terminal, then just change the name of 'input.mpg' to match your source clip before running it. If your video clip is too long you can set ffmpeg to just convert a few seconds of it by adding the option '-t' followed by the number of seconds you want to convert. Ten to twenty seconds will be enough for the examples here:

ffmpeg -t 10 -i videosource/input.mpg -s 320x240 videoframes/example2a_%3d.png

If you wish to export a section from somewhere in the middle of the clip, play the clip in a movie viewer and take a note of the time where you want to start exporting from, either as a total number of seconds since the start of the clip, or in hours, minutes and seconds - written in the form 00:00:00. Then use the '-ss' option to tell ffmpeg where to start exporting from. This command, for example, will export ten seconds of frames starting at one minute and twenty-three seconds into a clip:

ffmpeg -ss 00:01:23 -t 10 -i videosource/input.mpg -s 320x240 videoframes/example2a_%3d.png

Reading video frames into your program

As we are reading in existing frames rather than generating them from scratch the animation loop is slightly different. It includes a new function, readFrame, which gets the current video frame from the videoframes directory. This is called in place of the line creating a new blank image in our first set of examples:

for frameNumber in range(frames):
   # create image
   image = readFrame(frameNumber + 1)
   # draw frame
   drawFrame(image, frameNumber)
   # save frame
   writeFrame(image, frameNumber)

If you look at the readFrame function you will see that it is actually very similar to the writeFrame function except that this time it creates an image from a file rather than saving one:

def readFrame(frameNumber):
   """
   The reads a single frame from a file.
   """
   # format number with zero padding: 001, 002, etc
   fileNumber = str(frameNumber).zfill(3)
   # create filename for frame
   frameFilename = '%s/%s_%s.png' % (videoFiles, animName, fileNumber)
   # read file
   image = Image.open(frameFilename)
   return image

Extracting and painting pixels

Once a video frame is loaded into your program you can access and alter its pixels using the commands of the PIL Image module. Example2a.py is a simple demo that reads pixels from one part of an image and then paints them into another. As before, the main work is done in the drawFrame function:

def drawFrame(image, breakPoint):
   """
   This breaks the image at a certain verticle point
   and then stretches the pixels at that row down the
   rest of the image.
   """
   # store pixels at breakpoint
   pixels = []
   for x in range(width):
      pixels.append(image.getpixel((x, breakPoint)))
   # paste pixels back into picture
   for y in range(breakPoint, height):
      for x in range(width):
         image.putpixel((x, y), pixels[x])

This function reads one row of pixels at a specific point on the y axis using the getpixel command, and then copies those pixels over the remaining rows below using the putpixel command (see the Image module documentation for details). Visually this looks as though the pixels have been squeezed down the frame. The breakPoint value that is passed to drawFrame is the y axis coordinate. It is calculated in the main animation loop. Here, another value, called breakStep, is calculated by dividing the height of the frame by the total number of frames in the animation. On each step of the animation loop this is multiplied by the current frame number:

breakStep = height / (frames * 1.0)

for frameNumber in range(frames):
   # create image
   image = readFrame(frameNumber + 1)
   # draw frame
   drawFrame(image, int(frameNumber * breakStep))
   # save frame
   writeFrame(image, frameNumber)

As this determines the point from where the pixel row is copied it creates the effect in the final animation of the original video image being slowly revealed from a mess of coloured lines:

frames from example2a

The getpixel and putpixel commands used in this example can be quite slow, another way of accessing the pixel data from an image in PIL is the getdata command. This returns a list of all the pixels in an image which can then be handled as a big list of numbers. Each pixel is represented by a tuple of three numbers for RGB images (the red, green, and blue), 4 numbers for colour images with an alpha channel, and just single numbers for greyscale or one-bit images:

Example2b.py shows one way of working with pixels like this. In the drawFrame function it extracts the list of pixels and then reads through them, checking the levels of red and green in each one. If red is the predominant colour in the pixel, the original pixel is replaced by a pure red one. If green is the predominant colour it is replaced by a pure green one. All other pixels are set to plain white. After the pixels in the list have been processed, they are written back into the image with the putdata command:

def drawFrame(image):
   """
   This strips out any pixels whose red and
   green values are less than 120. Red or green
   pixels greater than 120 or kept, with the
   blue component removed.
   """
   # get pixels
   pixels = list(image.getdata())
   # fixed pixel values
   white = (255, 255, 255)
   red = (255, 0, 0)
   green = (0, 255, 0)
   # paste pixels back into picture
   for y in range(height):
      yp = y * width
      for x in range(width):
         xy = yp + x
         p = pixels[xy]
         # check red and green values
         if p[0] > 120 and p[1] < 120 and p[2] < 120:
            pixels[xy] = red
         elif p[1] > 120 and p[0] < 120 and p[2] < 120:
            pixels[xy] = green
         else:
            pixels[xy] = white
   image.putdata(pixels)

This generates an almost abstract red and green image from the original photographic video footage:

comparison of frames before and after processing

Generating drawings from video images

By combining pixel data with the drawing commands we learned in lesson one we can generate new images that render the video material in a different way. In example2c.py the video material is drawn onto a new image as a grid of randomised squares, each square coloured to match a pixel at that point in the original image:

original image, and image rendered as random squares

Rather than reading every single pixel it steps through the pixel list only reading every tenth or twelfth pixel based on a value set by the blockSize variable. This also provides the coordinate for the square whose size is randomly set in relation to this:

def drawFrame(videoImage, image):
   """
   Creates an image of randomised rectangles
   that replicate the colour values of the video image.
   """
   # create drawing surface
   drawImage = ImageDraw.Draw(image)
   # get pixels
   pixels = list(videoImage.getdata())
   # min random
   minRnd = blockSize/4
   # max random
   maxRnd = blockSize
   # read pixels
   for y in range(0, height, blockSize):
      yp = y * width
      for x in range(0, width, blockSize):
         xyp = yp + x
         p = pixels[xyp]
         rndSize = random.randint(minRnd, maxRnd)
         x1 = x - rndSize
         y1 = y - rndSize
         x2 = x + rndSize
         y2 = y + rndSize
         drawImage.rectangle((x1, y1, x2, y2), fill=p)

The resulting image looks as though someone had recreated the video image using small coloured tiles or scraps of paper, and is an example of what is called "non-photorealistic rendering". We can vary the effect further by processing the new image through a filter such as blurring it:

blur filter applied to images

This can be done by using one of the commands from the PIL ImageFilter module, and adding it onto the last line of drawFrame:

def drawFrame(videoImage, image):
   """
   Creates an image of randomised rectangles
   that replicate the colour values of the video image.
   """
   # create drawing surface
   drawImage = ImageDraw.Draw(image)
   # get pixels
   pixels = list(videoImage.getdata())
   # min random
   minRnd = blockSize/4
   # max random
   maxRnd = blockSize
   # read pixels
   for y in range(0, height, blockSize):
      yp = y * width
      for x in range(0, width, blockSize):
         xyp = yp + x
         p = pixels[xyp]
         rndSize = random.randint(minRnd, maxRnd)
         x1 = x - rndSize
         y1 = y - rndSize
         x2 = x + rndSize
         y2 = y + rndSize
         drawImage.rectangle((x1, y1, x2, y2), fill=p)
   # blur image
   return image.filter(ImageFilter.BLUR)

Try different filters to see how they affect the image. You will need to add ImageFiler to the import commands at the top of your program if you are going to use this:

import Image, ImageDraw, ImageFilter

And change the animation loop to:

for frameNumber in range(frames):
   # read video image
   videoImage = readFrame(frameNumber + 1)
   # create image
   image = Image.new(imageFormat, (width, height), bgColour)
   # draw frame
   blurImage = drawFrame(videoImage, image)
   # save frame
   writeFrame(blurImage, frameNumber)

Also try using different drawing commands, such as sets of lines, points or polygons. It's up to you how close you stick to the original video, it may be more interesting to depart from any kind of resemblance to the original and just use the video as a way of generating an entirely abstract animation.

It's just a jump to the left ...

Once way get into working with pixels as data, it opens up the possibility of using one image to process another. If we think of a greyscale image as a big list of values between 0.0 and 1.0, then we can use this to alter and manipulate the pixels in a different image. The beauty of this is that we can create our processor images in a conventional paint package, such as the GIMP, and create quite intuitive effects that would be complex to create purely from calculations.

Example2d.py shows one example of this, creating an effect known as a timewarp. In a video timewarp, rather than playing back the video frame by frame we control the playback pixel by pixel, with different parts of the image displaying different points in time simultaneously. There are two steps to producing this. Firstly, rather than working on one frame at a time, we read in every single frame of our video sequence and hold them in a big stack:

video as a stack of frames

Then we load a black and white image with varying areas of black and white. The effect works best when there are gradual changes between the black and white areas.

To make an image like this in the GIMP create a new image with a black background:

plain black image

Then use the round selection tool, set the feather radius of the selection to a large value, and select an area:

select area and set large feather radius

Now fill the selected area with white:

white soft-edged area

Alternatively you could use a large paintbrush with featured edges and create a different kind of shape.

The timewarp effect uses requires a lot of memory to process, so for demonstration purposes we are going to use a small video image 160 by 120 pixels. Scale your black and white image to the same size and save it as a TIFF file into the inputimages directory. There is already a readymade version of the above image in this directory called "timewarp.tiff". You can use the same name if you like and save over this, or use a different name. If you use an image with a different name, however, you will have to change the following line in the program:

# load control image
print "loading control image ..."
controlPixels = readControlPixels('timewarp.tiff')

so that "timewarp.tiff" matches the name of your new file.

Now generate a sequence of video fames at 160 by 120 pixels:

ffmpeg -i videosource/input.mpg -s 160x120 videoframes/example2d_%3d.png

Run the program, the effect works best if you render about 120 frames or so:

./example2d.py 120

First we see it telling us that it is loading the video frames:

loading video frames

then it starts to render:

rendering timewarp

The timewarp takes a long time to process and this may take several minutes, so go and have a smoke or coffee or go for a walk. When you come back, compile the frames into a new video clip:

ffmpeg -i renderframes/example2d_%3d.png -b 9800 video/example2d.avi

And watch it:

mplayer video/example2d.avi

You should see that areas which were white in your processor image animate and move in time, whilst those that were black stay still. Pixels that were grey move at varying rates in between these:

enlarged view of pixels in an image

The main code that does this is relatively simple, and is found in the drawFrame function:

def drawFrame(videoPixels, drawImage, processorPixels, frameNumber):
   """
   Creates frame in timewarp sequence.
   """
   # make new pixels
   newPixels = []
   # read pixels
   for y in range(0, height):
      yp = y * width
      for x in range(0, width):
         xyp = yp + x
         processorP = processorPixels[xyp]
         videoIndex = int((processorP/255.0) * frameNumber)
         videoFrame = videoPixels[videoIndex]
         newP = videoFrame[xyp]
         newPixels.append(newP)
   drawImage.putdata(newPixels)

On every step of the animation loop, a blank image is created and passed to drawFrame along with the stack of video pixels, the pixels from the processor image and the current frame number. An array for holding a new set of pixels is created. The processor pixels are then read one by one. As it is a black and white image, each pixel is represented by a single number between 0 and 255. We divide this number by 255.0 to convert it into a decimal fraction between 0.0 and 1.0, and then multiply that by the current frame number, giving us our videoIndex value. If the processor pixel was black, videoIndex will always be 0, but if the pixel was grey or white then we will get a number that gradually moves forward with the frame numbers, a white pixel resulting in a videoIndex value the same as the current frame.

We use videoIndex to read a single frame from the stack, and then use the same coordinates that we used to read from the processor pixels to copy this video pixel into newPixels. As each frame in the stack represents a different moment in time, by varying which frame we read from, we vary which moment in time that particular pixel represents. Finally we paint the pixels onto our blank image.

How interesting a result you get from this will really depend on how well the video material you chose matches the processor image. It works best if the white areas in the processor image coincide with interesting movement within the original video. What the example demonstrates, however, is just how far we can go with pulling apart and reassembling video data in this way. See if you can think of another way video could be processed like this and try writing a program that uses it. Alternatively, you may just want to do the timewarp again.

A more conventional use of multiple images in motion graphics is to combine them on top of one another. This is known as compositing. There are various ways of doing this.

Example3a.py combines two sequences of images by blending one into the other. One image sequence is a video clip and the other is an abstract animation generated by the program. Two images are combined using the PIL Image.blend command. This creates a new image from the two supplied images which are blended together according to a value between 0.0 and 1.0, with 0.0 being only the first image, 1.0 being only the second image and everything in between a varying mixture of the two. In the example, the two image sequences are blended together using a Bezier curve interpolation:

def drawFrame(image1, image2, frameNumber):
   """
   Creates frame in blending sequence.
   """
   # create drawing surface
   drawImage = ImageDraw.Draw(image2)
   # fill with rectangles
   drawRandomRectangles(drawImage)
   # blend images
   alpha = curveBlend(0.0, 1.0, 0.7, frames, frameNumber)
   blendImage = Image.blend(image1, image2, alpha)
   return blendImage


def curveBlend(start, end, control, steps, currentStep):
   """
   Creates a curved blend value.
   
   This uses an interpolation along a bezier curve.
   """
   if currentStep > steps:return end
   t = currentStep / float(steps)
   b = (start * (1.0 - t)**2) + ((control * 2 * t) * (1.0 - t)) + (end * t**2)
   return b

This creates what is known as a "cross dissolve" between the two images. The use of the interpolation routine means that we can control how they dissolve, with it starting slowly and then speeding up towards the end. Try varying the control point on the curveBlend function to see how this affects the dissolve.

cross dissolve with blend operation

Another form of compositing uses a mask image. This is similar to the timewarp in some senses in that an additional image is used to control the processing of the two combined images.

Example3b.py combines the same two kinds of sequence as before but this time using mask images. A mask image is a black and white image in which white pixels are mapped to one of the combined images and black pixels are mapped to the other. You could create a mask manually in the GIMP, or, as here, generate one from code. The createRandomMask function produces a random black and white polygon image which is used to mask between the video sequence and animation:

def drawFrame(image1, image2, frameNumber):
   """
   Creates frame in blending sequence.
   """
   # create drawing surface
   drawImage = ImageDraw.Draw(image2)
   # fill with rectangles
   drawRandomRectangles(drawImage)
   # combine images
   maskImage = createRandomMask()
   compositeImage = Image.composite(image1, image2, maskImage)
   return compositeImage


def createRandomMask():
   """
   This creates a 1-bit mask image with a
   random polygon as the mask.
   """
   # create image and drawing surface
   maskImage = Image.new("1", (width, height), 0)
   maskDraw = ImageDraw.Draw(maskImage)
   # draw random polygon
   numPoints = random.randint(4, 8)
   xy = []
   for i in range(numPoints):
      x = random.randint(0, width)
      y = random.randint(0, height)
      xy.append((x,y))
   # draw polygon
   maskDraw.polygon(xy, outline=None, fill=255)
   return maskImage

The mask image can be a one-bit image, as used here, which will create a mask with sharp edges:

two frames showing the random mask composite

If a greyscale or alpha channel image is used, then the mask can be a lot softer and vary how much the two images combine with one another. Example3c.py shows how to use an alpha mask by creating a plain black image that contains areas with different alpha values:

def drawFrame(image1, image2, frameNumber):
   """
   Creates frame in blending sequence.
   """
   # create mask
   maskImage = createAlphaMask(frameNumber)
   # combine images
   compositeImage = Image.composite(image1, image2, maskImage)
   return compositeImage


def createAlphaMask(frameNumber):
   """
   This creates an alpha mask image with
   random rectangles as the mask.
   """
   # create image and darwing surface
   maskImage = Image.new("RGBA", (width, height), (0,0,0,0))
   maskDraw = ImageDraw.Draw(maskImage)
   # draw random rectangles
   numRects = random.randint(1, 12)
   for i in range(numRects):
      drawAlphaRectangle(maskDraw, frameNumber)
   return maskImage

When drawing the alpha image, the current frame number is used to vary the alpha level, thus creating an effect of one image dissolving into the other over time:

def drawAlphaRectangle(drawImage, frameNumber):
   """
   Draws a single rectangle with alpha channel.
   """
   maxAlpha = int((frameNumber/float(frames)) * 255)
   # define random colour
   rgb = randomAlpha(max=maxAlpha)
   # define random coordinates for rectangle
   # top left
   x1 = random.randint(0, width/2)
   y1 = random.randint(0, height/2)
   # bottom right
   x2 = random.randint(width/2, width)
   y2 = random.randint(height/2, height)
   # draw rectangle
   drawImage.rectangle((x1, y1, x2, y2), fill=rgb)

Visually this is similar to the blend operation in example3a.py but allows us more control, such as only blending in specific areas of the image, or blending different areas at different rates.

cross dissolve with alpha masks

Draw your own

If we combine compositing with pixel processing and drawing we can create other types of non-photorealistic rendering. In example3d.py a video sequence is rendered in a black and white cross-hatch effect:

cross-hatch render

This is done by first creating a set of cross-hatch texture images each with a different density of hatching:

cross-hatch textures

In order to give a bit of variety to the animation, four variations of each texture are created:

def createHatchingTexture(density):
   """
   Creates a hatching texture with a density between 0.0 and 1.0.
   """
   # create image and drawing surface
   hatchImage = Image.new("L", (width, height), 255)
   hatchDraw = ImageDraw.Draw(hatchImage)
   # set density
   spacer = int(10 * density)
   doubleSpacer = spacer * 2
   # draw lines
   y1 = 0
   y2 = height
   for x in range(0, width, spacer):
      x1 = x + random.randint(-doubleSpacer, doubleSpacer)
      x2 = x + random.randint(-doubleSpacer, doubleSpacer)
      hatchDraw.line((x1, y1, x2, y2), fill=0)
   x1 = 0
   x2 = width
   for y in range(0, height, spacer):
      y1 = y + random.randint(-doubleSpacer, doubleSpacer)
      y2 = y + random.randint(-doubleSpacer, doubleSpacer)
      hatchDraw.line((x1, y1, x2, y2), fill=0)
   return hatchImage

In the drawFrame function the video image is converted to greyscale and then split it into four new images by applying different threshold functions. Each of the four new images is a plain black and white representation of particular levels of darknesses in the image, the first being only the darkest areas, and each one after that including more of the lighter regions:

different darkness levels

Two different operations from the PIL ImageChops module are then used to composit a new image out of these. The cross-hatch textures are combined with the threshold images, the image of the darkest areas using the most dense hatching, and the lighter areas the light hatching:

textures and darkness levels combined

This results in a set of four texture images which are then combined into the final image using the ImageChops.multiply operation:

def drawFrame(videoImage, textures):
   """
   Creates frame in texture sequence.
   """
   # convert video image to greyscale
   greyImage = videoImage.convert("L")
   # split video image into 4 levels
   greylevel_1 = greyImage.point(threshold_1, "L")
   greylevel_2 = greyImage.point(threshold_2, "L")
   greylevel_3 = greyImage.point(threshold_3, "L")
   greylevel_4 = greyImage.point(threshold_4, "L")
   # apply textures
   texture_1 = textures[0][random.randint(0,3)]
   textureImage_1 = ImageChops.screen(greylevel_1, texture_1)
   texture_2 = textures[1][random.randint(0,3)]
   textureImage_2 = ImageChops.screen(greylevel_2, texture_2)
   texture_3 = textures[2][random.randint(0,3)]
   textureImage_3 = ImageChops.screen(greylevel_3, texture_3)
   texture_4 = textures[3][random.randint(0,3)]
   textureImage_4 = ImageChops.screen(greylevel_4, texture_4)
   # composit textures
   compositeImage = ImageChops.multiply(textureImage_1, textureImage_2)
   compositeImage = ImageChops.multiply(compositeImage, textureImage_3)
   compositeImage = ImageChops.multiply(compositeImage, textureImage_4)
   return compositeImage

The result is something which looks like an almost hand-drawn or etched animation. By using different drawing, pixel processing, and compositing techniques, different forms of rendering can be explored.

All the examples we have done have been made using the Python Imaging Library. This is quite powerful and simple to learn, but does have some drawbacks. The drawing functions are quite basic and do not allow us to control factors such as the width of line used (it is always one pixel wide), or enable antialiasing, all diagonal lines have jagged edges.

Aggdraw

Aggdraw is an extension to PIL that uses the Anti-Grain Geometry library (AGG) to handle the drawing operations:

We have covered a lot of ground in these exercises but they only really scratch the surface of what motion graphics can involve. Other aspects, such as more advanced animation processes, however, will need to be addressed in different tutorials as these also involve more complex forms of programming.

Even though we have been working almost exclusively with Python there is no need to limit yourself to just using that to produce your work. Many existing graphics and video tools are capable of both generating and reading sequences of images such as we have been working with. Video editors, such as Kino, LiVES and Cinelerra, can import and work with image sequences, whilst 3D modeling and rendering tools, such as Blender and PoVRay, can generate sequences that could be used in combination with programs like those demonstrated here. Cinepaint is a variation of the GIMP which is specifically geared towards working on stacks of frames and is widely used by the film industry. Programming can be one part of a bigger toolkit you use in your work, and Python in particular is ideal to act as a glue between many different resources, or for testing and experimenting with ideas that might be realised later with other tools.

Python graphics libraries: