iPhoto is a clever program; just how clever is easy to underestimate, or overlook. It became clear for me when I stumbled upon some of the guts of one of its most popular features: face recognition. iPhoto is looking out for you, trying to help: it actually looks at everybody in your photos and finds all the faces. It then tries to figure out who’s who, so you can find all the pictures of your mom or your best friend, or even you cat.
If you search Google for the phrase “iPhoto face recognition” the tenth result (as of this writing) is an article about the fact that the program will recognize your cat. When the feature was first introduced, that’s all anybody talked about. That iPhoto sees your cat’s face is both amazing and completely unremarkable; what you won’t get out of Google is anything about how it actually does this, which is understandable – it’s magic, of course, and Apple wants to keep it that way. But the cat thing is a clue: The program is trained to look for patterns of light and dark. Roughly, two eyes, a nose and a mouth, or at least a pattern of light and dark more or less corresponding to that arrangement.
As I said, iPhoto’s looking out for us. It knows just how many pictures we’re taking these days, how much information we have to contend with, and it wants to help. In fact, it’s so eager to help that it doesn’t wait to be asked. It looks for those faces as soon as you drop a photo into the library. When it finds one, it makes a tiny thumbnail file, no larger than 240 pixels square, and literally hides it away on your hard disk. Later, when you’re looking through your photos, you can turn on “Faces.” iPhoto shows you what it’s found; you can name each person then sort through a set of other pictures that person might be in. Slowly the program gets better and better at figuring out who everybody is, but it still asks.
iPhoto asks because there’s no way for it to really recognize your cat, not the way you do. And there’s no way it can know it’s not dealing with just your family and friends, but perhaps with people that everybody recognizes, that everybody’s been looking at and living with for fifty years. It’s just looking for shapes; it doesn’t understand anything. If it were helping us read a book it would be sounding out the words aloud, discarding all the filler – adjectives, adverbs, articles, even verbs, putting together a story composed entirely of things, of nouns, “useful” units of information, content.
Sometimes the patterns of light and dark are just right. iPhoto will find almost every face in a picture: Those five people sitting on a trolley, looking out at the photographer. Of course, there’s no way for the program to know it’s the southern USA in the 1950s and all the white people are supposed to be at the front; iPhoto sorts them in the order it found them; it abolishes segregation. Fifty years later, and no longer on the trolley but instead inside your computer, maybe these people can finally all sit where they want.
Sometimes iPhoto fails miserably, or beautifully, or both, depending on your perspective. It looks at a photo of people at a funeral but the patterns of light and dark aren’t right. It sees none of actual faces, but invents a tiny ghost from shadows in a tree. Or it gets things partially right: It ignores the starlet, front and center in the picture. Instead it chooses a few anonymous, adoring fans in the background. It brings the periphery to the front, leaving us wondering what these women are looking at, why they’re so enthralled. We look harder at the edges, and think about what’s missing.
In order to get things right, iPhoto has to make some decisions, and sometimes it seems like it just needs to guess. If a photo’s not contrasty enough, or too light or too dark, it won’t see the faces. So before it starts looking at a photo, it makes some adjustments, until it finds some of the patterns it’s looking for. Maybe this is why it has to choose between the black nurse and the white baby she’s holding – maybe the baby’s got a bit more contrast than the nurse, so iPhoto concentrated there and darkened the picture to the point that the nurse lost definition.
What I’ve described is all guesswork, of course – Apple’s veil of magic obscures what’s really going on. But there are ways to reproduce the process that are free and transparent to anybody who wants to spend the time working through the code, and the results are close enough to iPhoto’s that it’s quite possibly using the same methods. For instance, in the Processing programming language, the OpenCV library can achieve remarkably similar results with very little work. The first step is to simplify the image considerably: convert it to black and white and increase the contrast – make the patterns of light and dark into shapes that are easy to find. Then adjust the brightness and, as if by Apple magic, you get faces.
Open up your Pictures folder. Find your iPhoto Library file, right click on it and choose “Show Package Contents.” Inside a folder called “Data.noindex” you’ll find everything: your pictures, as well a thumbnail of everything iPhoto thinks might be a face. Every face in your iPhoto library.
I developed this project while completing my MFA in Documentary Media at Ryerson University, 2010 – 2012. It existed first as a book, then as a print in a gallery.
If you want to read it, you can download the accompanying thesis paper here.
With sincerest respect and admiration for Robert Frank, and gratitude to Pierre Tremblay, Ed Slopek, Don Snyder, and the faculty, students and administration of Ryerson Docmedia.