THE EXTRAORDINARIES

Haiti Earthquake Support Center













"Thank you so much for
the work you and your
company are doing. I'm
glad there are people
out there doing the job
you're doing."
 
Renee Shropshire



"I want to thank you for
all the effort and great
work that you are doing.
We are still looking for
Denise. Thank you very
much for helping."

Leriette
Desir van Berge



"Thank You for your
Help to the People
of Haiti."

Michael Friedrich

The Haiti campaign has now come to an end.

On January 13th, one day after the devastating earthquake in Haiti, The Extraordinaries team sat down to see how we could use our tools to help desperate families get information about their missing loved ones.

But, we weren't the only ones trying to help. From Google, to Ushahidi, to Frontline SMS Medic, technologists from around the world came together to coordinate around how our community could use technology to help find missing people.

In four days, The Extraordinaries engineers built a system that allowed people just like you the chance to help find loved ones by tagging news images coming out of Haiti and matching those images to faces of missing people.

Twitter buzzed. News stations called. And, for two weeks, thousands of you came to help. 

Here are the results of your effort:




Did we find a match?

Yes and no. Out of 746 possible matches, we identified 24 that we were absolutely sure were matches.

But, reaching the families proved to be another challenge. Language barriers (many families lived in foreign countries; including Belgium, Columbia, and France), technology barriers (some families didn't have easy access to email), and more contributed to a much lower response rate than we had hoped. We continue to reach out to families and try and measure the success of our efforts.

See the results for yourself: Visit this link and click on the blue magnifying glass icons to see a pop-up of someone's submission.

Signup for updates on matches as we continue to contact families.
* indicates required


What does all of this mean? For us, it's a beginning.

This is one of the first cases where people can actively help disaster relief efforts from their own personal computer. And, their collective efforts form a crowd that can bring about big changes. After all, the crowd is very efficient when powered by the right tools. People want to do something, even if they live 2,000 miles away, and now they can.

To see how the system worked, we've kept the links live:

Image Tagger -- http://app.beextra.org/mission/show/missionid/605/mode/do
Facial Matcher -- http://app.beextra.org/mission/show/missionid/637/mode/do
Tag Search Engine -- http://app.beextra.org/appflickr/haiti

The Technology

From a technology perspective, our engineers spent sleepless nights doing things that we don't think have been done before. First, we imported a stream of uncategorized images from myriad sources - knowing only that they were tagged with "haiti, earthquake" and taken after the quake. These images consisted of a wide range of subjects from photos taken right after the quake on the ground in Haiti to images of wellwishers from Holland.

Then, we asked volunteers to apply a series of structured tags to these images by presenting a small branching survey. We asked questions such as:
* Is this photo related directly to the Haiti quake?
* Are there people in the photo
* Can you clearly see a person's face
* Gender: male, female, mixed
* Age: young, teen, adult, elderly, mixed

These questions generated a series of structured tags for each image, along with a freeform tag area where volunteers entered other descriptors of their own invention. This resulted in signs being transcribed and clothing being described.

This application was available both on our Web site and through our iPhone application. We're still compiling the numbers on usage per platform.

iPhone Haiti Landing Page & Image tagger
iphone image of haiti app

Screenshots from the Web application: Landing page & Image Tagger


After several thousand submissions, we were able to easily and usefully categorize the photos. We had a group of images showing women, men, teenagers, adults, faces, and buildings. The first thing we did was to build a search engine that allowed people looking for a lost loved one to search for that person specifically. Using this search engine, a person could select "woman, young" and then type "Pink dress" - and come out with 25 or so matches of young girls in pink dresses. So, for a family that was missing a young girl who loved her favorite pink dress, this was a quick way to filter through the news imagery to find the most likely of those to show their loved one.

Screenshots from the Web of the Search Engine and Search Results page showing results from a search for "Female, Young, Pink Dress"





The next software we built was the "Image matcher" - this pulled in photos from Google's database of missing persons (thanks to Zach at Exygy for the parsing efforts) on the left side of the page and then pulled an image from the news on the right side of the page - and then asked the volunteer to tell us if there was a match. If the record from the database contained gender information, the image on the right would be narrowed to show only photos marked by volunteers as containing that gender. Later in the campaign, because many of the records from Google did NOT contain gender information, we added the ability for the matcher to specify the age and gender of the missing person. The news images would be filtered accordingly to provide a better chance of a match.

Screenshots from the Web showing the Missing Person submitter (which used Google's API to draw records) & the Missing Persons Matcher. 



Note that the inspiration for this system came from watching a lecture by Jonathan Zittrain on YouTube a week prior to the quake. In this lecture, Zittrain posits that crowdsourced labor could be used for nefarious person-identification schemes by repressive regimes (minute 32:00 or so). When the quake hit and people were missing, we thought immediately of Zittrain's imagined use case - and of turning it, of course, towards more redeemable purpose. 

From a socio-technological standpoint, we leveraged the talents (eyes/brains) of 1000s of concerned individuals to do detailed and focused image recognition. The result was that a mass of news images was suddenly useful for the specific purpose of person identification. This feat would likely be impossible for even the most sophisticated artificially intelligent image recognition software. Of course, as humans, face-recognition is one of our innate and primary talents. Through the combination of structured tagging, filtering, and matching, we were able to direct these talents towards a critical need in a time of crisis.

Our hope is that the work we began here will be useful in the future- when the unfortunate but inevitable next disaster strikes. This system could be used to find missing people, locate buildings suitable for refuge, find plane wreckage in satellite photos, or to locate lost alpinists. More generally, the system could be used to make sense of the news. There's a constant stream of images coming from disparate sources from all over the world. Using volunteers, we can start to slice and dice this imagery along all kinds of useful lines.