I’ve got a new piece up over at PBS: Inside Vox Media’s Attempt to Build a Modern Media Stack
Computer magic
Make an Internet Time Capsule with Google Chrome
If you use the Chrome browser, you may have noticed that when you begin typing in the address bar, Google’s Autocomplete prediction service guesses where you might be heading to save you keystrokes. If you have Web History enabled, those guesses aren’t just popular websites, but rather the sites you’re historically most likely to visit.
I realized that this list of sites actually end up serving as a sort of internet time capsule of the last ten weeks (the amount of time the browser history spans). So, here are my Internet ABCs of my last three months of grad school at the Media Lab. Click any of the images to go to the site.
6 productive responses to PRISM
Along with the other free peoples of the internet, we’ve been discussing our reactions to PRISM, and whether and how US (and global) citizens might be able to organize against this unprecedented domestic spying. There are more questions than answers at the moment, and the enormous challenge of confronting an extra-legal entity like the NSA with people-power is strongly felt. But here are 5 things you can do that could prove more productive than petitioning the White House to respond. Thanks primarily to Sasha Costanza-Chock for the roundup:
1. Encrypt yourself
See The Guardian Project’s Android apps, Security in a Box, and Tor. If you have the skills, go further: build tools / better UI / How To Guides / visibility to encourage more people to encrypt themselves, too.
2. Support calls for a Congressional committee to investigate
You and your organization can sign on at https://www.stopwatching.us.
3. Organize, or participate in, a protest.
People are starting to plan for these in various locations; July 4th is a good date. Here’s one in DC. Continue reading
Holmes Wilson, internet activism, and why we need you
(originally posted on Civic MIT)
Fight For the Future is known for its massive viral organizing campaigns that changed Internet history both nationally and globally. Faced with the passage of Stop Online Piracy Act/SOPA and the Protect-IP Act/PIPA — legislation that would have jeopardized the open Internet as we know it — Fight for the Future organized the largest and most visible online protest in history. Holmes Wilson has also co-founded Miro, OpenCongress, and Amara. He’s been at the forefront of a range of open internet and participatory culture projects and campaigns.
The internet delivers newfound powers of expression
The key thing about the internet that drives Holmes’s passion for it is that it gives us a new power, which ultimately translates to freedom of expression. But not in the conventional sense. The freedom of expression the internet enables isn’t just about speaking. It’s about making art, starting a business, overthrowing a government, building a new government, realizing dreams, and the ability to give your greatest gift to the world. When we think about expression this way, it’d be unthinkable to fail in preserving this medium. It would stifle human potential.
But that power is inherently fragile
The internet is fragile. The power that is being given to people is not necessarily stable and there are significant threats to it. The most present threat of the recent year is SOPA/PIPA. In some ways, these seemed like very small reasonable changes to the law. There’s a law that says sites aren’t responsible for content that users generate and this would make site owners responsible for users’ content. The consequence would have been that any copyright holder could have taken down any site where their content appeared and any site that is built with user generated content would have to aggressively police user behavior and contribution. Most of the harm would have been invisible. If SOPA was in effect when YouTube was first invented, we wouldn’t have YouTube.
Encouraging Flexibility from Social Media Giants: How We Get Private Platforms to Support Public Speech
(originally posted on Civic MIT and PBS)
There are many problems with using commercial technology platforms to host democratic, social, or activist content and communications. These problems came up in multiple sessions at the National Conference on Media Reform last weekend. There are also obvious reasons to continue using these platforms (audience reach, most notably), and so we do. Some activist efforts that silo communications on more open, but relatively unknown platforms strike me as irresponsible, if the goal is to reach as many people as possible (but this is a fine line). The more I think about this issue, though, the more I see potential solutions and a future in working with the platform providers to build some degree of flexibility into their products and policies.
The spot on the carpet reserved for public ranting at #NCMR13
Social media giants do not have immediately obvious incentive to participate in such compromise. First of all, supporting individual humans doesn’t scale at anywhere near the order of magnitude they seek with their software. This model of customer support is perhaps best illustrated by Google, where serious and eminently solvable problems are routed through static FAQ pages, or, if you’re lucky, a forum page where a Google developer or superuser might stumble across your concern and provide some hint of illumination as to its origin or any hope of forthcoming resolution.
Coping with Hyperconnectivity
Liveblog of the “Coping with hyperconnectivity” panel at NetExplo at UNESCO in Paris. The speaker is Delphine Ernotte Cunci, Deputy Chief Executive Office of France Telecom Group.
One argument about information in our times is that we are submerged by the sheer amount of digital information we receive — we are drowning in it. So we do our best to disconnect ourselves. Others argue that this intentional disconnection phenomenon hasn’t been studied enough to confirm. One thing is certain: technology influences our relationship with time. The internet changes our temporal space. Technology tends to promise to save us time, but it’s clear that the internet can also eat into the few spare minutes we once had.
Delphine uses Twitter, but not Facebook. She uses Twitter for her job.
Her company, Orange, sits on a lot of insightful data about this trend. It’s empirically true that we’re more hyperconnected these days. More than half of cellphones sold today are smartphones. They allow us to jump between tasks quickly. These tasks are oftne micro tasks, not long thought. We spend an average of 2 hours a day on our cellphones, and 2.5 hours watching TV (the two overlap). 80% of television viewers who have a tablet use the tablet simultaneously. 68% of smartphone owners use their phones while watching TV.
Is hyperconnection good or evil?
In the Women’s Forum for Economy and Society last year, Orange presented a survey called Time and Connection. It found that hyperconnection is compatible with well-being. 68% of French people think the more connected and active you are, the happier you’ll be. And 92% of internet users surveyed feel more relaxed when they’re connected than when they’re not connected.
People feel they’re able to do about 80% more at work with hyperconnectivity. We can multitask multiple tasks concurrently. But multiple screens produce a behavior of zapping between small tasks. You may have noticed this yourself when you finally do need to read a long file and find you no longer have the attention span for it.
Are we addicted, imprisoned by these habits? We don’t like to live in a vacuum, and we go to our phones and computers in spare moments. 39.5% of French people felt the lack of connection when you want it is a real concern that heightens anxiety. And people are suspicious of possible addictive qualities of technology. This impacts our relationship to time. Even if we can accomplish more things in a day with technology, we are left with the feeling that we do not have enough time to do all that we must do.
49% of French citizens slow down at some point in the day. Orange decided to launch a noncommercial project called the Time Project. They’re talking with philosophers and sociologists about the human perception of time. This will inform the mana of the project.
Then, Orange will provide customers with tools to better manage our connectivity. They’ve introduced a modem with sophisticated controls to allow parents (or ourselves) to black out internet use at certain hours of the day. Parents can remotely disable the connection during homework hours.
Companies also struggle with changes to our sense of time. People told Orange they were collectively overworked: too many emails, too many poorly organized meetings, too many important interactions. An initial proposal to forbid internal emails after 8pm backfired. It turned out that some women were using late night email to allow them to leave the office earlier to take care of their families. This experience taught Delphine that blanket policies would not work. Technology allows flexibility between our private and public lives, but it can also be abused. Employees can be left without time to rest if their overbearing bosses expect replies to emails at all hours of the day.
Younger generations are deeply connected to online social networking. Orange used to forbid employees to use Twitter and Facebook, even while recruiting 10,000 new employees. For a 25 year old employee, inability to access Twitter or Facebook at work was a completely unacceptable policy. Orange changed their policy this January, but internally, the social networks weren’t opened up for fear of the detritus effect on work. Orange finally convinced the managers to allow access, and the world did not end. Younger employees were happy, and older employees saw no change whatsoever.
But there’s a difference between an official email from your company and a tweet. Employees who tweet sensitive information are at risk, because they’re not necessarily authorized to disclose the information. Social networks make the barrier between companies and the public a far thinner membrane. There are slipups.
We cannot fight connection. It’s a sign of the times. But Delphine believes telecom operators have a responsibility to help customers control this hyperconnectivity. A balance of rules and flexibility could provide the netiquette we need to find balance. We can agree to rules, like no phones at restaurants, or no laptops in meetings.
Asked how she, manages connectivity, Delphine answers that she has begun to restrict how many emails she sends, and has to wait to tell people things in person. As a result, she says, she doesn’t fact email overload. She does send a lot of SMSes.
How to Find your Dream Job by Playing Games
(more NetExplo liveblogging from Paris)
Epistemic gaming is the emerging field of games for assessment. It’s an evolution from the immensely profitable field of psychometric testing, where Myers Briggs and StrengthsFinder tests promise to help us identify our individual personality traits and intelligence styles to create better harmony and results in the workplace (for a healthy per-employee fee).
NetExplo luareate ConnectCubed is a meritocratic game-based approach to job recruiting. Founder Michael Tanenbaum found himself excluded from the banking recruiting process, which made him wonder how many other qualified employees were being missed. At its most basic layer, ConnectCubed is a platform of games to help job candidates identify the best fit.
In many cases, work doesn’t work. Gallup found that 72% of employees are disengaged. Michael calls this the pandemic of presenteeism. You show up to work but don’t care about the quality of the organization as a whole. Michael sees this problem’s root in two causes: our expectations of employees and how we found them to begin with. He’s focused on the latter.
How do we develop ourselves and specialize in a given career? As children, we learn about the realm of careers by proximity, not efficiency. [For example, children of teachers do well in schools, and it’s been posited that they are more comfortable with schools as institutions than other children].
We don’t have much guidance for our productive lives. How do we find the best career for our individual personality? Contrast this with the vast array of companies working to capture our expressed interests, like Netflix, Google, and Flipboard. All of these companies have one goal in common: get you to consume more. They collect tiny little breadcrumbs to get you to buy, search, read, and watch more. Yet, we spend so much more time at work producing than we do at home consuming. And there are no companies collecting this information to deliver us the job where we are naturally, intrinsically motivated.
ConnectCubed assembles a dataset about you to find you a job that will be meaningful to you day after day, year after year, regardless of compensation. Rather than be creeps observing your behavior, they ask people to come to their platform and help them create a multidimensional profile of you through games.
Michael points to the 150 million copies of The Sims, which he calls House Chores, the Game.
Released in 2011, ConnectCubed mimics specific jobs and determines if the people who show up to perform it are good at those tasks. They started with a stock-trading game, Ultratrader, developed in 2001. Gamers played for many hours, and follow-up research found that those who showed up and played the best were either already successful traders or were on their way to success in this career. The game positively identified those who would fit the job itself well.
Simulation gaming provides a very good sense for who you are and what you’ll be good at in under 20 minutes, far less time than psychometric tests. Michael believes the power of simulation gaming can solve the problem of job matching. A small library of simulations could demonstrate the various departments in your company, so that when people show up, you can place them properly.
Michael also sees an opportunity in the disconnect between high youth unemployment and survey results showing that companies are having trouble finding qualified candidates. We should develop games to better find candidates for professions with chronic shortages.
Two existing examples of this approach at work are Thinkful and Edelman. Thinkful, based in NYC, creates tailored academic programs to help anyone with drive learn new engineering. It’s paid for by students, but most of the funds come from the companies interested in hiring new engineers.
Edelman, on the larger end of the spectrum, began an internal learning and development program in 2006 to certify mastery in specific skills relevant to public relations. The company found that their various offices competed with one another to succeed int he program. This level of employee buy-in of an internal learning program is unheard of. Executives gain a huge level of insight of the distribution of their skills companywide.
[Yesterday, Jean-Marie DRU, President of TBWA\Chiat\Day, told us how they created skill-based hubs around the world. The company didn’t have critical mass of every skill (data visualization, graphic design, etc.) in every office worldwide. Their network allows them to fish for the skills and expertise they need for various client projects. This 12,000-person skills network is called the Digital Arts Network.]
Jobs can be expressions of what we love. Games give us the ability to be competitively nerdy about the thing which we love. And they attract a talented and enamored talent pool for recruiters to hire.
Remembering by living
Aaron has left us. Beautiful and honest eulogies pouring in from around the world make it clear how many people and ideas his short life touched. You should go read those. But for my own personal emotional processing, I’m going to borrow a mourning practice from my friend Sasha. When you lose someone important, you can personally honor their passing by incorporating some of their behaviors, values, and traits into your own life. Adopting a part of them keeps them alive in your own life, but also the lives of others.
From what I knew of Aaron and from what I’ve read this week, I’m going with these:
- Liberally get in touch with people whose work you admire, even if you have no clear reason or zero ulterior motive, just to let them know it’s great.
- Join the challenges that need champions. “That’s a problem I want to be a part of,” one post put it.
- Practice that more natural method of self-education, that homeschool spirit of pursuing the things you’re excited by without regard for when the test will be, to learn in very great detail because of the natural interest inside you that must be quenched. Embrace the Wikipedia wormholes that transport you so quickly through time.
- Accept people for their minds, not their appearance or position, and accept that many young people’s personalities are far more developed than we tend to give them credit for.
- Double down on your natural enthusiasm for improving accessibility to human knowledge. The advancement of the human race depends on us learning more, but also, and probably in larger numbers, teaching and sharing more of what “we” already know.
- And, of course, “Being a programmer is like finding out you have magic powers”:
LazyTruth mentioned in The Economist
The [print edition!] of The Economist mentions LazyTruth as an antidote to some of the misinformation we find online:
“Companies are developing programs, such as the wonderfully named LazyTruth, that help people assess the credibility of online information”
Political Tech vs. Civic Tech
The campaign post-mortems are pouring in, unveiling the computer magic behind the Obama campaign. We should probably be thankful that the conversation has evolved from 2004-2008’s obsession with social media into a newfound lay interest in data aggregation and empirically valid testing. I’m learning a lot reading these articles.
This stuff is cool, but the Holy Grail Subject Line is not why we got into this work in the first place. Tom Steinberg has responded with an inspiring post encouraging the talent behind campaign tech to consider building civic technologies. Tom outlines some really important (and thus far unfulfilled) goals for civic tech to aspire to, including:
- These tools, apps, platforms, whatever, need to scale way beyond even the most popular instances we have today to be considered worthy of actually reshaping society and impacting all of those millions of people in society who have never heard the phrase “Gov 2.0”.
- We should attempt to produce tech that creates transformative change at the seismic level of so many internet-driven disintermediators, from TripAdvisor to eBay. This means creating new possibilities and patterns of behavior so profound, we can’t even return to the Old Way of doing things.
I agree with Tom that it’d be good for the world if some of the campaign ninjas become improved democracy ninjas (barring the obvious counterargument that elections have very real results). There’s also the argument that these ninjas achieve similar ends when they open source tools built with the investments generated by the “vast amounts of cold hard cash” generated by US presidential elections. These tools, like improved election day reporting on the Ushahidi platform, could be and probably will be redeployed for general civic use.
I’m confused, though, when Tom’s argument focuses on the simple fact that, in a campaign, you have opposition working against you and your team. Many of the voter outreach tools the Obama campaign built this cycle will live on in various incarnations, including some version of trickle down or transfer to the opposition’s side. The rival campaigns’ technological contributions worked to dilute one another’s chances of winning the election, but I don’t see a strong argument that their tech cancels out the other team’s contributions or general progress.
TripAdvisor and its ilk made the customer stronger in their relationship with hotels, but the battle is hardly over. The many services and products we rate and review are learning to game these systems, whether by creating bots, paying people on Mechanical Turk, or just providing better customer service at key moments (which is clearly a win of some sort for the customer). There’s far more money involved in hospitality industry booking than there was in the entire US Presidential race, and it, too, is a technological arms race.
Partisan tech might drive competition in the election season, but in general, I feel that the left-right political divide weakens the political tech industry by cutting the potential market in half. I’ve always had the impression that one factor causing innovation in political technology to lag significantly behind the commercial technology industry is potential market value. Political campaign tech is a subcategory of the much larger tech industry, and was until recent years, a sleepy backwater of the tech industry. The fact that the market for campaign technologies and its suppliers is, in many places, split in half by the non-economic force of partisanship only contributed to the lack of real investment that drives commercial tech. This split also, in my opinion, allowed some mediocre consultants and products to maintain market dominance where, in a more competitive market, they would have been unseated. Experienced political technologists on the left and right can likely point to tools or people on the other side they’d love to have access to for their campaign.
There are probably many other reasons that campaign tech traditionally lags behind commercial tech, like a political campaign culture that for far too long saw developers as IT managers here to fix your printer, and boom and bust election cycles. And Obama’s tech team, in 2008 and 2012, has clearly made huge strides in changing this reality. But otherwise, I think partisanship’s division of market value is at least a factor. Why else would social change companies like NationBuilder and Change.org risk their roots as generally progressive firms, and frankly, certain large-but-partisan clients, to pursue a much larger, richer global market far outside of US politics? And isn’t this how they’ll scale to the levels Tom challenges them to?
I’ll now await pushback from much more experienced political technologists.
Talking Fast II: More CrisisMapper Ignite Sessions
Luis Capelo (@luiscape) of Digital Humanitarian Network loves volunteers. DH exists to stimulate more interaction between humanitarian volunteers and large humanitarian institutions.
There’s information overload in humanitarian responses. How do we collect and make sense of all this information? Luis credits humanitarian orgs with doing the hard work of adapting, but it’s a rough sea to navigate. Volunteer & Technical Communities thrive in this environment. They’re nimble, lightweight, and advanced, technically. Luis thinks its time to stop questioning whether VT&Cs can help, and begin to dive into how these groups can collaborate.
DH aims to create a consortium of groups that faciliates between the two worlds, and reduces the cost of collaboration
They have a simplified activation process: activate volunteers, triage the volume, and forward them to VT&Cs. They’ve produced a guide to manage the activation of VT&Cs.
July and August of this year saw the first two activations. OCHA and ACAPS came to the DH network for help. OCHA wanted to build a pre-crisis profile of every country. ACAPS wanted to include VT&Cs in the formal assessment process.
Join at digitalhumanitarians.com.
Cat Graham (@Peaceful_intent) of Humanity Road works in multinational crisismapping. They specialize in the first hours of an event (12, 24, 48 hour windows). Self-directed work teams with training and a mission come online.
Their forthcoming QuickNets microtasking platform is open source, free, and will stay that way. Each row of data has an ‘anonymize’ button. Its been tested at RIMPAC and Pacific Endeavor exercises. 20 volunteers from 8 nations stepped up to model communications for the tabletop exercise.
Ka-Ping Yee (@zestyping) is an engineer at Google’s Crisis Response team. He uses Stratomap, the open-source tool behind Google Crisis Map. It’s on Google Code, and there’s a hosted version available. There are important datasets for crisis response, but they’re all hosted on different websites, so it’s difficult to get them in front of the right decisionmakers. Some of these maps have crappy UI, or don’t allow the databases to be combined. Data publishers have tools to publish, but map curators could offer even greater value if they were able to mashup maps and databases between various providers. We gain great insight when we can synthesize various pieces of information.
The Google Crisis Map provides a range of useful layers, from traffic to weather to user-submitted YouTube videos. Your map mashup can point to live feeds around the web, and it will be updated in realtime as those data sources are updated.
Users can share a customized view of the map, with layers
Brian Root (@brian_root) of Human Rights Watch shows us a US map depicting ICE’s deportation patterns. The group produced a report on the human rights implications of the US Immigration department’s detainment and deportation policies. They needed data to show ICE’s movements, but only ICE had the data they wanted. Through Freedom of Information requests, they were able to procure some data.
After cleaning the data, they were able to show the number of facilities involved, the facilities sending the most cases, and identify problem cases, where a detainee has been transferred numerous times across the country. They were able to visualize findings about the costs, human and financial, of transferring detainees.
But maps and data do not effective advocacy make. Drilling down to the state level was more useful with getting the attention of local media and local politicians. In January of this year, ICE issued a directive to limit the number of transfers, in large part due to HRW’s report.
Brian asks the audience to consider the human rights research that could be done with the mapping experience sitting in this room.
Clarence Wardell (@cwardell) led a research team at University of Arkansas following the Social Media and Emergency Management conference. One of the main concerns highlighted in their summary report was high-level resistance to use data because of verification. They took the strategy of conceding the “is it perfect?” argument up front, and instead arguing that the data was still nevertheless useful. There is room between horseshoes and hand grenades.
They mapped the verified and unverified data points together, creating the Traveling Salesman problem for disaster responders. Which relief tour is the optimal use of time and ground covered? They tested the multiple approaches.
Munish Puri (@RecordedFuture / site)time travels by looking at data over time. Hindsight + insight leads to foresight. Text has a predictive power when it is loaded with temporal references.
They looked at temper, time, and tone in the Georgian Conflict. Volume and velocity didn’t equal veracity. Licklider’s Intelligence Amplification.
Discovery consits of seeing what everybody has seen and thinking what nobody has thought. Albert Szent-Györgyi
Clionadh Raleigh (@acledinfo) is Director of Armed Conflict Location and Event Dataset (ACLED). They report on all violent events across Africa. Dates, locations, actor types, event types, and territory exchanges are collected to produce trend reports for others.
What’s happening in Africa?
We can use SpatialKey to detect and investigate patterns. The agents of violence have changed drastically in the last few years. Civil wars are declining, but political militias increasingly threaten civilians. We can follow the movements and attacks of groups like the LRA and Boko Haram. Boko Haram, for instance, never attacks troops.
We’re seeing more trans-national threats from Islamist groups. We see Ethiopia’s increasing violence over the last 15 years.
Analysis sees increased urbanization and other factors driving today’s violence. The data informs policy, academic, and public research. They offer special reports on topics like Islamist violence.
Steven Livingston (@ICTlivingston) introduces Mapping the Maps and Crowdglobe, an Internews platform to visualize geospatial data. Different map themes emerge in different regions. Western Europe sees crowdmaps used for entertainment and leisure and media reports.
They also surveyed crowdmappers, 80% of whom were men at an average age of 40 years. Many of the dead maps are simply a result of users experimenting with no intention of creating a map to begin with.
Only 6% of respondents promoted their maps using traditional media.
Jonne Catshoek is a conflict analyst working in the Republic of Georgia. In 2008, Russia and Georgia battled over South Ossetia. Conflict continues despite the international community’s involvement. Jonne blames security strategies that are not responsive enough to the needs of local communities. The elva platform (code here) they developed allows community representatives to SMS in community needs, where the information is then put online for the wide range of non-community actors.
Jonne estimates they spend 20% of their time developing software, and 80% of their time understanding local community need. This local trust allows better information sharing, and more reliable information.
They use SMS because smartphone penetration remains low. Trained monitors code a lot of information into a single SMS.
They’re expanding beyond violence into weather and agricultural information feeds for communities. The community is also heavily reporting security incidents, helping security providers respond appropriately.
Patrick Vinck (@developmentdata) is at the Harvard Humanitarian Initiative. They conduct surveys on countries’ peace and conflict, the way we have regular reports on health and other human factors. KoBo Toolbox is a data collection instrument designed to assist in this effort. It helps surveyors create their forms and export the questions. It even has recommendation engines, to suggest questions based on the topic being queried.
The same tool can collect voice, written text, images, and videos in one place. It works offline.
KoBo Map links to a spreadsheet (Google, CSV, etc.) and visualizes the data it contains. It’s lightweight for slow connections.
Taylor Owen (@taylor_owen) did a PhD thesis to map the historical US bombing of Cambodia. He quotes Nixon telling Kissinger to “crack the hell out of them” with an unlimited budget. He demands it. The result is over 200,000 sorties over an 8-year period. The 115,000 records detailing the planes, bombs, and tonnage remained secret until President Clinton opened the records to Vietnam for the purposes of de-mining the land.
Taylor used the data to produce timelines and fact-check the official timeline. The data can change our understanding of history. The bombings started sooner than we’ve said, lasted after the peace treaties, and hit civilian areas where Kissinger said we wouldn’t. We can see Watergate’s effect on bombs dropped in Cambodia.
We can see how the US bombing pushed the Khmer Rouge east, into the Vietcong territory, where they developed from agrarian socialist revolution to an anti-imperialist group.
It doesn’t take much to radicalize a population, Taylor says. In one instance, a single bomb in a village drove 70 radicalized recruits.
Henry Kissinger’s record of claims on this issue, including Kissinger’s Second Rule of Engagement (We won’t bomb within a mile of a village) turns out to be wildly incorrect once we map the bombing patterns. The US bombed heavily populated areas near Phnom Penh, for example.
Government data is often deleted, or at least classified for long periods of time. We need to work with the data when it does get released, so we can understand its historical implications.
Patrick Florance is in town from Tufts University to talk about the Open Geoportal (OGP). It’s an open source project at Tufts to rapidly discover, preview geospatial data. It’s a collaborative effort to take on big geospatial datasets.
You can shop around for datasets and drop them into your virtual shopping cart. You can incorporate third party web services and share the data all over the web. It works with common web mapping tools and will have over 20,000 data layers available by December.
Josh Campbell (disruptivegeo) is a geographer at the Department of State. He’s working to link together the US government’s purchasing power of commercially available satellite imagery, and the need of VTCs for this imagery.
In a few weeks, Haiti went from being barely mapped at all, to being mapped in such great detail so as to support on-the-ground action. State attributes this incredible development to the existing OSM community, empowered by web-service satellite imagery.
The US government buys a LOT of satellite imagery, and is contractually required to share it. In the Horn of Africa crisis, they experimented with letting volunteer mappers map refugee camps. 29 volunteers produced 50,000 nodes of data on a previously blank refugee camp map. The volunteers provided not just roads and streets, but also footways, paths, hydrologic features, and other rich data.
The project showed that people would map where there is imagery. But they’re interested in mapping the human elements, as well. The Red Cross hosted a mapping party, and went to Uganda to train locals to use OSM and fire responders. Locals annotated data with places names and restaurants. The local community received a map of the density of grass huts (a fire hazard).
John Crowley (@jcrowley) works at Camp Roberts to connect the top-down and bottom-up aid groups.
1. In law, agencies are having trouble navigating the policies that govern their use of crowd data
2. Trust in data, and trust in processes of VTCs
3. Security – the Arab Spring and Anonymous have shown that we can’t secure all the voices in a system
4. Voice – We could have a bigger collective capacity than we’ve ever had in human history. But what happens when that moves faster than our governments?
Agencies ask, how can we control this? Bad news is, you can’t. Good news is, we can begin to see how we can coordinate in this space. But we need to step outside of bureaucracies that only allow information to flow down.
We need space to fail, where it won’t disrupt actual operations. That is the purpose of Camp Roberts. Crashing is allowed. They bring together the players in the space to bridge capability gaps.
How do we repeat Haiti?
A range of actors not traditionally in the same space were brought together. An 18-month exercise brought them up against many legal and policy walls, but they were able to show the process worked.
FEMA came and asked about doing the same with tornadoes. Civil Air Patrol and many VTCs came together and designed a new workflow, which was used in Hurricane Isaac two weeks after it was created.
Can we scale this innovation process to all the other agencies around the world? It’s an Open Humanitarian Initiative.
This process of bringing people together into safe spaces requires combining the wisdom of the old with the innovation of the new. How can we bring together the human race to learn to heal itself?
Talking fast at CrisisMappers: the Ignite Talks
Dr. Jen Ziemke (co-founder of CrisisMappers) welcomes a room packed with a wide variety of professionals and volunteers. CrisisMappers started in 2009 as a network, designed to stay in touch
The group has grown to 5,000 members, organized on a Google Group and Ning network.
For the newbies in the room, what is CrisisMapping?
Jen breaks it down into the data coming in, the visualization of the data, and the response: how does it affect decisions on the ground?
Changing technology is clearly a primary driver of crisis mapping. Mobile technology and the ability to crowdsource shared experiences and visualize it on a map, or elsewhere, has enabled crisis mapping. Beyond mapping, this community quickly becomes a broader group of digital humanitarians, using technology to help communities affected by crisis.
Patrick Meier (the other co-founder of CrisisMappers) hops on stage. He’s currently at the Qatar Foundation Computing Research Institute, where they’re researching how to gather information from social media
In 2009, the field of crisismapping was just beginning to take shape, and it was easy to know every project and get to know all of the people behind the projects, often over drinks. The field has exploded, and matured, and the problems the field faces have grown more difficult.
The community met last year in Geneva to begin tackling some of the challenges. The computer security behind emerging humanitarian technologies was an obvious area of concern. John Crowley spearheaded this effort at the Camp Roberts event. Phoebe Winpope* has taken on issues of data privacy and security, working with professionals in that space.
Wendy Harman and her team at the American Red Cross have launched the Digital Operations Center, driving home the point that social media for disaster response is here to stay.
Andrej Verity has launched the Digital Humanitarians Network to facilitate the space between volunteer technologists and large humanitarian organizations.
Geeks WIthout Bounds and Digital Hacks of Kindness are also driving forward innovation in this space.
Disaster-affected communities are increasingly the source of big data. When Japan was struck by earthquake, tsunami, and nuclear disaster, a TON of data was shared online. Patrick argues that we need hybrid methodologies that combine crowdsourcing and the speed and scalability of advanced machine learning algorithms. We need to use multiple channels to listen to communities.
Verification remains an important challenge for aid organizations looking to make use of social media. Media organizations are actually leading the way in this department. The BBC has had a User Generated Content hub in London since 2007.
Monitoring and Evaluation are also important. Our perception of digital humanitarian technologies is high, but the real evidence supporting them is pretty thin. We need strong, independent evaluations of these technologies’ impact.
CrisisMappers 2013 will be held in Nairobi, Kenya (applause). This will be the fifth annual conference, and Jen and Patrick have decided to step down as organizers, and adopt OpenStreetMap’s model, where members of the network pitch to host and organize the annual conference.
Patrick is also inspired by TEDx, and hopes to see the CrisisMappers brand, logo, and website repurposed to support far more local events in this space.
Ignite talks!
Lin Wells (@STAR_TIDES) gives an overview of the STAR-TIDES network and its 1500 nodes. It is public-private and trans-national. They work post-disaster, post-war, impoverished, and short term and long term (disaster vs. refugees). They work in domestic or foreign situations, whether military is involved, or not. Shelter, water, power, cooking, lighting, sanitation, and ICT technologies.
Lin says technology alone is never enough. Building social networks and developing trust, as well as understanding how policy is adapted in the field, all matter.
TIDES hosts annual technology demos at Fort McMayer. They’re also present at Camp Roberts. The real world events they’ve supported include floods, wildfires, election monitoring, and more.
Jakob Rogstadius (@JakobRogstadius) introduces Crisis Tracker to crowdsource the curation of information like tweets during a crisis. Twitter poses a challenge, in that 140 characters is too short for computers to understand, but the rate of incoming tweets is too voluminous for humans to cope with.
The Crisis Tracker platform helps people sift through the noise and identify the novel information pieces in the stream, reduce the inflow rate, and enable volunteers to act on the actionable content. Similar messages are clustered, junk is filtered out. 30,000 tweets become 2,000 stories, and then 7 unique pieces of information per hour. The system looks at metadata like the timestamp, and then the crowd annotates the information with stories.
Why human involvement?
Humans can process text and images in ways computers cannot. Humans are also adaptable to rapid changes.
The system is up and running for the Syrian civil war. You can drill down into individual stories and see who shared it, links to multimedia content, and similar stories.
Volunteers have appreciated the platform’s ability to aggregate and filter. The system picks up stories and ranks and auto-sorts and filters the top stories. A 6-8 volunteer team can pull out the top items. With 40-60 volunteers, you can have a detailed log of the event at a very local level.
The project is free and open source.
Mona Chalabi (@MonaChalabi) brings us back to the challenges brought up by both slow and sudden-onset challenges. Crisis maps were unable to stop delays in aid distribution in Haiti. Have we reached a plateau in the use of GPS? FedEx uses RFID to track packages. It’s a barcode on steroids. A chip stores coded data, an antenna transfers the signal, and a computer accepts it for processing. Many subway cards use this technology.
In Haiti, hundreds of containers arrived each day, and those that were recorded were tracked using manual processes like Excel spreadsheets, leaving plenty of room for error. RFID would be much more efficient at managing the supply chain. The chips can be re-used.
The private sector has used RFIDs to great effect. The effective distribution of resources in a crisis can be the difference between people being fed and people being tear-gassed by UN troops afraid of a large crowd. RFID also combats information asymmetry in the space because the information is automated rather than input by one group and verified by another.
Simple supply chain logistics remain a major challenge for aid agencies and governments in crisis.
The effective distribution of resources in a crisis can be the difference between people being fed and people being tear-gassed by UN troops afraid of a large crowd.
RFID also combats information asymmetry in the space because anyone can use the information.
Simple supply chain logistics remain a major challenge for governments in crisis, and make it more difficult for corrupt officials to interfere with the chain of supplies.
Nate Smith (@nas_smith) is with Development Seed and Mapbox. There are many factors in communicating the many complicated factors that go into a crisis situation.
Context is important, like the conditions in the Horn of Africa prior to the droughts.
The Sahel Food Crisis project was much more than mapping, Nate says. Getting access to the raw data and communicating it in an appropriate way was part of the challenge.
Workflow is a problem. How we access data, and do things with it, can slow things down. Fews Net maintains their data in PDFs, which slows down developers looking to do anything with it.
What are the right colors and pieces of information in a map to express information?
We must design for shareability. Tools must be equipped for people to use them. If you’re publishing a lot of maps, your website should have the map endpoint on your site, so others can take and use it.
MapBox is also looking at building large data browsers that will support applications. The Sahel Food Crisis site is open for collaboration on Github.
Richard Stronkman (@rstronkman) is founder of Twitcident, another service to listen to the voice of the community during incidents. So much information is produced, with 400 million tweets per day. It’s a lot of noise, but when there’s a crisis, the information rates go up.
Early warning to identify increased risks and potential incidents.
And when incidents occur, they do crisis management.
In the Netherlands, they’ve worked with police forces, event security company, and the Dutch railway operator. On Queen’s Day, the Utrecht police force asked them to produce a map of incidents in a real-time dashboard. The team was able to identify threats towards the Royal family, leading to police visits.
At summer carnaval, the team worked to intervene against false rumor propagation. They found rumors at an early stage, and helped the police publicly disprove rumors early in the process before the rumors took off.
The team was able to identify a lack of drinking water at a large scale water fight early in the event, and help organizers react.
The group publishes their findings as academic research and in the technical press.
Shadrock Roberts (@Shadrocker) works at USAID’s Geo Center. The USAID Development Credit Authority uses credit guarantees to encourage local banks to lend to underserved communities. They wanted to map their work to see how they could do it better. They had a beautiful dataset of 100,000 records over the program’s 12-year history, but the location information was all over the place.
The Geo Center team cleaned up the dataset with a hybrid human-computer method. An automatic process parsed the easy fixes, but they raised a lot of eyebrows when they suggested crowdsourcing the remainder. They built an application on Data.gov. The technical infrastructure was a hurdle, but so were the legal policies restraining how the agency could use the data. They cozied up to the lawyers and received help from VTCs to clarify what volunteers would do with the data.
Involving volunteers went beyond crowdsourced tasks. Volunteers began a much broader discussion about development and the Development Credit Authority. Social media mentions of the organization went up significantly. 300 volunteers took on 10,000 records in 16 hours with 85% accuracy.
The goal wasn’t just a map, but to make the data open for others. You can find it by Googling “USAID Crowdsourcing Transparency” or tweeting at @USAID_credit.
Jerri Husch is here to talk Action Intelligence. How do we make sense of the massive amounts of data. Like electrical outlets, we have competing standards and cultural influences in different places. Jerri argues that we need an adaptable standard. If we look at societies from afar, we might make the mistake of assuming everything’s objective. But the world doesn’t work that way. A cricket is a pest in one place, and a delicacy in another.
Action Intelligence allows us to manage and analyze multiple dimensions and link data. We need to know who’s doing what, where, when. We need the data immediately in a crisis, and we need it around for the long-term as legacy data.
They can link actors to one another, or link actors to actions.
The process goes like this:
1. Collect data, often with university teams, in standardized ways
2. Classify and code that data, by place and time, at micro or macro levels
3. Visualize the data. They use free, open source tools, which allows for an adaptable standard that works anywhere in the world.
Andrew Turner (@ajturner), formerly of GeoIQ, has joined ESRI as CTO of their Research & Development Center.
“Big Data” means different things to different people, but we know it’s huge. In a crisis, with very short, life-threatening situations, data can help. In Haiti, crowdsourced information was really good, but still needed someone on the ground to act on the information.
GIS analysis lets us count and look at geospatial analysis of information. We need to evolve and build learning algorithms, because our current techniques are pretty easily fooled. @Racerboy8 provided useful data from his house during the Colorado wildfires, but his house wasn’t actually in danger. He was just being helpful.
In the NYC Marathon last year, FEMA looked at sensors throughout the crowd to visualize the crowd’s movement over time and space.
We can model situations before they occur, and detect communities in advance of a crisis.
Anahi Ayala (@anahi_ayala) works for Internews, which supports local media across the globe to empower communities. Anahi’s at the Center for Innovation and Learning, looking at how to incorporate new technologies.
Merging crowdsourced data with official information has proven difficult because of the difficult of verification. In the Ukrainian elections, they’re collecting information not only from social media, but also trained electoral monitors and journalists. Users can see verified reports vs. untrusted sources on a map.
The team dissects all of the information that comes in in an attempt to verify. They look at the context, the content, and the source of the information. There are digital traces everywhere online. Who are you already friends with, followed by, directly engaging with?
The content itself can be verified. We can crowdsource, triangulate, follow up with the source, and look at the weather in the video you submitted.
Every event occurs in a context in a country. Reports can be verified based on knowledge of the existing situation in a place and time.
Everyone’s adopting their own verification methods. Yes, falsification of information is always possible. But so is verification. Machine learning makes verification faster and cheaper for organizations. The question today is not whether or not you can verify information, but how to make it of high enough quality and timely enough to be acted upon.
Kuo-Yu Slayer Chuang (@darkensiva) goes by Slayer. He brings us back to the Titanic, which sank in a time when SOS technology wasn’t standardized. Open GeoSMS is a standard that combines SMS and location. Smartphones can embed all sorts of geo information in messages. They are designing a user-centric application to make collaboration easier.
Lars Peter Nissen (@ACAPSproject)
How do we make sure the data we collect actually becomes useful for decisionmakers?
Mistakes happen in large-scale, multi-agency responses. Potential impact is stymied.
In Haiti, we only used 1/3 of the data gathered. That’s a waste of the effort exerted collecting that data.
Disasters are never what we expect. Decisions are made when they have to be made, not at some ideal point in time. And we’ll always have massive information gaps, with plenty of known unknowns and unknown unknowns.
Three principles:
1. Know what you need to know. It sounds obvious, but do you?
2. Make sense, not data. Don’t collect data if you don’t know what you’ll use it for.
3. Don’t be precisely right, be approximately right.
With Internews, they designed the GEO, Global Emergency Overview. A snapshot gives you a quick overview of what’s happening, globally. Short summaries provide a basic understanding of specific crises. Then, you can drill down into 20-page analyses that help you discriminate between different types of needs in the fields.
Sara Farmer (@bodaceacat) is a core team member at Standby Task Force. It’s 2 years old with 1,000 volunteers. They’re generally known for turning social media feeds into maps. But they’re not just about information; they’re also about knowledge and analysis. They support HXL and other standards.
Their Disaster Needs Analysis (DNA) reports provide information about locales before disasters occur. Teams investigated and mapped available data for countries. They created baseline indicator sets, and set up a workflow to collect, store, and distribute the data. A fleet of scrapers converted online data into machine-readable tables. The data was cleaned into standard formats for country names, dates, and geo references. Gaps are filled in with estimates and proxies. Expert (Hunchworks) also help fill in the data.
Phil Harris (@geofeedia) sees every demographic using social media more and more. It’s not just Facebook and Twitter – there are image-rich services that didn’t exist two years ago. Smartphones drive more user generated content.
They set geofences on London and monitored the Olympics, aggregating 175,000 posts from YouTube, Twitter, Flickr, Picasa, and Instagram. Instagram’s a surprisingly large source of posts (36%). Only 31% of the posts contain keywords like London or Olympics. The majority of these pots wouldn’t have been tracked by traditional keyword search methods. Geofeedia sells a service to monitor social media.
Colleen McCue (@geoeye) works in geospatial predictive analytics. Again, we can learn a lot from private sector marketers. Product positioning in the supermarket is critical. Location also matters when you’re talking about bad actors in the humanitarian world. The Lord’s Resistance Army has struck over the borders of several African nations. We can target them based on past behavior. And we can segment the population by crime types, like marketers. Looting, abduction, incidental homicides, and murders produce different geospatial patterns. Individual factors can influence violent crimes. Roads and porters are critical to a successful abduction. IDP camps and other population clusters are attractive to the LRA, just as banks are attractive to thieves.
“Distance from murders” turns out to be a major factor in instances of isolated murders, suggesting a key behavioral difference from incidental homicides that occur over the course of another crime. The group is compiling signature profiles of different behaviors, and using advanced analytics to produce actionable recommendations prior to events occurring.
Kalev Leetaru brings us back to 38,000 years ago, where we find the first written records. 550 years ago, we get the printing press. Today, we’re producing incredible amounts of information and written records as a species.
The history of conflict can teach us about the present. We can visualize NGO reports and the global media tone towards a nation like Egypt or a leader like Mubarak to see when they’ve lost global credibility. We can track geographic affinity for a leader like Osama bin Laden. We can map conflicts within a nation by various factors.
Dave Warner is “dangerously over-educated and works in a memo-free environment.” He wants to make smart people smarter and explode the dots on the map into more complicated pins that contain significantly more information. His maps look much more like something out of a strategy video game than GIS software.
Dave mapped an audience by their WIkipedia entries, and academics by geography across the country.