Map of the Week: Creating a Daily Dashboard of Global Conflict

Summary

Explore how CartoDB and GDELT collaborate to create an automated, daily-updated map of global conflicts. Visualize and analyze real-time events now.

This post may describe functionality for an old version of CARTO. Find out about the latest and cloud-native version here.
Map of the Week: Creating a Daily Dashboard of Global Conflict

Welcome Kalev H. Leetaru  a fellow and adjunct faculty in the Edmund A. Walsh School of Foreign Service at Georgetown University in Washington DC.  In this guest post Kalev talks about how he used CartoDB to create a map of global conflict that updates every morning.

Imagine a map that uses computers to “read” news reports from across the entire world each day  identifying the events described within and their locations  and placing all of that into a single interactive map  zoomable and clickable down to the level of a city or hilltop across the globe  summarizing at a glance our world in motion. Add on top of that an animation of global events over the past six months  placing current events into context and illuminating macro-level global trends.  Through the support of the United States Institute of Peace and using the CartoDB platform  this dream has become a reality.

Mapping the Geography of Text

My work centers on how creative applications of the world’s largest datasets  computing platforms  algorithms  and mindsets can reimagine the way we understand and interact with our global world.  Towards this end  a lot of my projects revolve around extracting  analyzing  and visualizing the locative information in the textual narrative that surrounds us  from mapping the world’s news media  to Wikipedia  Twitter  diplomatic communication   and even television.  

This past December I worked with the Internet Archive   to create what we believe is one of the first large-scale glimpses of the geography of American television news  revealing which areas receive outsized attention and which are neglected  by geocoding the closed captioning stream of all half million hours of their television archive.  To visualize such a massive dataset  we turned to CartoDB to create an interactive clickable map where visitors could instantly view every mention of a city or landmark in context.  

On top of this we decided to create an animation showing how the geographic focus of American news has changed over time.  Instead of a traditional pre-rendered animation  we wanted to offer an interactive experience where visitors could live-zoom into the map  literally watching the floodlight of American journalism move across a region or country.  One of CartoDB’s most powerful capabilities lies in its ability to create precisely these kinds of animated interactive maps with just a few mouse clicks.  In the end  we created both an animated map and a comparison map that allows you to drill down by time period and specific station to compare coverage.  Read more about the maps.

Mapping Global Conflict

Using the lessons learned from mapping American television news  could one create a similar map of global conflict  this time bringing to bear CartoDB’s SQL API to update the map every morning completely automatically?  The GDELT Project I founded  monitors the world's broadcast  print  and web news from nearly every corner of the world in over 100 languages and uses sophisticated computer algorithms to compile a daily list of the people  locations  organizations  counts  themes  sources  and events driving global society  creating a free open platform that makes global society “computable”.  Each day GDELT compiles a catalog of over 300 categories of events from around the world georeferenced to the city level  from riots to peace appeals  totaling more than a quarter-billion records 1979-present and updating every morning.

GDELT provides the ultimate mappable database of global conflict  translating the world’s textual news reports into quantitative spreadsheet entries  each geocoded to the level of a specific city or landmark worldwide.   CartoDB  in turn  provides the ultimate interactive mapping platform to visualize this massive geographic database.  To me one of the most powerful aspects of CartoDB is its ease of use.  Unlike desktop GIS software  you don’t have to spend hours carefully preparing a dataset for importing  trying to figure out why a “float” column keeps truncating to 2 decimals or fighting your way through screen after screen of prompts.  Instead  you literally drag-and-drop a spreadsheet onto the CartoDB window and the rest  as they say  is “magic”.  CartoDB automatically recognizes which columns contain geographic coordinates and transparently converts it all to a point-based map.  Want to add additional layers such as shapefiles?  Once again  just a trivial drag-and-drop and your new layer appears a few seconds later.  The resulting maps are among the best looking cartographic visualizations I’ve seen  supporting complex raster operations and sophisticated render styling with ease.  In fact  combined with the built-in visualization wizards  it is possible to go from spreadsheet to final production map in just a few minutes.

I’ve twice used CartoDB to create maps of GDELT for columns I’ve written for Foreign Policy magazine  mapping protests and violence in Ukraine the day before President Yanukovych fled and the spatial divisions of unrest in Nigeria. However  in both cases I had to do a fair bit of manual work to query and translate GDELT into the final spreadsheet to load into CartoDB.  Once created  these maps were also static  reflecting only snapshots in time.  Could this entire process be automated and update each day?


 



To begin with  the manual workflow used for the Ukraine and Nigerian maps was automated into a PERL script that runs each morning on the GDELT servers.  Each day around 6AM EST it downloads the daily GDELT event export  which is a CSV file listing all of the global events that GDELT monitored in the previous day’s media.  The PERL script scans all of these records and collapses them down by city  applying some additional filtering to try and minimize false positives and surface only the most significant events.  The final output of the PERL script is a simple four-column tab-delimited file listing all events from the previous day  collapsed to the city level and containing latitude/longitude  event type (protest or conflict)  and the number of events found in each city.

The CartoDB SQL API   is then used to upload all of these new entries to the underlying map table on CartoDB’s servers.  In a nutshell  the SQL API offers an incredibly easy-to-use HTTP-based access modality by allowing SQL commands to be passed via GET or POST web requests.  This allows the daily PERL script running on the GDELT servers to remotely manage the hosted map table running on CartoDB’s cloud service to upload the latest new records each morning.  Similarly  the SQL API is used to delete the oldest records  maintaining a 180-day rolling window of events for the animated layer.  Essentially the PERL script “remote controls” the CartoDB map  updating it each morning.  Making the map required just a few mouse clicks to add three layers: a pink layer for yesterday’s protests  a red layer for yesterday’s conflict  and an animated layer showing both protests and unrest over the past 180 days.

The resulting map can be seen below  protests in pink and conflict in red read more about the specific filtering and settings used.

CartoDB’s unique browser-based animation system means that you can see global-scale patterns one moment and the next moment zoom into a specific country or region to visualize its localized trends.  ISIS’ march across Iraq is instantly visible  civil war in Ukraine bursts into view  and the clustering of protests and violence in Nigeria leaps from the screen.  

Using CartoDB’s ability to separate the underlying data from the maps that visualize that data  an alternative version of the map was also developed  using a bright daytime colored map with a different color scheme for the events.  In addition  the ability to specify different versions of the same map that have different centerpoints and zooms has made it possible to create embeddable maps for different countries and regions without changing any code.  To make an embeddable map of Iraq  one simply uses the main map visualization and hands it different zoom/center parameters.  Instead of manually creating 20 completely different maps  the same map can be used for infinite different localization needs.  CartoDB’s incredible ease of use  gorgeous nearly art-like visualizations  powerful interactivity  unique browser-based zoomable animations  and SQL API made the creation of this global conflict dashboard incredibly easy.

If you think about it  this map represents an entirely new way of thinking about  visualizing  understanding  analyzing  and interacting with our world’s news. News is intrinsically rooted in space: events happen or are announced or discussed at particular locations and involve people  organizations and actors situated and moving through their own array of locations. Yet  in an era where one cannot spend more than a few minutes without encountering a map of some kind  the world’s news media is still presented as a pile of words and images completely detached from space. Despite a typical news report averaging one location every 200-300 words  today we still interact with the news through the humble keyword search box or selecting by country. Yet  retrieving 1 000 000 matching articles for Iraq does little to shed light upon the “big picture” of how unrest is moving through the country and region.  Through the power of GDELT and CartoDB and the support of the United States Institute of Peace  we were able to create a first glimpse of a global map of conflict  updated each day.

Acknowledgements

We’d like to thank the United States Institute of Peace for making this map possible and Google Ideas for their support of the GDELT Project.

Take a look at our Map Gallery and check what you can build using CartoDB. Also go to our website  sign up for a free account and start creating customized maps.