Today we are excited to bring you a guest post from our friends at KimonoLabs. If you aren’t familiar with Kimono, they bring a set of tools that enable you to create APIs from data and information scattered throughout the Web. We highly recommend checking them out. Below, they will share with you the steps for turning a website into an API and then how to use CartoDB to map that API in some really interesting ways. This was posted originally over on Kimono’s blog here, thanks for the permission to repost!
Data is more accessible, tangible and interesting when you can visualize and interact with the figures on a page. That’s why we love teaming up with our friends over at CartoDB! Kimono is a smart web scraper that let’s you turn data on a website into an API – a structured feed of updating data. CartoDB let’s you take that data set and create beautiful interactive maps. In this post, we will use kimono to get over a year’s worth of bike trip data from New York City’s Citi Bike bike sharing program. We’ll then use cartoDB to plot our friend Andrew’s movements on a map. A big thanks to Andrew for riding his bike a lot and sharing his data with us!
Here’s all that you’ll need to build your own data-driven map:
Navigate to the website with the data you want to map. For this example we are using data from Andrew’s Citi Bike account, which looks like this:
Click on the kimono chrome extension…
and the kimono toolbar will appear on top of the webpage:
Notice the flashing lock icon on the toolbar. This indicates that the page requires you to log in. Click on the lock icon.
Kimono will then direct you to the site’s login page (if you need to navigate further to get to the login page, click the navigation icon and go to the appropriate login page).
Once at the login page, you must identify the username, password and submit fields – this teaches kimono how to login to this site. Do this by clicking the username, password and submit icons on the toolbar and then clicking on the matching field on the webpage.
Click done and then enter your login credentials. Kimono securely stores the credentials so that it can access your data automatically on the schedule you specify.
Once you’ve completed the login cycle, you will see the original page with your data. Here we want to grab four types of data – the start station, start times, end station and end times. To do this, click on one of the start stations. Kimono will suggest other start stations to you. Click the check mark to accept all the start stations into your first data property. Click the check mark to accept all the start stations into your first data property. The number in the yellow circle…
…will increase to reflect the number of data points in that property.
Now click the grey plus…
…to add a new property and repeat this process with start times, end stations and end times. You can preview the structured data that kimono will extract in the data preview pane and make more granular adjustments in the data model view.
Click DONE to create an API. Select daily to make sure your data is refreshed every day. Once it’s done, click the link to check out your new API.
(Pro-tip: if you are having trouble selecting just the data you want, try clicking and dragging to select just the part you want and kimono will strip off extraneous text.)
You can view the first page of data extracted on the API detail page for your new API.
The API we just set up only extracts from one page of rides. If you have several pages of data, you’ll need to configure your API to extract from multiple pages as well. To do this, go to the crawl setup tab, click on ‘crawl strategy’ to specify the type of crawl you want to do – select generated URL list in this case. Then, on the lower right you will see the URL generator.
Kimono has broken your source URL into its relevant sub-components. For us, the number after ‘trips’ specifies the page you’re on. To the right of the number parameter, click ‘range’ and specify 1 to 20 (instead of 20 use whatever number corresponds to the total number of pages of data that you have).
You will see a list of URLs generated by kimono below. Click save changes, then hit ‘start crawl above’. Once the crawl completes, go back to the data preview tab and download the CSV. Open it up in excel and remove the top row – the row that says ‘collection1’ to get it formatted for use with cartoDB.
Now that we have our structured data set, let’s start mapping our route data.
Log in to your cartoDB account and select ‘tables’. Click the large plus on the right to add a new table.
Choose ‘data file’ and select the kimono csv file that we just downloaded. Once the data is loaded into cartoDB, click the drop-down next to the property with your start station data. Select ‘georeference’ to translate this into coordinates, i.e. latitude/longitude pairs.
Select referencing ‘by street address’, specify the city and country and hit continue. You’ll see a new column appear with latitude and longitude data for each station. We’re almost done.
At the top, click on ‘map view’ and click the wizard/wand icon on the right.
To create an animated map with categories, select ‘Torque Cat’, then use the drop down menus to set ‘time column’ to your start time property. Then set the category column to the end station and use the fields below to map colors to end stations by region, for example.
Ta-da! You’re done! You just built an awesome animated map. But, suppose you wanted to calculate a few more interesting things and plot the output? With kimono’s filter functions you can do just that. We’re beta testing this feature right now, so just email us at email@example.com and we’ll give you early access to the feature.
Copy in our frequency function below, to start:
Using cartoDB’s bubble plot setting, we can quickly turn this into a heatmap of where Andrew spends his time.
That’s just a quick preview of some powerful maps you can build with kimono and cartoDB. We’re excited to see what you will build with the tools. Tell us what you create at firstname.lastname@example.org and reach out to us at email@example.com if you get stuck.
You have probably heard about Microsoft’s work this past year, in training a neural network to polygonize satellite imagery into building footprints. This work resulted in ...Cartography & Visualization
Please fill out the below form and we'll be in touch real soon.