Today at the Spatial Data Science Conference in New York City we announced CARTO Workflows, an exciting new tool that enables users of all types and skill levels to harness the power of cloud data warehouses, spatial SQL, and advanced spatial analytics.
Spatial data and analysis are increasing in complexity and scale. It is no secret that more and more geospatial data is being produced. Larger volumes of data and more frequent updates mean insights derived from geospatial data can become stale quickly, and business demands for accurate updates are only increasing.
This shift puts an increased burden on geospatial teams and roles of all types. From data engineers who need to create consistent and scalable pipelines, to analysts who need to ensure data is accurate and up to date, to data scientists and developers who have to work with larger volumes of data, all of which can slow progress down.
Traditional infrastructure cannot meet the demands of the modern geospatial team. The good news is we have the tools to solve these problems. Cloud data warehouses that make use of spatial SQL provide a scalable and cost effective infrastructure to manage larger volumes of data. Spatial indexes like H3 and Quadbins make data storage and querying much more efficient. And new file types such as Geoparquet or Cloud Optimized GeoTIFF make data interoperability between systems even more performant.
However these can be difficult to implement in practice. For example, implementing a complete shift to H3 as a geospatial indexing system can be complex depending on your technical stack, while still having to deliver on critical business demands. This complexity only increases with the scale and volume of data - a task as simple as creating a dashboard can become overwhelming quickly.
These shifts make it even more important to be able to unlock accessible data and insights across your organization. The risk of siloing geospatial data and insights for critical decisions is high given the current specialized knowledge and tools used by geospatial teams.
For close to two years, we have been working on the vision of a complete geospatial platform that leverages the power of the cloud, in particular cloud data warehouses. We believe there are three major shifts that are driving innovation in geospatial analytics today:
It is this vision that we have been working towards and building since 2020 with open-source components and alongside our amazing cloud partners, including Google Cloud, Snowflake, AWS and Databricks.
Spatial SQL provides a common language that can be used to perform spatial analysis and harness the parallel processing power of data warehouses. With that said, spatial SQL can be simple, but generally looks more complicated in practice.
We can also see this trend. In the State of Spatial SQL Report, we found users of all types are using spatial SQL: Yet spatial SQL is generally learned on the job or independently, and 74% of respondents are only using it weekly or less:
CARTO Workflows is built on the premise of making this analytic power more accessible, for users of any skill level or role.
Workflows allow you to:
Workflows can be scheduled to run at specific times or based on changes to data inputs in remote files or data in your data warehouse. You can create CARTO Builder Maps, export data, send emails, and more as outcomes of Workflows. Each workflow can be parametrized, duplicated, combined, and shared.
Let’s take a look at this in practice. First, we can take a list of customers, and geocode those addresses to add points to the dataset. We can also import the Wildfire Risk Dataset from NASA with an import node. Since we only want to see results in California, we can then grab a dataset of state shapes and filter it to only show California. From here we want to perform a spatial join to find fires within the state. Next we want to ensure we find customers that are near fires, not only within the fire area with a buffer of 50km.
Finally we can join the buffered customer locations and the fires in California. The final output of the Workflow will be a CSV report sent via email to the necessary stakeholders.
At any point you can see the results of the analysis, and make changes to the Workflow at any time by adjusting any single node.
Below is the comparable SQL to run the analysis in this Workflow, not accounting for the CSV export and email trigger.
WITH --Source datasets customers AS (SELECT * FROM california_section.crm_data.customer), geocoded_customers AS (ST_GEOCODE( customers, address, 'United States of America') AS geom), nasa_fires AS (SELECT imported.* FROM ( SELECT URL_IMPORT("https://www.earthdata.nasa.gov/learn/find-data/near-real-time/firms/active-fire-data#", true) as imported), state_boundaries AS ( SELECT * FROM california_section.spatial_data.usa_states_boundaries WHERE name = 'California'), --Intermediate analysis nasa_fires_california AS (SELECT nasa_fires.* FROM nasa_fires, state_boundaries WHERE ST_INTERSECTS(state_boundaries.geom, nasa_fires.geom)), buffered_customers AS (SELECT `carto-un`.carto.ST_BUFFER(customers.geom, 5000, "meters",8) as geom from customers), --Final analysis SELECT nasa_fires_california.* FROM nasa_fires_california, buffered_customers WHERE ST_INTERSECTS(buffered_customers.geom, nasa_fires_calfornia.geom)
CARTO has always worked to make advancements in geospatial accessibility for all users. Workflows represents our newest effort in this area, to bring modern techniques and scale to a wider audience. Everything from simple enrichment, dashboard creation, spatial models, feature creation, and more can be built with workflows. You can create any number of outputs:
We are excited to see how you will make use of Workflows in your work. We are currently in a private beta, and releasing the public beta in the coming weeks. Please sign up below if you are interested in participating in the beta evaluation process, as your feedback will be critical to make this new tool helpful for all users.
When CARTO was founded in 2012, the main areas of focus were conservation and biodiversity projects. Fast forward exactly 10 years and CARTO is proud to be one of the leadi...Use Cases
Please fill out the below form and we'll be in touch real soon.