Key takeaways
- Data-drivenness is about decision change, not dashboards.
- Measure how often data alters outcomes.
- A simple thought experiment reveals gaps.
FAQ
What's the core metric?
The rate at which data changes decisions. If choices don't change, dashboards don't matter.
How can teams improve?
Make hypotheses explicit and review results. Close the loop after decisions.
Airbnb runs 8,000 data pipeline tasks every single day. How many does your company run?
Most organizations have no idea. And that might be the problem.
The Insight
I stumbled on this metric while watching a talk by Maxime Beauchemin, a data engineer at Airbnb and creator of Airflow. Airflow's origin story is a useful reference point for why orchestration became so central to data work.[1] Watch 50 seconds of this video (7:49–8:39):
As of six months ago, Airbnb processes 8,000 daily ETL tasks, orchestrated on a cluster of computers that play air traffic controller. What struck me: most companies don't even know their number.
Data maturity consistently correlates with performance outcomes, which is why a simple metric can be so clarifying.[2]
Also interesting—authoring pipelines is not centralized inside Airbnb. Data engineers, data science, engineering, growth, and engagement people all write their own workflows.
The Metric
So you want to be data-driven? Put your data assets to work?
ETL jobs and data pipelines actively put your data assets (internal data, third party data, open data) to work. The more data processing you do, the more data-driven you likely are.
Could Daily ETL Tasks ("DETLT") be an effective indicator for data-drivenness?
The Thought Experiment
If you wanted to boost your company's DETLT, what data processing jobs would you create? Assuming you're generating valuable data and sharing it with stakeholders, wouldn't this have a massive positive effect on the business?
Most of us don't know how many data pipeline tasks our company runs each day. We have no sense of how that metric has trended historically.
Could be an opportunity.
— Ry
Related Essays
How to Succeed in the Data Revolution
A self-assessment and prescriptions for becoming data-driven: hire a data scientist, establish a warehouse, and track your Daily ETL Tasks.
From Behavioral Analytics to Data Science
How Astronomer evolved from USERcycle analytics to a data engineering platform. The organic growth from clickstream to Apache Airflow to Kafka.
The Growing Data Opportunity
Data silos are a growing problem as SaaS proliferates. How cloud computing, mobile, and machine learning are creating a data revolution.