All Models Are Wrong, But Some Are Useful
Embracing Uncertainty in Data Analysis
Back when I first dipped my toes into the world of data analysis, I remember my statistics professor sharing a quote that would eventually become a guiding light in my career:
“All models are wrong, but some are useful.”
At the time, I didn’t fully appreciate its weight. It felt like an abstract observation—something to keep in the back of my mind. However, it wasn’t until I met Jack, a seasoned data scientist, that I began to see the real-life impact of this idea on digital services and, more importantly, on understanding human needs.
In this article, I want to take you on a journey—a story of discovery, humility, and action in the face of uncertainty.Along the way, I’ll share personal anecdotes and insights that have helped shape my approach to data. Think of this as a conversation between friends. So, grab a cup of coffee, settle in, and let’s explore how embracing uncertainty can lead to better decisions, more effective communication, and ultimately, solutions that serve real people.
The Expectation Gap: When Reality and Assumptions Don’t Meet
Let’s start with a common scenario in our tech-driven world. Picture the sleek, modern offices of tech companies and government agencies. Every day, product managers, engineers, and stakeholders engage in what I like to call the dance of expectations. There’s an intricate interplay between ambitious timelines, lofty goals, and, all too often, a gap between what’s expected and what’s actually possible.
I remember a particular conversation with a colleague during one of those long strategy meetings. “There’s going to be times where different stakeholders have no idea about the amount of time it takes to build and refine a product,” I said, leaning back in my chair as I recalled one of my early career lessons. This line of thought always reminds me of the “90-90 rule” in software development: the first 90% of the code takes 90% of the time, and the remaining 10% takes another 90%. It’s a humorous yet poignant reminder that our best-laid plans often collide with the realities of iterative development.
Personal Anecdote: Learning the Hard Way
I recall an early project when I was tasked with developing a dashboard. I underestimated the complexity involved in gathering, cleaning, and visualizing the data. Stakeholders were excited about a flashy final product, but the process was fraught with unexpected delays and challenges. This misalignment of expectations taught me an important lesson: communicating the intricate details of data work is as crucial as the final product itself. Like many of us, I had to learn that managing expectations isn’t just about delivering on time—it’s about being transparent about the process and the uncertainties involved.
When Data Whispers Instead of Shouts
In the realm of data analysis, there’s a pervasive myth that more data equals more certainty. This belief, which I refer to as the Data Certainty Fallacy, suggests that with enough numbers and enough analysis, we can achieve absolute answers to every question. It’s a comforting thought, but as I’ve come to learn—and as my experiences have shown—it’s often misleading.
A Tale of Two Projects
Let me share a story that illustrates this point. Not long ago, I was involved in a project with a state government agency. The issue was seemingly straightforward: eligible citizens were visiting a benefits website, exploring the information available, but then, for reasons unknown, they weren’t signing up for the assistance they desperately needed. The digital footprints were there, and the engagement metrics were recorded, but the conversion was missing.
This situation brought to mind an old study from a grocery store in 2000, where researchers discovered that increasing the variety of jams on display actually decreased sales. The conventional wisdom that more choice leads to better outcomes has been challenged, revealing the subtle complexities of human behavior. In our digital benefits case, the data seemed to suggest that something was amiss, but it wasn’t immediately clear what the underlying issue was.
The Detective Work: Following the Digital Breadcrumbs
One of the most fascinating aspects of data analysis is the detective work involved. In the benefits project, as I poured over the analytics, I noticed an intriguing pattern. Users were not following a smooth, linear path through the website; instead, they were bouncing back and forth between the service page and the unemployment page. It was as if they were pacing in front of a locked door, desperately searching for a key that simply wasn’t there.
The “Aha!” Moment
I remember sitting with the team, staring at the flow diagrams and the heat maps, when the pattern finally clicked. I turned to my colleagues and said, “I see a loop here—people are shuttling between these pages. It tells me something is off.” This wasn’t a conclusion drawn from a grand experiment or an overwhelming set of data points; it was an observation—a clue—that pointed to a deeper problem.
The theory that emerged was simple yet profound: people weren’t finding the information they needed to move forward. However, I was cautious. I knew that data, no matter how compelling, can only take us so far. It’s a window into behavior, not a definitive explanation of motives.
Reflecting on Uncertainty
At this juncture, I was reminded of a personal experience early in my career when I worked on a project analyzing customer churn for a retail company. We had access to vast amounts of data, yet the reasons behind customer departures remained elusive. It wasn’t until we conducted follow-up interviews—talking directly to the customers—that we uncovered the actual issues, such as dissatisfaction with customer service and unmet expectations. That experience taught me that while data is robust, it often whispers its truths rather than shouting them loudly.
The Humility of Uncertainty: Embracing What We Don’t Know
One of the most liberating lessons in data analysis is recognizing and admitting the limits of what we can know from numbers alone. In the benefits project, I came to a pivotal realization:
“The only way that I can actually know why someone left the site without taking action is if I ask them.”
Embracing the Unknown
At first glance, this admission might seem like a concession to imperfection—a sign of weakness in our data-driven world. But in reality, it’s a mark of intellectual humility and scientific integrity. This approach is akin to the scientific method itself: we propose hypotheses, test them, and always remain open to the possibility that our understanding is incomplete.
In my own work, I’ve found that acknowledging uncertainty opens up opportunities for richer insights. Instead of asserting absolute conclusions based solely on data, I learned to pair analytics with qualitative research. For instance, after noticing the back-and-forth pattern in the benefits project, we conducted user interviews to understand the underlying issues. These conversations revealed that the website’s navigation was confusing, and the information architecture did not align with user expectations.
Personal Anecdote: Learning from a Misstep
I once made the mistake of presenting a data-driven solution to a client without highlighting the underlying uncertainties. The data looked robust, and the proposed changes were backed by solid numbers. However, when the results came in, they didn’t match our projections. The client was disappointed, and I realized that I had overstepped by not communicating the nuances and potential blind spots in our analysis. Since then, I have always made it a point to say, “Here’s what we think is happening, based on the data—and here’s what we’re not entirely sure about.” This transparency not only builds trust but also paves the way for iterative improvements.
Taking Action in the Face of Uncertainty: Informed Uncertainty as a Strategy
The next chapter in our story is about turning uncertainty into a catalyst for action. Even when we’re not 100% sure of our conclusions, there comes a point when we must act. This is what I call “informed uncertainty”—making decisions based on substantial evidence, while openly acknowledging the limits of our knowledge.
The Benefits Project: A Case Study in Action
In the state benefits project, despite the uncertainty about the exact reasons behind user drop-offs, we decided to make a few targeted changes to the website. The goal was to make the critical information more accessible and to streamline the navigation process. We didn’t wait for every detail to be ironed out; instead, we moved forward based on the evidence we had gathered.
“Even though we weren’t 100% sure that my theory was correct, we were confident enough to take action,” I remember explaining to the team. The results spoke for themselves: post-implementation metrics showed a significant improvement in user engagement and an uptick in benefit applications.
Reflecting on the Experience
This experience was a turning point for me. It reinforced the idea that waiting for perfect certainty can be paralyzing. In the fast-paced world of digital services, acting on the best available evidence—while staying open to adjustments—often leads to better outcomes than endless deliberation. It also reminded me of a time when I was working on a new feature for a mobile app. Despite the initial data being somewhat inconclusive, we rolled out a beta version to a small group of users. Their feedback, though mixed at first, allowed us to refine the feature iteratively, leading to a robust final product that resonated with our users.
The Art of Data Storytelling: Communicating Uncertainty with Clarity
If there’s one skill that every data analyst should master, it’s the ability to tell a compelling story with data. But what happens when that story is imbued with uncertainty? How do you communicate the nuances and gaps in your analysis without undermining your credibility?
A Conversation with Stakeholders
The key lies in honestly framing the narrative. When presenting findings from the benefits project, I didn’t claim that I had all the answers. Instead, I framed my insights as a theory supported by evidence. “I have a theory that people are not finding the information they need to take action,” I would say, and then I’d walk the audience through the data points and user feedback that led to this conclusion. This approach, combining candor with clarity, helped bridge the gap between the technical world of analytics and the practical concerns of stakeholders.
Personal Anecdote: From Data to Dialogue
I recall one particularly challenging presentation early in my career. I was tasked with explaining a complex set of metrics to a group of non-technical stakeholders. Instead of overwhelming them with charts and statistics, I decided to share a story—a narrative of how our data mirrored the customer journey. I compared the experience to trying to navigate through a maze with missing signs. The room lit up with understanding, and what could have been a dry presentation turned into an engaging dialogue. That experience taught me that data, at its best, is a tool for connection, not just calculation.
Practical Tips for Effective Data Storytelling
Start with a Clear Hypothesis: Begin your presentation by outlining the theory or question you’re exploring. This sets the stage for why the data matters.
Use Visual Aids Thoughtfully: Graphs, flowcharts, and heat maps can illustrate patterns and trends, but they should be simple and focused.
Acknowledge Uncertainty: Be upfront about the limitations of your data. Phrases like “Based on our current evidence…” or “While we’re still exploring…” can build credibility.
Invite Questions: Encourage dialogue. Allow stakeholders to ask questions and express concerns. This not only deepens understanding but also fosters a collaborative atmosphere.
Connect to Real-World Impact: Always tie the data back to human behavior. Explain how these insights can lead to better decisions and improved outcomes.
The Human Element: Why Data Analysis Is More Than Just Numbers
At its core, data analysis is about people. Behind every data point is a human being with a unique story, needs, and behaviors. This understanding is what transforms raw data into meaningful insights and actionable strategies.
Remembering Why We Do This
There’s a moment, often quiet and reflective when you realize that all the hours spent wrangling data, learning SQL, and perfecting your models have a deeper purpose. For me, that moment came when I saw the direct impact of our work on people’s lives. When users found the information they needed on a government website, or when a well-designed app feature made everyday tasks easier, I felt that the technical challenges were completely worth it.
Personal Anecdote: The Joy of Impact
I’ll never forget the day I received an email from a user of a service we had improved. The email was simple, thanking us for making it easier to access critical benefits. It wasn’t a glowing review or a detailed case study—it was a heartfelt note from someone whose life had been made a little bit better through our efforts. That email reaffirmed my belief that behind every statistic is a story, and that our work, flawed as it might be, is ultimately about serving real human needs.
Bringing It All Together
This understanding of the human element is what brings us full circle to that old quote: “All models are wrong, but some are useful.” In the end, data analysis isn’t about achieving perfect certainty or flawless predictions. It’s about using imperfect models to uncover insights that help us make better decisions, improve experiences, and ultimately, help people. Whether you’re a product manager, a stakeholder, or a fellow data scientist, the lesson is clear: embrace the imperfections, celebrate the uncertainties, and always remember that data is a tool for understanding the complex tapestry of human life.
Final Thoughts: Embracing Uncertainty and Moving Forward
The journey from raw data to meaningful insights is rarely a straight path. It’s filled with unexpected twists, moments of doubt, and the ever-present challenge of uncertainty. But it’s precisely in this uncertainty that the true art of data analysis lies.
A Reflection on My Journey
Looking back on my own experiences—from those early lessons in college to complex projects with government agencies—I’ve come to appreciate the beauty of not having all the answers. It’s in the questioning, the iterative testing, and the willingness to adjust our course that we find real progress. I often think of the maze analogy: navigating through unknown territory with only a few clues and a lot of curiosity. Each step, even if taken in uncertainty, brings us closer to understanding and, ultimately, to creating solutions that make a difference.
The Courage to Act
One of the most powerful takeaways from my journey is that sometimes, the most courageous thing you can do is act despite the uncertainty. Whether it’s tweaking a website to better serve its users or presenting a nuanced theory to skeptical stakeholders, the willingness to take action—even when you’re not 100% sure—can lead to breakthroughs that pure certainty never would.
Looking Ahead
As we move forward in this era of data-driven decision-making, I encourage you to adopt a mindset of curiosity and humility. Embrace the fact that every model, every analysis, and every strategy is inherently imperfect. But remember that their usefulness isn’t diminished by their imperfections—it’s defined by their ability to spark action, foster understanding, and ultimately, improve the human experience.
So the next time you’re faced with a daunting dataset or a perplexing user behavior, take a deep breath and say,
“I have a theory.”
Then, use the data as a guide, communicate your insights with honesty, and act with both confidence and the willingness to learn from what you don’t know.
In Conclusion
Data analysis is not just about numbers—it’s about the stories behind those numbers and the people they represent. It’s about understanding that every chart and every graph is a window into human behavior, replete with its complexities, uncertainties, and surprises. In the end, it’s our responsibility, whether we’re data scientists, product managers, or stakeholders, to honor that complexity by being transparent about what we know and what we’re still trying to understand.
For me, that’s what makes all the late nights learning SQL, all the frustrating moments of debugging code, and all the challenges of data cleaning completely worth it. It’s about making a tangible difference in people’s lives, even when the road ahead is a little murky.
Thank you for joining me on this journey. I hope that by sharing these experiences and insights, you feel empowered to embrace the uncertainties in your own work. Remember: every theory, every observation, and every action taken in the spirit of inquiry brings us one step closer to making sense of the beautiful, messy world we live in.