Modern tech companies have never been more willing to invest in data analytics tooling and centralized data teams, yet at the same time these efforts often end in frustration and wasted potential for both the business teams that expect data support and the data analysts themselves. On the business side, instead of seeing improvements to the clarity and availability of reporting, metrics become more obfuscated and dependent on an overworked data team; at the same time, data analysts don’t spend any of their time doing actual data analysis and become disillusioned spending their time as glorified SQL engineers. In some cases, it is truly the “worst of both worlds” with the outcome being unsatisfying for both parties.
Having been part of data teams with various levels of success in navigating dysfunctional organizations and poor positioning, I have come to believe that there is a fundamental contradiction in how a centralized data function is positioned: data teams are defined and organized with respect to a relatively limited set of technical skills (manipulating data), but the intended impact of data teams is much broader and involves skills that overlap considerably with the business teams that they support. As long as this contradiction exists, data analytics will always be a fraught proposition at any company.
Personally, I still believe there is tremendous value that can be added via a well-implemented analytics practice, but this is often in spite of these underlying challenges. There’s no way to resolve this fundamental challenge, only to cope with it, and it’s not surprising that so many organizations are unable to thread this needle.
Data analytics teams typically have two layers of scope: nominally, they exist to partner with business teams to provide business intelligence and “insights”. Broadly speaking, this might be articulated as “getting value out of our data”, and involves both managing the technical aspects of serving data to stakeholders as well as the analytical aspects of deriving and communicating these insights.
In practice, data analysts rarely take on this breadth of scope when partnering with business teams. After all, these teams already likely have grunts with titles like “sales analyst” or “operations analyst” whose jobs are to wade through excel documents and put together TPS reports for management. The perceived bottleneck for these teams is not “insights”; it’s the manipulation and preparation of data. This is where the de-facto scope of a data team becomes clear: data analysts are on a separate team and defined by their technical skillsets in manipulating and accessing data, not their analytical prowess. When analysts partner with other teams, they are called upon to solve “data-shaped problems” rather than actually conduct data analysis, because the former is what actually separates them from the rest of the business.
This is not to say that all data analysts lack analytical skills, but this pattern is common enough that many junior analysts never have a chance to develop them. They simply come to expect that being a data analyst means operating a dashboard help desk. Some of them, having had higher expectations, will become disillusioned and move in a new direction, either crossing over to the business side or leaning into technical skills and becoming a data engineer or “analytics engineer”. Others will persist as mediocre pseudo-analysts, and may even thrive in their own way as part of this corporate niche.
Efforts to proactively engage with the business analytically often end in failure and disillusionment anyway. Business leaders are unlikely be inclined to listen to some kid from another team who wants to help them “see the big picture”, and in a lot of cases they’re right! They’ve already staffed their team with experts with experience in the relevant business domains, they’re probably dealing with some amount of operational dysfunction resulting from poor data quality and availability, and they really need someone who can solve these “data-shaped problems” to enable their existing team members, not supplant them. In the worst case, a business leader might see a centralized data team as a threat to their budget, since data analysts are attempting to both streamline and take on substantive work that their team is meant to do.
As long as data exists as a separate team in an organization this problem is a structural problem that can’t be solved in any simple way. Some organizations have attempted to sidestep the issue by creating decentralized data functions, with individual analysts placed directly in each business team rather than as part of a centralized team. However, this tends to unleash a lot of chaos on data infrastructure which needs to be somewhat centralized (and if I keep hearing the meaningless phrase “data mesh”, I’m going to set myself on fire).
Data is supposed to be an analytical function that sits outside of the operational functions of a business. This creates the opportunity to evaluate the effectiveness of those operational functions and guide them via prescriptive analysis. However, without any kind of operational component, data teams basically have no power in the organization: they don’t “do” anything! The ability of a data team to actually influence the strategy or direction of the organization as a whole basically comes down to the power of persuasion and the ability of individual data team members or leaders to convince actual decision makers in the company to change course based on the results of analysis.
While it’s not a bad thing that data practitioners should be adept at persuasion, in practice the separation between data teams and the rest of the business turns this process into an exhausting uphill battle that quickly burns out analysts. This is exacerbated in cases where analysts don’t have the necessary domain expertise to make recommendations, which can lead to a negative feedback loop: without domain context, analysts can’t add value, and without adding value, analysts can’t build the right level of trust to get that context.
This friction ultimately pushes most data teams away from even trying to be an analytical function and instead incentivizes them to focus on the basic operational aspects of data engineering: building structured data models and dashboards to the specifications handed down by business teams.
This is a real shame, because most companies still have a lot to gain from a centralized analytics function. Data analysts have the opportunity to provide business teams with an outside perspective on their performance and strategy that’s top-down and derived from first principles, rather than day-to-day operations. To be clear, both the analytical and operational perspectives are important, but blending these viewpoints helps teams stay accountable and avoid bias.
We can’t ever solve for the tension and friction that comes from having a separate data team, but we can cope with it in a few key ways:
Despite my own cynicism, I’ve witnessed enough glimmers of hope throughout my career to still believe that data analytics can be successfully implemented at organizations of any scale. My freelance work in particular has been a great opportunity to gather more data points on what works and what doesn’t work when introducing these practices at different organizations. More than anything, I’ve learned that despite the presumed technical complexity of the data world (and the vast number of useless data-related tools and services being hawked on LinkedIn), the key to conducting effective analytics comes down to keeping it simple by focusing on meaningful business outcomes and being helpful and responsive to stakeholders.