Much has been written about the high failure rate of data science projects. Data science teams often have difficulty moving their insights and algorithms into business processes. At the same time, business teams often can’t articulate the problems they need solved. And they ignore their cultural resistance to data science.
There is a designed-in structural tension between business and data science teams that needs to be recognized and addressed. Structural problems demand structural solutions, and we see a way forward through a data science bridge: an organizational structure and leadership commitment to develop better communication, processes, and trust among all stakeholders.
To understand the challenges, we use the metaphor of a factory (standing in for any sort of business operation) and a data science lab (such as R&D). The “factory” might literally be a factory, but it could also be a business operation that produces decisions on mortgage applications, reads MRIs, or drills oil wells. Factory goals may involve meeting production plans and keeping unit costs low. The watchword for these operations is stability: no major disruptions and few surprises. Factory managers work very hard to establish and maintain stability, so quite naturally they resist anything that threatens it. One important exception is that good factory managers support incremental improvement when it’s quickly followed by stability at a new level, provided that it is driven within the factory. We call these efforts small data projects.
The data science lab, on the other hand, is specifically designed to disrupt the factory. Its job is to find bigger improvements than can be made with small data, change the way decisions are made, and come up with new products that obviate old ones — with a focus on large, complex, unstructured, and data-rich problems. The lab represents the very antithesis of stability — exactly the sorts of things that factory managers hate!
A certain amount of friction between any two departments is to be expected. Factory managers do not appreciate interference from finance when it changes accounting practices, or from HR when it mandates improvements to its employee review system. But these are mere annoyances; they don’t change the ways the factory fundamentally does its core work.
Contrast these annoyances with the full-frontal attack and disruption signaled by the lab. In some companies, this tension has become so toxic that it has led to two informal rules in the factory: “Never ask the lab for help,” and “If the lab offers help, refuse.”
Tension Is Baked Into Traditional Company Design
For many companies, there’s value in separating business operations from the data team. An emphasis on stability in operations keeps costs low. An emphasis on long-term, disruptive work by the data team is essential to foster innovation and future success.
But these divergent goals can create tensions that have serious repercussions. Lack of communication makes it more difficult to identify the factory areas in greatest need of innovation. Poor communication cripples the factory’s ability to take advantage of lab inventions. Over time, the factory falls further behind as potential innovations stack up. Nimbler competitors cut into the factory’s business. The best data scientists leave, seeking opportunities where their work will be appreciated.
This tension is inevitable: Its root lies in the organizational design choice to manage the two separately.
Credit that model to Thomas Edison, who is generally recognized as launching the world’s first industrial research lab in Menlo Park, New Jersey, in the late 19th century. His focus was on inventing the next generation of products and services. Edison seems to have recognized that embedding lab resources into a factory, where there would be distractions from day-to-day production responsibilities, would not work. As we have seen ourselves in organizations that have attempted this organizational model, lab resources located in-house are quickly pulled into day-in, day-out stability issues, effectively thwarting their efforts at true invention. Conversely, by separating the lab from the factory, all too often the lab becomes isolated, making it the proverbial ivory tower.
Edison was able to mitigate the tension because he led the lab while also controlling the factory. But for most other organizations since then, this ongoing tension has never been adequately resolved.
We share this history to make an important point: Strain between business operations and data teams did not originate with the rise of data science. Rather, data science labs have been engulfed in a long-existing, unresolved battle. Based on our conversations with experts, including Tom Davenport (Babson College), Blan Godfrey (North Carolina State University), Ron Kenett (University of Jerusalem), and Ron Snee (Snee Associates), this friction is tacitly accepted and recognized in all industries, types of organizations, and locations all over the world.
So, what’s new now? The explosion of data science has changed things in the factory. There’s a need for better decisions, better data-driven processes, product and service improvements, gains in efficiency, and reduced costs. In the lab, the sheer volume of potentially useful data and of people qualified to analyze it have both grown considerably. Resolving the tension is both more urgent and more potentially beneficial than ever before.
A Solution: The Data Science Bridge
As previously noted, structural issues require structural remedies. Our proposed solution is for companies to work to create a data science bridge. To extend our factory/lab metaphor, imagine that the factory and lab sit on opposite sides of a river. A bridge spans the river and connects the two, resolving some of the essential strains and enabling the introduction of more, and more useful, data-driven innovations into the factory.
A data science bridge would have four major responsibilities:
- Developing and maintaining high-bandwidth, bidirectional communication channels between the factory and the lab. This includes developing a common language (so the factory and lab don’t talk past each other), identifying and clarifying which innovations are most needed (so the lab focuses on the right things), and ensuring that feedback is provided and understood.
- Developing and operating a process by which lab inventions are made fit for the factory. This may include embedding newly minted algorithms into factory technologies and/or IT systems and training factory employees.
- Selecting and allocating needed resources. People and funding would be assigned to prioritized innovative opportunities.
- Building trust between the factory, the lab, and senior management. Ultimately, only trust can mitigate the tension.
We see forerunners for this bridge in a technology transfer process that connected Bell Labs and AT&T a generation ago, more generic D4 (data, discovery, delivery, dollars) processes, and the analytics translators who stand between technical and business organizations.
Selection of the individual to lead the bridge, perhaps with the title innovation marshal, is the most critical decision in its construction. This person cannot be simply a seasoned manager or the most talented data scientist. Tom Davenport and Wayne Eckerson have written about the need for “purple people” in the sense of being well respected by both the factory (red) and the lab (blue). To be effective, this person must report to the CEO and have adequate funding to sponsor improvement projects requiring lab innovations to enhance the factory. Without funding and direct access to the CEO, the bridge will devolve into a passive organization offering advice or, even worse, into a “bad cop” criticizing the work of both the lab and factory and being resented by everyone. Bridge leadership must be on equal footing with lab and factory leadership.
To build a sturdy, sustainable bridge, there are several questions that organizations must ask themselves. A starting point is reaching agreement on an operational definition of data science, analytics, or whatever name the lab goes by. Definitions vary greatly, so defining terms is important for enabling productive dialogue. Other preliminary questions include the following:
- What current lab projects would most benefit from better business connections and could serve as test cases for the bridge?
- Who in the factory needs to be on board first? Who in the lab?
- Are there candidates to lead the bridge from within the factory or lab who are respected enough by the other function to be effective?
- How do we align the bridge to the company’s strategic interests?
Getting Started: Footbridges Lead to Bigger Bridges
In many cases, the leader of the data science lab has the most to gain and could be the player to reach out to the factory leader to initiate dialogue. Initially, a footbridge, or informal connector between the lab and factory, may suffice. Open-minded lab and factory leaders can take the initiative to discuss the concept across organizational boundaries and go to senior leadership with specific proposals. (They should abide by the principle of bringing senior leaders a solution rather than a problem.)
Lower-level managers and technical resources do not need to wait for top-down direction either. We propose that they identify areas within their own organizations where the tension between the lab and the factory is inhibiting progress. They can begin a dialogue to discuss how to foster better cooperation. A series of discussions on addressing the tension also constitutes the beginnings of a footbridge.
Eventually, of course, in order to achieve a sustainable solution, top-down direction needs to intersect with bottom-up, and a structural solution will be required. This means creating the bridge organizationally: that is, naming a senior executive to lead the project, and funding the improvement initiative. Only the CEO can do this.
The Time to Act Is Now
Competition in the data science space, including competition for qualified resources, has never been fiercer than it is today. Frustrated data scientists can and do leave for greener pastures, taking their talents where they might be put to better use. While numerous efforts are being made to better align data science labs with business operations, we suggest that these efforts will continue to be insufficient because they are not based on a proper diagnosis of the root causes of the problem.
Only by taking structural steps to address the deeply entrenched frictions can organizations expect to reap the full benefits of their investments in data science. We propose that a data science bridge, hinted at but not fully elaborated by previous authors writing about this topic, is the most logical step forward.
Now is certainly the time to act. Senior leaders have a unique opportunity to resolve a previously unrecognized and debilitating tension, thereby putting their data science initiatives on a new and more productive path.
To Succeed With Data Science, First Build the ‘Bridge’
No comments:
Post a Comment