If you are interested on the impact of big data on business, you might want to check a new paper that I co-authored with Alessandro Merendino, Sally Dibb, Maureen Meadows, Lee Quinn, David Wilson and Lyndon Simkin; and which has just been published in the Journal of Business Research. This paper reports on a research initiative to investigate how big data impacts on strategic decision making, which received financial support from (1) the Economic and Social Research Council, Digital Social Research programme, and (2) NEMODE, an initiative under the Research Councils UK (RCUK)’s Digital Economy (DE) research programme.
The last few years have seen the emergence of new roles and new techniques to support decision making. Yet, little is known about how those in very senior positions draw on big data, or not, to make critical decisions on behalf of the organisations that they lead. We studying decision making by board directors; the way boards work; and associated changes across the organisation. To that end, we collected data from 19 organisations and 20 board level managers, in financial services, management consultancy, FMCG manufacturing, air travel, information technology, leisure, etc… as summarised in table 1. [Side note: it is NOT easy to get access to board level interviewees; it is even harder to convince them to talk about the challenges that they are facing; and, when you do manage to overcome those challenges, it can be very hard to find a time in their diaries for an interview.]
You can read the whole paper, here. [<–This link should take you to an open access version of the paper. If it doesn’t work, you can access a pre-print copy of the paper, here]. Below, is a summary of the findings.
Impact of big data on decision making – How managers think and act
First, we looked at managerial cognitive capabilities, i.e. the mental models and skills that allow them to perceive, analyse and process information about the environment. We discovered that most of our interviewees are finding it difficult to integrate big data resources in their decision-making processes, to develop related capabilities, due to the following three challenges.
A shortfall in cognitive capabilities – Directors recognise the potential of big data to improve the quality of the decisions that they need to make, as well as of the process of making those decisions. However, many confess to feeling ill-equipped to do so, be it in terms of their own technical skills, or in terms of new type of (non-linear) thinking required.
The presence of cognitive biases – When processing the information available, the directors often found themselves reverting back to old ways of thinking about their organisations or about how to use the information available, a phenomenon that is known as anchoring. This tendency was further composed by the common practice by others elsewhere in the organisation of only providing the directors with simplified, data synopses.
The occurrence of cognitive overload – This refers to what happens when individuals are exposed to more information than they are able to process. That is, while big data have the potential to improve the quality of decision making, many directors struggle to cope with the variety and quantity of information available. As one interviewee put it, ‘the people who are doing well are the ones that can cut through all the crap and make decisions based on (key) facts.’
Impact of big data on decision making – boardroom routines and dynamics
Second, we looked at how boards use big data to make decisions, and in particular, the impact on board routines and dynamics. We found evidence of big data threatening the cohesiveness of board-level decision-making, as a result of the three following effects.
Process disruption – Many boards are facing big changes not only externally, but also in terms of how their organisations work or even the roles of those sitting on the boards. They need to rethink how they do business; however, they can’t ‘hit pause’ while they make sense of the challenges and adjust their ways of working. So, instead, what we see is a delicate balance between attempting to change the systems in place while needing to continue to use them in order to keep functioning.
Clash between velocity of big data and the pace of strategic decision-making – Big data offers the possibility of capturing evidence quickly and in real-time. However, this is at odds with the board’s ability to respond quickly – for instance, the interviewees mentioned the challenge of deciding when to stop consulting the data that keeps being generated and make a final decision. There is also a clash between the board’s need to be future oriented vs. big data’s focus on the present or, even, the past. As one interviewee put it, ‘(Big data) is a hindsight thing, and strategy is a foresight and intuitive thing’.
The suitability of the board’s composition – Interviewees felt that, at its core, strategic decision making had not changed, in the sense that it still requires intelligence, ability to identify and interpret the relevant signals, and a vision for the future. However, it does require board members to understand what is going on in the market and the changes to their business. Being typically middle-older aged, many board members are ill equipped to do that, given their background, training, technical skills, etc…
Impact of big data on decision making – organisational responsibility and control
Finally, we looked at how boards re-configure their relationship with stakeholders in the organisation and beyond, specifically around issues of responsibility and degree of organisational control. Here, we identified two key challenges, as described below.
The need to break-down functional silos – Given the cognitive and structural changes mentioned above, the directors see the need to create new senior roles that bridge functions and bridge the gap between technical knowledge and decision making – examples mentioned include roles such as Chief Science Officer, Chief Data Science Officer or Chief Analytical Officer. This development requires the recruitment of new employees with suitable technical skills, typically located within a big data team and supporting board decision-making. However, the new roles may still struggle to develop joint messaging.
The outsourcing of responsibilities – In some cases, the lack of technical knowledge at the strategic level leads to the outsourcing of certain responsibilities to consultants and other external stakeholders. However, this, too, has risks, such as board members getting information second-hand, or the organisation giving away proprietary, commercially sensitive information to third-parties.
In summary, our study identified a shortfall in capabilities for dealing with the challenges of big data. At the individual level, there is a gap in the cognitive capabilities that organisations possess to cope with big data, capabilities that are crucial in avoiding the cognitive biases and overloads to which big data can contribute. At the board level, cohesion is being disrupted by big data, with directors struggling to introduce changes within the shortened timescales created by an ever-changing backdrop of new data, and feeling under pressure to adapt the way strategic decisions are made. Finally, at the organisational level, boards are seeking new ways of working that cut across traditional ‘silos’, such as through the introduction of sub-boards or relying upon the capabilities of third parties to help handle big data.
Let me know what you think of our paper.
Ana Canhoto 
Interesting tthoughts
LikeLike