Key ingredients to being data driven
March 2019
Transformation towards a data-driven company
We live in the world, in which we describe everything using enormous volumes of data. It’s generated on an increasing amount of levels and ability to analyse it on the fly is crucial. All this so that companies and organisations can make better decisions based on real information.
As business owners or decision-makers, our news feeds bombard us with articles or presentations showing us the power of data analysis and the use of Artificial Intelligence in our operations and business processes. As you look closely, it might appear that these solutions, in the end, are exclusive to technology giants. However, the truth is that every organisation can benefit from them. The question remains: where to start?
New technologies often automate data processing and decisions based on that data. It requires good quality input to make sure the output will be of equally high quality (so-called “Garbage In, garbage out”). So, to begin the adventure of using data-driven technologies, let’s try to answer the question of what “good data” is. Many people confuse it with just a significant amount of information. Organisations decide to implement very large data warehouses, spending millions of dollars on systems that collect and process inputs from even hundreds of data sources. However, these tools often do not provide one fundamental thing — easy and shared access to collected information. Shared, that is, providing an equally interpretable understanding of the given situation.
Every organisation that begins the adventure with data analysis faces the problem of data silos. Often, managers believe that their teams make decisions based on facts, like business events from the past recorded in data warehouses. Moreover, there are whole lots of kinds of business facts happening in our systems — describing the productions of goods related to the business, logistics, financial data, HR, online business activities and many more.
Many times, we may face a situation where teams which have data, will feel masters and rulers of their collected information. They will protect access to their resources. For example, I’m sure you faced situations where you had to ask for permission to access specific reports (and unfortunately not for security reasons!). To give you another perspective, think how quickly you can receive information about revenues generated by a particular segment of your clients in a given, non-standard period (i.e., in the vicinity of a marketing campaign)? In most companies, you will not be able to get it immediately. Furthermore, it can be particularly dangerous in situations in which the organisation’s remuneration model supports unhealthy competition between teams. Then the disaster of an illusion of ability to make data-driven decisions is at your fingertips.
Our activities are often described by hundreds, if not thousands of business indicators. In the illustrated situation we give the data that creates these metrics into hands of closed-communication teams (or at least separated ones). We end up creating a culture in which everyone can see only their part of the business. Moreover, that means that one can make decisions based on a partial understanding of product knowledge or business processes.
To be able to successfully take actions based on the analysis of other activities and using Artificial Intelligence, the organisation should undergo a transformation towards the so-called Data-driven Organisation. What is essential at the early stage is creating a strong vision of the product, appropriate organisational culture and, if it was not done so far, to build the so-called domain knowledge. It will also be essential to involve the proper experts to ensure a smooth process.
However, I can’t emphasise enough, that the critical element is the culture. Microsoft’s CEO, Satya Nadella, said: “A data culture isn’t just about deploying technology alone, it’s about changing culture so that every organisation, every team and every individual is empowered to do great things because of the data at their fingertips” (Official Microsoft Blog, “A data culture for everyone”).
The key to success is the ability to respond to changes in the world around us quickly. Innovation is a vital aspect of competitiveness. Thanks to the analysis of data and use of machine learning, we can analyse existing situations, describe the complex business processes in full, and finally draw conclusions for the future. We open the possibility of creating a whole new spectrum of products and services, which are more extensive but yet more straightforward to use. We can experiment ahead of our competition. From our client’s point of view, we can create a solution that they will feel great with, and our support team will know about their needs before contacting us.
Often the IT department raises terrible associations. Think about it. You may see dark rooms filled with eccentric people communicating using some cipher. As IT, we usually work hard for such a picture. For us what matters is new shiny technologies and tools supported by use of a steadily growing number of more and more complex programming patterns.
In a Data-driven Organisation, the technology team can’t be a source of problems but must help the business solve those, sharing their expertise. Coming back to the roots of our profession is necessary. Listen to the problems of other teams so they can be understood in full and solved using the correct approach, not to deepen or ignore them. As IT department, we need to support each company member at every step.
Hence, the technical team must be open for open communication with the business and understanding its needs. The willingness to learn and strive to achieve an ambitious vision of the product is also essential. However, from a technical point of view, we must find a solution that will allow us to build even the most complex products in a light and agile way.
Many tools can help us here. An example that we use in practice is the migration of the product to an architecture based on business events (the so-called Event-Driven Architecture). It promotes an excellent understanding of processes, like systems or users activities in our software, as well as the creation of events in its relevant parts (modules) and their recording. As a result, they correspond to the actual company image of the product or the process. Furthermore, the cost of maintenance and development significantly decreases.
Exploring and describing business processes precisely in and around the product is a very complicated task. At the same time, it requires a light and an agile approach. Event Storming is the right tool to do the job. People from different departments and developers working on any business process meet to model it together. Thanks to this, they can describe existing modules (to find gaps and potential problems) or design new ones. During such sessions, the team of programmers also learns the business domain language, which they will use later in modeling the solution. For implementation, Domain-Driven Design comes in handy. It’s an approach where system components faithfully reflect the reality they represent. It also allows you to involve experts from your company who are not necessarily familiar with the design and implementation of IT systems but have strong business knowledge. As a result, we receive a product that creates real information about itself based on business events from the very beginning.
Such approach provides an excellent mean to describe the product or business processes in the organisation faithfully. Depending on the scale of operations and the number of observed activities, such events can come in massive amounts — we are talking about hundreds of millions a day in our case. On this level technologies for collecting, processing, and analysis of collected information, together with machine learning, will come to help. In our case, especially in connection with legal restrictions, we decided to invest in a data warehouse based on Big Data technology, using a dedicated Hadoop cluster. To perform operations on the data we collect, we use the following tools: Apache Spark, for more extensive batch processes, and Apache Flink, for analysis of the vast number of events in real time.
Cloud technologies can also be used to address these needs. The leading cloud players, like Amazon Web Services, Google Cloud Platform, and Microsoft Azure, offer comprehensive solutions in this area. They allow the creation of appropriate software services, collection and processing (also in real time) of events, their recording and both simple, and extensive tools to implement AI based features.
Many companies are at the beginning of the journey in which they will benefit from the data they collect. How to start? From the very beginning, the best is to focus our work on the culture of the organisation. That’s a critical moment — understand where you are and how you operate as the business. Furthermore, work with your teams on answers to such questions as: What drives my teams? How do they make their decisions? What do they need to make the right decisions based on the data?
The significant strength of the Data-driven Organisation is the change in who and how can access the data. Everyone becomes an independent analyst and can draw information about the activities and processes within the organisation. Having it in mind, think if you have a team that is open and ready for it? Will they be willing and able to make the right decisions? These are probably even more important and crucial questions.
The second step is to develop a common language. Making sure that the same things in the entire organisation mean precisely the same. Let us consider how many inaccuracies or conflicts could arise when we used many different ways of describing the same fact or situation, and thus various people thought about something in a completely different way.
It is also worth remembering that it is impossible to buy a quick recipe for transforming your organisation. There is no such thing as an all-around silver bullet, if only because of multitude and diversity of solutions, tools, and processes. In preparing the plan appropriate for your business, it will be helpful to carry out audits and engage professionals to help.
Your company can also change thanks to improving the culture of the organisation and the new possibilities of IT technologies. If you care about the actual understanding of business processes, cost reduction or becoming a pioneer in your field, use your data to help your organisation to achieve it. Make sure that it will be used in the right way by every person in the organisation. Possession and processing the right kinds of data will make the use of machine learning possible, and it will help to optimise processes bringing your business to a different level. Welcome to the 21st century, in the era of data, where the new gold is the opportunity to collect and use information.
You can learn more about Wojtek Ptak here.
Originally posted here
