The Data Analytics Lifecycle: Making the Most of the Data You Use Every Day

A business is a dynamic entity that is defined in terms of the data that it collects and makes use of. At its simplest, a business needs to pay attention to the day-to-day and capture relevant operational records so that data is true, complete, and compliant. You can unlock value from business intelligence trends through aggregation, statistics, and visualization, though this typically only allows your business to react to the world around you. If you want to see your business vision become a reality, then you need to work toward predictive modeling and the proactive benefits it affords while avoiding potential costs and risks associated with this transformation.

In this article, we trace the data analytics lifecycle from data management to business intelligence to predictive modeling, explaining how these key steps define what we call attention, memory, and vision for your business.

Data Management: What Does Your Business Pay Attention to?

What records, numbers, and figures do you pay attention to every day as part of your work? This is the data that determines the success of your business now and in the future.

Every business needs to manage data in order to perform day-to-day operations. This can be something as simple as keeping track of customers and purchases or something as complicated as assessing risk associated with long-term financial plans and relevant macroeconomic trends. In every case, businesses rely on this operational data as proof of past performance and the building blocks for looking ahead. It pays to pay attention to how this data is recorded and handled.

Since data is the record of what happened and the source of sound planning, it is important to maintain accurate and timely records. In other words, data management is the foundation for all types of data analytics. This means that it is well worth the time to choose a data management strategy that meets current operational needs and opens up possibilities for valuable analytics.

We can sum up a few basic requirements for data management as follows:

• One source of truth: each new record is generated or captured exactly once and stored in such a way that it cannot be changed, except of course if relevant law or company policy requires that the data be removed from records. A copy of this record may be used for other analytics, but the original record remains true and unchanged.

• Consistent content: each new record has the expected fields and format required for daily operations

• Compliant with the law and company policies: each new record is retained only as long as required for operational and analytical purposes that are also compliant with relevant laws and regulations

In short, good data management means keeping track of information so that data is true, complete, and compliant.

To get a better understanding of the benefits of good data management, let’s take a look at an illustrative example.

Example: Online Retailer

Consider an online clothing retailer that sells a variety of products through a website and app.

As part of day-to-day operations, they will track records like product listings, customer profiles, online shopping carts, completed orders, product returns, and payments. These values are recorded in some combination of e-commerce software, customer relationship management (CRM) software, and accounting software. In this case it is valuable to decide up front how and where to track each type of data.

For example, customer profiles may be tracked through the CRM and then linked to in the e-commerce and accounting systems. This means that CRM data is the one source of truth for customer identifiers. It also means that customer data can easily be scrubbed from all records by simply removing the customer data from the CRM and marking the customer identifier as inactive. Any downstream records, such as purchase history, can still link to the customer identifier without running the risk of maintaining divergent sets of customer records.

Now imagine that their customer base grows so large that their original CRM no longer meets the operational needs of the business. They decide to migrate their existing customers to a new CRM. There are several ways to perform this migration, each of which will have different effects on daily operations and the possibility of using customer data for analytics.

For example, they may assign the same customer identifier from the old CRM to the customer in the new CRM. In this case they need to reconcile the fact that the records in the different CRMs may contain different fields, so downstream systems will need to be aware of the subset of fields that are available for each customer record. If the online retailer is consistent in how they manage and migrate customer data, they will be able to rely on their new records and their downstream systems, even though the fields have changed. Said another way, the new records will still be consistent among themselves, even if they contain different fields from the old records.

This case is also particularly interesting because the retailer will most likely not continue to run the old CRM. They will eventually completely migrate and then turn off the old CRM. While this process does disrupt truth and completeness as far as the original records are concerned, it can be done in such a way that it is still true and complete from that point onward. The source of existing customer data is just the old CRM that is now gone. This is not significantly different from collecting customer data directly from the customer the first time; both processes are short-lived. This emphasizes the important point that truth and completeness of data are taken relative to the current state of the system as a whole.

No matter the size of the business, it is always worthwhile to practice good data management because this is a requirement for smooth daily operations. In the absence of true, complete, and compliant data, it is difficult or impossible for a business to perform the basic functions of a business, such as managing customer relationships, figuring revenues and profits, and budgeting even for the short-term future. Beyond daily operations, good data management opens up valuable possibilities for discovering trends, hunting for opportunities, and mitigating issues before they arise.

We now discuss how attention to data management can give a business a memory of what worked and what did not, thereby improving prospects for the future.

Business Intelligence: What Does Your Business Remember?

With a sound strategy in place for capturing operational data, it is possible to look beyond the day-to-day and start finding new opportunities. This involves looking at historical data in order to find patterns that extend beyond one day or one quarter in the life of the business.

A major goal of business intelligence is to provide a snapshot of the health of the business at any given time. This typically involves some mixture of the following:

• Aggregation: historical data is combined and sliced to provide basic insights into periods of time, cohorts of customers, and overall performance. Something as simple as quarterly revenues can count as business intelligence, but it can also be something more nuanced, such as finding which day of the week tends to see the most sales.

• Statistics: related to aggregation but different because it involves assuming something about the structure of the data and how different records are related. These are the means, medians, and standard deviations you can use to compare different aggregates to each other in a more nuanced way.

• Visualization: reports, graphs, and dashboards that track aggregate trends over time, ideally by updating automatically from well-managed data sources

By setting up these facets of business intelligence, it can be easy to tell at a glance how the business is doing. For example, a sudden drop off in sales volume relative to the past may signal an issue with a payment provider or website hosting. Rather than waiting for angry customers to reach out, you can catch these issues as they happen and start working on a remedy right away.

It is natural to ask at which point in its growth a business should start building out its business intelligence. As far as the raw data is concerned, you can technically perform business intelligence as soon as you have more than one record. This means that business intelligence is a viable strategy almost from the inception of a business.

For example, you could analyze performance of multiple leads before you have even made the first sale. What marketing materials lead to engagement? What sales copy gets a call back from a potential customer? What products drive the most traffic to the website? Business intelligence can help you answer these questions and determine how and where to spend those first valuable dollars.

In fact, the cost in money and time for simple business intelligence has fallen drastically in the last few years, opening up this strategy to almost any business on the market.

Of course, the benefits of business intelligence only increase as you gather more data. This offers a distinct competitive advantage for existing businesses over new competitors. Over time, the business that makes better use of data in identifying trends tends to be the business that wins out. In other words, the memory your business has of how things worked in the past only matters if you put that memory into practice.

We have seen how paying attention to individual data records can offer almost immediate benefits in terms of business intelligence. More generally, a business with a memory is one that can make better and better decisions as time passes. However, this is still largely passive and reactionary engagement. This works for many small businesses, but it can be a roadblock to growth for large enterprises.

Now we turn to how a business can use its memory to drive its vision in the future.

Predictive Modeling: What Vision Does Your Business Have?

So now you have a business that pays attention to its data management and remembers what worked and what did not through various forms of business intelligence. As this business grows, it is natural to look beyond what has already happened and start looking forward to how these trends can be used to change what will happen. In short, you can predict the future by answering the most important question of all: what if?

As far as the mechanics for predicting the future, there are many tools at your disposal. These can range from simple statistical modeling up to full-fledged artificial intelligence systems. The most important thing at this stage is to maintain focus on what will be most beneficial to your business.

For example, imagine your business wants to improve sales growth. You start by gathering all of the relevant data that you know has some relation to sales. Ideally you will have already identified some interesting trends from your business intelligence efforts. The goal now is to find the set of variables that affect sales and then to find the subset of those variables that you can control. A few potential variables include time of day when marketing material is sent, the type of industries to which your salespeople are reaching out, and how long to wait between follow-up messages. Once you have identified some trends related to these variables you control, then you can build predictive models that will forecast what would happen if you changed each variable. Then you can determine how to change the variables so that you drive the result you want. This could be something as simple as changing the time of day when messages are sent, or it may involve a more complex multi-part strategy. In this way, you move from diagnostic business intelligence to proactive predictive modeling.

Unlike basic business intelligence, the barrier to entry for predictive models is typically high. This is because of several reasons, including the cost of expertise, the cost of infrastructure and software, and the risk of working off of imperfect information. While recent efforts toward the democratization of predictive analytics have shown some promise, the current reality is that only a larger enterprise can comfortably afford the necessary costs and risks of these systems. Unless you can commit the required resources to these efforts, it is best to rely on your experienced staff more than on models and processes you do not fully understand.

Nonetheless, this does not mean that predictive modeling systems need to be prohibitively expensive, though there are subtle risks to be aware of. Typically the software and data processing costs fall into ranges that would be affordable for many SMEs. The hidden costs of these systems lie mainly in the risks associated with following their guidance. If you make decisions based upon an imperfect model, then it can place undue strain on your bottom line and threaten your business as a whole. For this reason, it is important to assess these risks and make an informed decision before proceeding with predictive modeling.

Once you have decided to commit adequate resources to predictive modeling, it is worthwhile to bear the following in mind:

• Keep it simple: A more complicated, more expensive, PhD-certified model will not always drive better business results. In fact, a simple model that is only correct some of the time can often be much more valuable than the most cutting edge research or the newest machine learning offering from AWS.

• When in doubt, try it out: experiments are how you will gather the missing data you need to build a truly valuable predictive model. A typical SME will be conservative when it comes to their business, and this means that their historical data tends only to sample a small number of the possible outcomes of the parts of the business they control.. For this reason, larger enterprises will want to run experiments to determine how changing one or more variables changes how their business performs. Of course, these experiments can be kept small and contained, though they always carry the risk of not performing as expected. For example, an experiment of what time of day to send out marketing messages may lead to some lost sales since certain times of day will not be ideal. This has costs upfront but can lead to significant benefits in the long run if the trends that can be identified turn out to be profitable. This is yet another reason that only a larger or more specialized enterprise should consider predictive modeling.

• Monitor your predictions: what worked yesterday may not work the same tomorrow. This is related to experimentation but also extends into how you build and update models over time. Before you begin, it is important to understand how you will assess how well a model is performing, and you should define exactly what you plan to do if a model underperforms.

It is one thing to have a vision and another thing entirely to see that vision become a reality. By bearing in mind these caveats associated with predictive modeling, your business can start to answer the most important what if questions and work toward a future you can forecast and control.

Conclusion

Using data well is largely a matter of paying attention to details, remembering what worked, and having the wisdom to make the most of that which you do control.

If you have questions about the material covered here, or you want to discuss how to make the most of your data at any point in the analytics lifecycle, feel free to get in touch at hello@decode-ds.com