If you are a product leader then you must be wondering what are the various steps to deploy a machine learning model. This article is a quick handbook to guide you through the steps needed to achieve the goal. The diagram above will help you reference each of the stages visually.
Business Objective: It all starts here, understanding the problem space that you are trying to solve. I can’t emphasize the importance of clearly outlining the business objective, the business metric that you are trying to solve. Discuss various hypotheses that led to the problem statement and state Goals and Non-Goals.
At this point, we are not even thinking about machine learning, we are purely thinking about whether there is a problem or not and whether it is worth solving or not.
Before getting into the next phase something worth iterating. Once you discuss the objective with your team, they will be more than excited to solve the problem using machine learning but a golden rule is:
Machine learning models are not the end in itself, they are means to an end. It is one of the many tools in a toolkit to solve a problem. It is important to explore rule-based system aka a heuristic model or a traditional program before jumping to machine learning models.
The advantages of a rule-based learning model are:
1. Rules are finite and give a good first start to solving the problem.
2. You don’t have enough data to train an efficient model. Machine learning models are typically hungry for data. Many a times startups don’t have enough data.
3. ROI of machine learning on other problems is much higher and you have a limited data science team bandwidth
4. Explainability of ML models: While there are many models that are easy to understand, the problem category where such models could be used are limited.
5. Time to market: Many a times a rule-based system would be much faster to get to market. You can use machine learning models later to make the system more efficient. This may also give you a first-mover advantage, which may or may not be a sustainable competitive edge depending upon your business model.
6. Helps evaluate the business objective before investing on a machine learning model. With speed of heuristic model also comes opportunity to evaluate the hypothesis, verify the metric, and ROI of the solution.
Collect Data: Now that we have established the business objective and also evaluated the ROI of investing on a machine learning model, the next step, which will also be one of the most critical steps, is to collect data.
Eventually machine learning model is all about data. The more and richer the data you have, the better will be the machine learning model. This has also led to data-as-a-service startups in the recent times.
Clean Data: Once we collect data, the data needs to be cleaned up.
Data cleaning typically involves — Handling missing data either by dropping the row/column or by imputing the values, Standardizing the data to have a mean of zero and a standard deviation of 1, normalizing the data to scale to values between 0 and 1, handling outliers, De-duplicating data and removing redundancies
Feature Extraction: Suppose I asked you to predict the price of a house on market. What factors you would consider. There are millions of factors we could consider but some are more important than others such as Location, #Bed Rooms, Sq Ft of the house, Walk score, Crime score, median salary of the populace etc. Identifying the relevant factors / features is what is called feature extraction.
This is where domain expertise comes into play. Using an off-the shelf model to solve a business problem is not complex with scores of libraries providing these functionalities. The real challenge is developing a core expertise in the field because real competitive advantage does not come from model but from domain expertise.
This requires a close collaboration of product team & data science team with marketing, customer success team, design, UX, Sales, and finance team. It requires an ensemble of cross functional experts to come together.
Choose various models that you will evaluate: There are many off-the shelf or customized models that one can use for a business problem. To identify a set of models that helps achieve the outlined business objective and has the power to explain(or not as is needed) is the goal of this step. This would need understanding of advantages and limitations of various models. How these models have been used to solve similar problems, what are various tuning parameters associated with the model all go into the decision of choosing a set of models to try.
Train the model: We will now split the data into training, which will be used to train the model, and test data, which will be used to measure the validating of the model. There are many techniques or ways to split the data. How much to use for training and how much to use for test again depends upon the availability of data, how data hungry the model is, and many other factors. This is the arts part of the data science that is only learnt through experimentation — there is no golden rule.
We now train the various models and measure the cost function, which for a linear regression would be Mean Squared error or for a classification exercise could be accuracy error or F1 Score.
Test the model: Now that we have trained the model, we would need to measure the efficiency of the model on the test data. Why are we doing this? Going back to the house prices example, does it matter whether we could predict the house prices of already sold houses or not? No! We want to develop a model that can help us find prices of the house that are still in the market. Hence, we will see how well our model is doing on test data, a data that model has not seen yet. Our objective is to minimize the loss on the test data as much as possible.
This step would also typically involve tuning various parameters of the model. These parameters in the context of a decision tree classifier for example could be the maximum depth of the tree, number of leaf nodes etc.
Choose the winner: Now we have value of cost function for various models. We need to declare a winner. The winner will be deployed into production. But how do we choose a winner? We could use our cost function to choose a winner but then if it were so simple then data science will not be fun.
We choose winner based on many factors such as how easy it is to explain the model to business stakeholders, how much gain in accuracy we get by trading off the explainability, how interpretable the model is, how much time does it take to run the model, how much processor or memory hungry the model is, how easy it is to tweak the model in the future etc.
To be able to choose the right model that helps achieve the business objective again requires domain expertise and trade-off analysis.
Deploy and A/B test: Now that we have a model that we have identified, it is time to deploy. Deployment is a process in itself and at some point in time I will discuss the various steps involving deployment but for now you can assume that the model is in a production environment and starts taking the real data.
However, we will have to test the model against the current production environment. In order to do that, we will run a Randomized control trial (A/B test) or some other kind of hypothesis testing to measure the uplift.
Complete the test and announce: At some point in time you will have a winner — the one that achieved the business objective by creating the desired impact on a metric of interest. And hopefully it is the model that the team developed. Once this is done, we declare the test a success and communicate to the stakeholders.
I can’t stress enough communicating to all relevant stakeholders each major deployment that you do. This will help create a sense of cohesion in the organization and avoid any confusion.
Learn & Iterate: Once we have deployed the model, it is only the beginning of a fun journey that the team has undertaken. From here on, we will start observing how the model is performing with the real world, real time data. Did we miss some factor such as seasonality, does some other feature has become more important, is the business model changing, has the customer profile or demography changed? Any of these factors could necessitate a need to iterate on the model which will send us back to step 1 — define the objective.
These are the typical steps to deploying a machine learning model. It all starts with objective and ends at learning and iterating. An effective machine learning model requires collaboration among inter-disciplinary stakeholders and experts.
In the next article we will discuss how to create a culture that can be iterative and learning because innovation sits on bedrock of collaboration.
Read my other articles on Product Leadership, Product Growth, Pricing & Monetization strategy, and AI/ML here.
