Celonis Product Documentation

How will you know if your system has succeeded or failed?

Your success and failure metrics should be phrased independently of your evaluation metrics such as precision, recall, or AUC. Instead, specify the anticipated outcomes. Set your success metrics before you begin, to prevent sunk costs from motivating you to launch a mediocre model.

A measurable metric provides enough information for successful real-world evaluation. For example, a system monitoring the health of an orchard might want to reduce the fraction of sick trees that die. But if you can't measure how many trees are sick, this is not a useful metric.

Ask the following questions:

Ideally, you want to fail fast. Watch for too little signal in your data, or data that isn’t predictive, to determine if your hypothesis might be wrong. Failing fast will enable you to revise your hypothesis earlier in the process and prevent lost time.

Consider engineering and maintenance costs over the long-term.

Watch for failures that are not related to your success metric. For instance, a video suggestion system that always recommends "clickbait" videos is not successful at providing a high-quality viewing experience.

Your model can make predictions at either of two points:

How will your product use the predictions? Remember, you want to use ML to make decisions, not just predictions. How will you turn your model's predicted outcome into a decision?

Think about how the implementation would look in pseudocode. For example, say you are building a model to predict the likelihood a user will click on a notification sent by your app, and you want to only send notifications to users likely to click them. Your pseudocode could look like the following:

click_probability = call_model(user)
if click_probability > 0.05:
    send_notification(user)

The preceding step helps turn your model's predictions into decisions.

Next, figure out where in your architecture this code would live. Doing so will help you address the following important questions:

The requirements discovered by answering the preceding questions can affect the features used in your model. You can only train on features that you would also have access to when you call the model; otherwise, there's no point in training on those features since you can't use your learned knowledge at serving time. Additionally, make sure you can get each feature within your latency requirements. Using data from remote services may make certain features expensive from a latency standpoint. For example, if you wanted to use the current weather as a feature for your camera app, you might find that the latency from looking up the current weather is too high and causes unacceptable slowdown in your app.

Lastly, be wary of using out-of-date data. Note that training data is sometimes several days old. When running on live traffic, you might encounter cases where data you'd like to use isn't yet available. Perhaps your database only updates a user's history every 30 minutes, so you don't have access to their most recent data when making your prediction.