Managing next best action model
Overview
After the model is created, you can navigate through these tabs:
- Results. Review the model's results and prediction quality
- Goal. Edit the configuration of the goal event
- Actions. Edit the actions that were selected for the model
- Configuration. Change the training timeframe and user inclusion (filters)
Good to know
For editing Goal, Action and Configuration sections refer to Creating the next best action model.
Results
The results tab will display your model's prediction results.
After creating the model, you must wait until the training is completed—depending on the database size, it may take up to 30-60 minutes.
Prediction quality
Prediction quality indicates if the model is ready to be used in targeting.
The quality of any predictive model created in Intempt varies depending on several factors, including (but not limited to) the following:
- Data items that comprise the dataset and how complete of a picture the combined attributes provide.
- The daily volume of users and events in the dataset. A larger volume means more data is available for the model to learn from.
- The training date range. Training a model for a longer period generally equates to more data available for the model to learn.
To understand whether a given model is high-quality for use in the real world, you can use Intempt's recommendations, which are generated depending on the model's AUC.
Good to know
AUC (or AUROC, short for Area Under the Receiver Operating Characteristics curve) is one of the most commonly used measures of performance for machine learning classifiers. It is represented as a percentage from 0-100%. The higher your pipeline’s AUC, the better its predictive power.
| Model quality | AUC | Explanation |
|---|---|---|
| Excellent | > 85% | Your scoring model performs exceptionally well regarding discrimination ability, effectively distinguishing between positive and negative cases, and has a high degree of predictive power. |
| Very Good | 75-85% | Your model performs well in terms of discrimination ability. It learns useful patterns from the data and effectively distinguishes between positive and negative cases. |
| Good | 65-75% | Your model has some discriminatory power but may not be performing at an optimal level. We recommend waiting until you gather more data or excluding some data items to improve the model performance. |
| Average | 55-65% | Your model has limited ability to distinguish between the positive and negative classes. It ranks a randomly chosen positive instance higher than a randomly chosen negative instance with a probability typically between 55% and 65% of the time. We recommend focusing on model improvement to enhance its predictive power by gathering more data or excluding some data items to improve the model performance. |
| Below Average | <55% | Your model has a very limited ability to distinguish between the positive and negative classes. It ranks a randomly chosen positive instance lower than a randomly chosen negative instance with a probability of less than 55% of the time. We recommend focusing on model improvement to enhance its predictive power by gathering more data or excluding some data items to improve the model performance. |
Action prediction results
The NBA model's "Action prediction results" section provides insights into how each action may impact the likelihood of users performing the goal event across the project.
1. Action Name
- Description: Lists specific actions evaluated by the model, such as "$5 off," "$10 off," or "no coupon." These include all actions that you added during the action configuration step (refer to Creating the next best action model).
2. Users
- Description: Shows the number of users for whom the respective action is predicted to be the most effective.
- Example: "35,000 users" next to "$5 off" means that for these many users, a $5 discount is predicted to be the most motivating factor to make a purchase.
3. % of Users
- Description: Represents the percentage of each action's distribution across the whole user database.
- Example: "50.00%" next to "$10 off" indicates that half the targeted users will likely respond best to this discount amount.
Interpreting the data
- Resource planning: Allocate resources based on the most effective offers for specific user segments.
- Customer segmentation: Segment your customers based on their predicted responsiveness to different offers, enabling more targeted and effective marketing campaigns.
Using the next best action model for targeting
After creating the model, it will appear in the targeting dropdowns that support using calculated attributes (e.g., segment editor, personalizations, journey triggers, etc.).
After selecting the attribute, you must assign a specific experience to influence the user's behavior. Refer to the section below for an example.
Example: using NBA in personalizations
The NBA model will dynamically output the attribute value based on the highest uplift, increasing the likelihood of the user performing the goal event.
To use NBA in personalization, you must create an experience with relevant targeting configured.
For example:
| Action (targeting) | Experience |
|---|---|
| Placed an order | $5 off | Show a widget with a $5 off coupon |
| Placed an order | $10 off | Show a widget with a $10 off coupon |
| Placed an order | no coupon | No content to display |
First, you create the experience and its content.
After saving the content, you must configure its targeting based on the NBA attribute relevant to the matching experience.
Once the personalization is configured, the user will see the experiences that will be optimized to increase their likelihood of performing the goal event (in this example, "Purchase.")
Updated 14 days ago