EstimatorReport: Get insights from any scikit-learn estimator

This example shows how the skore.EstimatorReport class can be used to quickly get insights from any scikit-learn estimator.

Loading our dataset and defining our estimator

First, we load a dataset from skrub. Our goal is to predict whether an online shopping basket is fraudulent, so that the payment can be reviewed before money leaves the account.

Baskets and products live in two tables, so we aggregate product-level information (including the cash amount of the basket) into one feature matrix with pandas. Using a skrub DataOp to keep those joins inside the estimator (and replay them on unseen data) is shown in Tracking all the data processing.

from skrub.datasets import fetch_credit_fraud

dataset = fetch_credit_fraud(split="train")
baskets = dataset.baskets
products = dataset.products

basket_features = (
    products.groupby("basket_ID")
    .agg(
        basket_amount=("cash_price", "sum"),
        n_items=("cash_price", "count"),
        mean_item_price=("cash_price", "mean"),
        max_item_price=("cash_price", "max"),
        n_makes=("make", "nunique"),
        n_item_types=("item", "nunique"),
    )
    .reset_index()
)
top_product = (
    products.sort_values("cash_price", ascending=False)
    .groupby("basket_ID", as_index=False)
    .first()[["basket_ID", "make", "item"]]
    .rename(columns={"make": "top_make", "item": "top_item"})
)

df = (
    baskets.merge(basket_features, left_on="ID", right_on="basket_ID")
    .merge(top_product, on="basket_ID")
    .drop(columns=["ID", "basket_ID"])
)
y = df.pop("fraud_flag")

from skrub import TableReport

TableReport(df)

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

TableReport(y.to_frame())

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

Looking at the distributions of the target, we observe that this classification task is quite imbalanced. This means that we have to be careful when selecting a set of statistical metrics to evaluate the classification performance of our predictive model. In addition, we see that the class labels are specified by an integer 0 or 1.

For our application, the label of interest is 1 (fraudulent).

pos_label, neg_label = 1, 0

Let’s create a predictive model. Thankfully, skrub provides a convenient function (skrub.tabular_pipeline()) when it comes to getting strong baseline predictive models with a single line of code. As its feature engineering is generic, it does not provide some handcrafted and tailored feature engineering but still provides a good starting point.

So let’s create a classifier for our task.

from skrub import tabular_pipeline

estimator = tabular_pipeline("classifier")
estimator

Pipeline(steps=[('tablevectorizer',
                 TableVectorizer(low_cardinality=ToCategorical())),
                ('histgradientboostingclassifier',
                 HistGradientBoostingClassifier())])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Introducing the EstimatorReport

Let’s gather some insights from our predictive model. We can use skore.evaluate() for this: the function will perform a train-test split and create a EstimatorReport containing the model fitted on the training data, ready to investigate.

from skore import evaluate

# Reserve 20% of the data for the test set
report = evaluate(estimator, X=df, y=y, pos_label=pos_label, splitter=0.2)
report

Pipeline(steps=[('tablevectorizer',
                 TableVectorizer(low_cardinality=ToCategorical())),
                ('histgradientboostingclassifier',
                 HistGradientBoostingClassifier())])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

Once the report is created, we get some information regarding the available tools allowing us to get some insights on our model by calling the help() method.

report.help()

Be aware that we can access the help for each individual sub-accessor. For instance:

report.metrics.help()

Measuring model performance

Let’s have a first look at the statistical performance of our model. skore knows that we are doing classification, and can give us an array of classic ML metrics, all at once, with summarize():

import time

start = time.time()
metric_report = report.metrics.summarize().frame()
end = time.time()
metric_report

metric
accuracy        0.986856
precision       0.444444
recall          0.050314
roc_auc         0.846260
log_loss        0.057310
brier_score     0.012101
fit_time        3.224487
predict_time    0.353211
Name: HistGradientBoostingClassifier, dtype: float64

print(f"Time taken to compute the metrics: {end - start:.2f} seconds")

Time taken to compute the metrics: 0.00 seconds

Since the output is a pandas dataframe, we can also use the plotting interface of pandas.

ax = metric_report.plot.barh()
_ = ax.set_title("Metrics report")

An interesting feature of the skore.EstimatorReport is its caching mechanism. Indeed, when we have a large enough dataset, computing the predictions for a model can be expensive. To amortize this cost, the report will cache the predictions when it is first created; this way, calculations that need the model predictions can get them from the cache and save a lot of time. This is why the metrics computation above is so fast.

When the model is fitted or the predictions are computed, we additionally store the time the operation took:

report.metrics.timings()

{'fit_time': 3.2244872669999722, 'predict_time_train': 1.2858006820000014, 'predict_time_test': 0.353210593999961}

By default, the metrics are computed on the test set only, but we can also compute them on the train set:

report.metrics.log_loss(data_source="train")

0.04527233488970059

Defining custom metrics

skore can compute user-defined metrics as well. It accepts metrics in the form of scikit-learn scorers, i.e. functions taking estimator, X and y (and optional keyword arguments). Let’s take a look at an example.

def operational_decision_gain(y_true, y_pred, *, amount):
    """The monetary gain we obtain depending on our predictions.

    May be negative, in which case our predictions actually *cost* us money.
    """
    mask_true_positive = (y_true == pos_label) & (y_pred == pos_label)
    mask_true_negative = (y_true == neg_label) & (y_pred == neg_label)
    mask_false_positive = (y_true == neg_label) & (y_pred == pos_label)
    mask_false_negative = (y_true == pos_label) & (y_pred == neg_label)

    REVIEW_COST = -20
    REPUTATION_COST = -30
    MARGIN = 0.02

    # Fraud correctly flagged: we pay the review costs, but do not incur
    # more cost
    fraudulent_refuse = mask_true_positive.sum() * REVIEW_COST

    # Fraud missed: the payment goes through and we lose the full basket amount
    fraudulent_accept = -amount[mask_false_negative].sum()

    # Legitimate basket wrongly flagged: we pay the review costs, but we also annoy
    # the customer and risk losing them, so it is penalized compared to a correct
    # refusal
    legitimate_refuse = mask_false_positive.sum() * (REVIEW_COST + REPUTATION_COST)

    # Legitimate basket correctly accepted: we earn a margin on the sale
    legitimate_accept = (amount[mask_true_negative] * MARGIN).sum()

    return fraudulent_refuse + fraudulent_accept + legitimate_refuse + legitimate_accept

In our example use case, each classification decision has a different monetary gain. The function above models this by translating the confusion matrix into a gain (payoff) matrix that depends on the basket cash value. Let’s test adding this metric to our report.

from sklearn.metrics import make_scorer

amount = report.X_test["basket_amount"]

# We use `make_scorer` to convert the metric to the right format (a function
# that takes `estimator`, `X`, `y`)
report.metrics.add(metric=make_scorer(operational_decision_gain, amount=amount))

Our custom metric is now registered in the report, and will be shown in the summary. In fact, since the underlying metric function takes y_pred as input, skore can use the cached predictions again to speed up the computation.

# The metric name is derived from the function name unless it is explicitly given
report.metrics.summarize().frame()

metric
operational_decision_gain    73462.980000
accuracy                         0.986856
precision                        0.444444
recall                           0.050314
roc_auc                          0.846260
log_loss                         0.057310
brier_score                      0.012101
fit_time                         3.224487
predict_time                     0.353211
Name: HistGradientBoostingClassifier, dtype: float64

Effortless one-liner plotting

The skore.EstimatorReport class also implements a number of the most common data science plots. As for the metrics, we only provide the meaningful set of plots for the provided estimator.

report.metrics.help()

Let’s plot the ROC curve for our binary classification task.

display = report.metrics.roc()
display.plot()

<Figure size 600x750 with 1 Axes>

The plot functionality is built upon the scikit-learn Display objects. We return those Display objects (slightly modified to improve the UI) in case we want to tweak some of the plot properties. We can have a quick look at the available attributes and methods by calling the help method.

display.help()

fig = display.plot()
fig.axes[0].set_title("Example of a ROC curve")
fig

<Figure size 600x750 with 1 Axes>

Similarly to the metrics, the cache allows us to avoid recomputing the model predictions, which speeds up the display generation.

start = time.time()
display = report.metrics.roc()
_ = display.plot()
end = time.time()
print(f"Time taken to compute the ROC curve: {end - start:.2f} seconds")

Time taken to compute the ROC curve: 0.11 seconds

You can learn more about the cache system in the corresponding example: Cache mechanism.

Visualizing the confusion matrix

Another useful visualization for classification tasks is the confusion matrix, which shows the counts of correct and incorrect predictions for each class.

Let’s start with a basic confusion matrix:

cm_display = report.metrics.confusion_matrix()
cm_display.plot()

<Figure size 600x600 with 1 Axes>

In binary classification, a confusion matrix depends on the decision threshold used to convert predicted probabilities into class labels. By default, skore uses a threshold of 0.5, but confusion matrices are actually computed at every threshold internally.

To visualize the confusion matrix at a different threshold, use the threshold_value parameter. For example, a threshold of 0.1 will classify more samples as positive:

cm_display.plot(threshold_value=0.1)

<Figure size 600x600 with 1 Axes>

We can normalize the confusion matrix to get percentages instead of raw counts. Here we normalize by true labels (rows):

cm_display.plot(normalize="true")

<Figure size 600x600 with 1 Axes>

More plotting options are available via heatmap_kwargs, which are passed to seaborn’s heatmap. For example, we can customize the colormap and number format:

cm_display.set_style(heatmap_kwargs={"cmap": "Greens", "fmt": ".2e"})
cm_display.plot()

<Figure size 600x600 with 1 Axes>

Finally, the confusion matrix can also be exported as a pandas DataFrame for further analysis:

cm_display.frame()

	true_label	predicted_label	value
0	0	0	12080
1	0	1	10
2	1	0	151
3	1	1	8

See also

For using the EstimatorReport to inspect your models, see EstimatorReport: Inspecting your models with the feature importance.