{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n\n# Skore: getting started\n\nThis guide illustrates how to use skore through a complete\nmachine learning workflow for binary classification:\n\n#. Set up a proper experiment with training and test data\n#. Develop and evaluate multiple models using cross-validation\n#. Compare models to select the best one\n#. Validate the final model on held-out data\n#. Track and organize your machine learning results\n\nThroughout this guide, we will see how skore helps you:\n\n* Avoid common pitfalls with smart diagnostics\n* Quickly get rich insights into model performance\n* Organize and track your experiments\n\n## Storing reports in Skore Hub\n\nAt the end of this example, we send the reports in Skore Hub\n(https://skore.probabl.ai/) that is a platform for storing, sharing and exploring\nyour machine learning reports.\n\nTo run this example and push in your own Skore Hub workspace and project, you can run\nthis example with the following command:\n\n```bash\nWORKSPACE=<workspace> PROJECT=<project> python plot_getting_started.py\n```\nIn this gallery, we are going to push the different reports into a public\nworkspace.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Setting up our classification problem\n\nLet's start by loading the \"toxicity\" dataset, a classification\nproblem where we classify tweets as \"toxic\" or \"not toxic\".\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from skrub import TableReport, datasets\n\ntoxicity = datasets.fetch_toxicity()\nX, y = toxicity.X, toxicity.y\nTableReport(toxicity.toxicity)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We create a held-out test set to evaluate our final model once we are done\nexperimenting.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.model_selection import train_test_split\n\nX_experiment, X_holdout, y_experiment, y_holdout = train_test_split(\n    X, y, random_state=0\n)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Model development with cross-validation\n\nWe will investigate two different families of models using cross-validation.\n\n1. A :class:`~sklearn.linear_model.LogisticRegression` which is a linear model\n2. A :class:`~sklearn.ensemble.RandomForestClassifier` which is a more\n   powerful model.\n\nIn both cases, we rely on :func:`skrub.tabular_pipeline` to choose the proper\npreprocessing depending on the kind of model.\n\nCross-validation is necessary to get a more reliable estimate of model performance.\nskore makes it easy through :class:`skore.CrossValidationReport`.\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Model no. 1: logistic regression with preprocessing\n\nOur first model will be a linear model, with automatic preprocessing of the text feature.\nUnder the hood, skrub's :class:`~skrub.TableVectorizer` will adapt the\npreprocessing based on our choice to use a linear model.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.linear_model import LogisticRegression\nfrom skrub import tabular_pipeline\n\nlogistic_regression = tabular_pipeline(LogisticRegression())\nlogistic_regression"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We now evaluate our model with cross-validation, using :func:`~skore.evaluate`\nwith `splitter=5` to perform 5-fold cross-validation.\nThis returns a :class:`~skore.CrossValidationReport` object, which can be used to\naccess the performance metrics and other information about the model.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from skore import evaluate\n\nlogreg_cv_report = evaluate(\n    logistic_regression, X_experiment, y_experiment, pos_label=\"Toxic\", splitter=5\n)\nlogreg_cv_report"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "A report will quickly show important information regarding the performance of the\nmodel, the dataset used and the architecture of the model. This information is only\na quick overview and one can dig deeper into the report to get more information.\n\nIndeed, Skore reports allow to structure the statistical information\nwe look for when experimenting with predictive models. First, the\n:meth:`~skore.CrossValidationReport.help` method shows us all its available methods\nand attributes, with the knowledge that our model was trained for classification:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "logreg_cv_report.help()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "For example, we can examine the training data, which excludes the held-out data:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "logreg_cv_report.data.summarize()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Additionally we can run automatic checks on the model and get a summary of the findings:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "logreg_cv_report.checks.summarize()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "But we can also quickly get an overview of the performance of our model,\nusing :meth:`~skore.CrossValidationReport.metrics.summarize`:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "logreg_metrics = logreg_cv_report.metrics.summarize()\nlogreg_metrics.frame(favorability=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<div class=\"alert alert-info\"><h4>Note</h4><p>`favorability=True` adds a column showing whether higher or lower metric values\n    are better.</p></div>\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In addition to the summary of metrics, skore provides more advanced statistical\ninformation such as the precision-recall curve:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "precision_recall = logreg_cv_report.metrics.precision_recall()\nprecision_recall.help()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<div class=\"alert alert-info\"><h4>Note</h4><p>The output of :meth:`~skore.CrossValidationReport.metrics.precision_recall` is a\n    :class:`~skore.Display` object. This is a common pattern in skore which allows us\n    to access the information in several ways.</p></div>\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We can visualize the critical information as a plot, with only a few lines of code:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "_ = precision_recall.plot()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Or we can access the raw information as a dataframe if additional analysis is needed:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "precision_recall.frame()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "As another example, we can plot the confusion matrix with the same consistent API:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "confusion_matrix = logreg_cv_report.metrics.confusion_matrix()\n_ = confusion_matrix.plot()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Skore also provides utilities to inspect models. Since our model is a linear\nmodel, we can study the importance that it gives to each feature:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "coefficients = logreg_cv_report.inspection.coefficients()\ncoefficients.frame()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "_ = coefficients.plot(select_k=15)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Model no. 2: Random forest\n\nNow, we cross-validate a more powerful model using\n:class:`~sklearn.ensemble.RandomForestClassifier`. Again, we rely on\n:func:`~skrub.tabular_pipeline` to perform the appropriate preprocessing to use with\nthis model.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.ensemble import RandomForestClassifier\n\nrandom_forest = tabular_pipeline(RandomForestClassifier(random_state=0))\nrandom_forest"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "rf_cv_report = evaluate(\n    random_forest, X_experiment, y_experiment, pos_label=\"Toxic\", splitter=5\n)\nrf_cv_report"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "rf_cv_report.checks.summarize()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We will now compare this new model with the previous one.\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Comparing our models\n\nNow that we have our two models, we need to decide which one should go into\nproduction. We can compare them with the :func:`~skore.compare` function that returns a\n:class:`~skore.ComparisonReport`:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from skore import compare\n\ncomparison = compare(\n    {\n        \"logistic regression\": logreg_cv_report,\n        \"random forest\": rf_cv_report,\n    },\n)\ncomparison"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "This report follows the same API as :class:`~skore.CrossValidationReport`:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "comparison.help()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We have access to the same tools to perform statistical analysis and compare both\nmodels:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "comparison_metrics = comparison.metrics.summarize()\ncomparison_metrics.frame(favorability=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "_ = comparison.metrics.precision_recall().plot()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Based on the previous tables and plots, it seems that the\n:class:`~sklearn.ensemble.RandomForestClassifier` model has slightly worse\nperformance due to overfitting on this small dataset. We make the choice\nto deploy the linear model to make a comparison with the coefficients study shown\nearlier.\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Final model evaluation on held-out data\n\nNow that we have chosen to deploy the linear model, we will train it on the full\nexperiment set and evaluate it on our held-out data: training on more data should help\nperformance and we can also validate that our model generalizes well to new data. This\ncan be done in one step with :meth:`~skore.ComparisonReport.create_estimator_report`.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "final_report = comparison.create_estimator_report(\n    report_key=\"logistic regression\", X_test=X_holdout, y_test=y_holdout\n)\nfinal_report"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "This returns a :class:`~skore.EstimatorReport` which has a similar API to the other\nreport classes:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "final_metrics = final_report.metrics.summarize()\nfinal_metrics.frame()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "_ = final_report.metrics.confusion_matrix().plot()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We can easily combine the results of the previous cross-validation together with\nthe evaluation on the held-out dataset, since the two are accessible as tables. This\nway, we can check if our chosen model meets the expectations we set during the\nexperiment phase.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "final_frame = final_metrics.frame().to_frame()\ncv_frame = logreg_cv_report.metrics.summarize().frame()\nfinal_frame.merge(cv_frame, on=\"metric\", how=\"outer\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "As expected, our final model gets better performance, likely thanks to the\nlarger training set.\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Our final sanity check is to compare the features considered most impactful\nbetween our final model and the cross-validation:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "final_coefficients = final_report.inspection.coefficients()\ncv_coefficients = logreg_cv_report.inspection.coefficients()\n\nfeatures_final_coefficients = final_coefficients.frame(select_k=15)[\"feature\"]\nfeatures_cv_coefficients = cv_coefficients.frame(select_k=15)[\"feature\"]\n\nprint(\n    f\"Most important features available in both models: \"\n    f\"{set(features_final_coefficients).intersection(set(features_cv_coefficients))}\"\n)\n\nprint(\n    f\"Most important features available in final model but not in cross-validation: \"\n    f\"{set(features_final_coefficients).difference(set(features_cv_coefficients))}\"\n)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We can further check if there is a drastic difference in the ordering by plotting\nthose features with the largest absolute coefficients.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "final_coefficients.plot(select_k=15, sorting_order=\"descending\")\n_ = cv_coefficients.plot(select_k=15, sorting_order=\"descending\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "They seem very similar, so we are done!\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Tracking our work with a skore Project\n\nNow that we have completed our modeling workflow, we should store our models in a\nsafe place for future work. Indeed, if this research notebook were modified,\nwe would no longer be able to relate the current production model to the code that\ngenerated it.\n\nWe can use a :class:`skore.Project` to keep track of our experiments.\nThis makes it easy to organize, retrieve, and compare models over time.\n\nUsually this would be done as you go along the model development, but\nin the interest of simplicity we kept this until the end.\n\nWe are using Skore Hub (https://skore.probabl.ai/) to store and review our reports.\n\n<div class=\"alert alert-info\"><h4>Note</h4><p>Here, we are using Skore Hub to store and analyze the reports that we computed.\n   Note that you can store reports as well locally using `mode=\"local\"` when creating\n   or loading projects via `skore.Project`.</p></div>\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from skore import login\n\nlogin()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We load or create a hub project:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "project = Project(name=PROJECT, mode=\"hub\", workspace=WORKSPACE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We store our reports with descriptive keys:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "project.put(\"logreg_cv\", logreg_cv_report)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "project.put(\"rf_cv\", rf_cv_report)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "In this example, we created a read-only Skore Hub project that you can visit by\nclicking on the link above and explore the reports.\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now we can retrieve a summary of our stored reports:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "summary = project.summarize()\nsummary"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<div class=\"alert alert-info\"><h4>Note</h4><p>:meth:`~skore.Project.summarize` returns a :class:`~skore.Summary` object. In a\n    Jupyter environment it renders as an interactive table where you can filter rows\n    and pick reports across the different views; the selection produces a query string\n    ready to pass to :meth:`~skore.Summary.query` so you can recover exactly those\n    reports.</p></div>\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Once you filtered the summary (e.g. to keep only the cross-validation reports), if\nyou now call :meth:`~skore.Summary.compare`, you get only the\n:class:`~skore.CrossValidationReport` objects, which\nyou can directly put in the form of a :class:`~skore.ComparisonReport`:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "new_report = summary.query('report_type == \"cross-validation\"').compare(\n    return_as=\"report\"\n)\nnew_report"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        ".. admonition:: Stay tuned!\n\n  This is only the beginning for skore. We welcome your feedback and ideas\n  to make it the best tool for end-to-end data science.\n\n  Key benefits of using skore in your ML workflow:\n\n  * Standardized evaluation and comparison of models\n  * Rich visualizations and diagnostics\n  * Organized experiment tracking\n  * Seamless integration with scikit-learn\n\n  Feel free to join our community on [Discord](https://discord.probabl.ai)\n  or [create an issue](https://github.com/probabl-ai/skore/issues).\n\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.14.7"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}