{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\"Open" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "0Mj9VgFnsx7J" }, "source": [ "# Binary classifier 5 or not 5 ?\n", "\n", "Obbiettivo: Implementare un algoritmo capace di classificare il numero 5 da un immagine. Se il numero 5 appare nell'immagine il rusultato sará vero altrimenti falso.\n", "\n", "Tematiche:\n", " * Accuracy, Precision, Recall, F1 Score\n", " * Roc curve\n", " * Comparison binary vs random forest classifier" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "IfTWXst7dovr" }, "source": [ "## Teoria\n", "Quando parliamo di classificazione binaria possiamo avere quattro possibili risultati.\n", " * TN True Negative\n", " * TP True Positive\n", " * FN False Negative\n", " * FP False Positive\n", "\n", "| Risultato Esatto | Risulato Predetto | |\n", "|------------|---------------------------|--|\n", "| | Negativo | Positivo |\n", "|Negativo | TN | FP |\n", "|Positivo | FN | TP |\n", "\n", "### Metriche\n", "\n", "* Accuracy: $\\frac{TP+TN}{TP+TN+FP+FN}$\n", "\n", "Proportion of correct classifications (true positves and negatives) from overall number of cases.\n", "\n", " Quante volte il nostro algoritmo ha predettto il risultato giusto? Sarebbe quante volte il classifcatore ha predetto vero ed era vero o a predetto falso ed era falso.\n", "\n", "* Recall or Sensitivity: $\\frac{TP}{TP+FN}$\n", "\n", "Proportion of correct positive classifications (true positives) from cases that are actually positive.\n", "\n", " Quante volte il classificatore ha predetto vero ed era vero diviso lo stesso più quante volte il classificatore ha predetto falso ma in realtà era vero. La recall mi da la percentuale di quanto il mio algorimo ha risposto vero. Quante volte l'algorimo ha predetto che non era 5 ma in realtà lo era (in percentuale)? Percentuale di volte che ho riconosciuto che era 5 rispetto a quante volte ho detto che non era 5 ma lo era.\n", "\n", "* Precision: $\\frac{TP}{TP+FP}$ \n", "\n", "Proportion of correct positive classifications (true positives) from cases that are predicted as positive.\n", "\n", " Percentuale che rappressenta la frazione tra quante valte l'algoritmo ha predetto vero ed era vero diviso lo stesso più quante volte l'algoritmo ha preddeto vero ma in realtà era falso. Nel nostro esempio quante volte l'algoritmo ha predetto 5 ma in realtà non lo era (in percentale)? Percentuale di volte che ho riconosciuto che era 5 rispetto a quante volte ho detto che era 5 ma non lo era.\n", "\n", "\n", "Esempio:\n", "\n", "| Risultato Esatto | Risulato Predetto | |\n", "|------------|---------------------------|--|\n", "| | Negativo | Positivo |\n", "|Negativo | TN=97750 | FP=150 |\n", "|Positivo | FN=330 | TP=1770 |\n", "\n", "\n", "\n", "* Accuracy = $\\frac{TP+TN}{TP+TN+FP+FN} = \\frac{1770+97750}{100000}=0.9952$\n", "\n", "* Recall = $\\frac{TP}{TP+FN} = \\frac{1770}{1770+330}=0.8428$\n", "\n", "* Precision = $\\frac{TP}{TP+FP} = \\frac{1770}{1770+150}=0.9219$\n", "\n", "È un modello buono?\n", "\n", "Recall: percetuale di sbaglio dell'algoritmo nel predirre 5. 1 - 0.84 = 0.26%\n", "Quanto l'algoritmo sbaglia nel dare come risulta che il numero è 5. \n", "Voglio essere sicuro che tutte le volte che un numero è 5 io (classifier) lo riconosca come 5.\n", "\n", "Precision:Quanto l'algoritmo sbaglia nel dare come risulta che il numero non è 5\n", "Voglio essere sicuro che tutte le volte che un numero no è 5 (classifier) io lo riconosca come NON 5.\n", "\n", "Quindi possiamo concludere che il modello che abbiamo sarà ottimo nel riconoscere che in numero non è 5 (Precision=92%) mentre invece non sarà granche nel dire che il numero è 5 (Recall=84%)\n", "\n", "Tumori:\n", "Recall 95%: se vedo il tumore X lo riconosco al 95%. Quindi ho una bassa probabilità di di dire che \"X\" non è un tumore quando il realtà lo è.\n", "Prcecision 60%: se vedo X ed non è un tumore ho una percentuale alta 40% di dire chè lo è. Cioè ho una probabilità alta di sbagliarmi sul fatto che un un X non è un tumore e per me lo è. (Diciamo che tendo un po a dire che tutto è tumore per non sbagliare) \n" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "x7j8e4He1YM5" }, "source": [ "## Import Libraries" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "8z2H2V_-s3ya" }, "outputs": [], "source": [ "from sklearn.datasets import fetch_openml\n", "import matplotlib as mpl\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "from sklearn.ensemble import RandomForestClassifier\n", "from sklearn.model_selection import cross_val_score\n", "from sklearn.model_selection import cross_val_predict\n", "from sklearn.metrics import precision_score,recall_score\n", "from sklearn.metrics import confusion_matrix\n", "from sklearn.metrics import f1_score\n", "from sklearn.metrics import precision_recall_curve\n", "from sklearn.metrics import roc_curve, roc_auc_score" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "O7yGdPf-1avi" }, "source": [ "## Load the Dataset (MNIST)" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 107 }, "colab_type": "code", "id": "HKsVUCGYv1TW", "outputId": "3ca79c8a-e68e-470c-efe7-b196beecfae8" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "dict_keys(['data', 'target', 'frame', 'feature_names', 'target_names', 'DESCR', 'details', 'categories', 'url'])\n", "X_train shape: (60000, 784)\n", "Y_train shape: (60000,)\n", "X_test shape: (10000, 784)\n", "Y_test shape: (10000,)\n" ] } ], "source": [ "mnist = fetch_openml('mnist_784', version=1)\n", "X = mnist = fetch_openml('mnist_784', version=1)\n", "X = mnist[\"data\"]\n", "Y = mnist[\"target\"]\n", "print(mnist.keys())\n", "Y = Y.astype(np.uint8) # convert string target to int\n", "X_train, X_test, Y_train, Y_test = X[:60000],X[60000:], Y[:60000], Y[60000:]\n", "# override\n", "Y_train = (Y_train == 5) # True for all 5s, False for all other digits.\n", "Y_test = (Y_test == 5)\n", "print(\"X_train shape: \", X_train.shape)\n", "print(\"Y_train shape: \", Y_train.shape)\n", "print(\"X_test shape: \", X_test.shape)\n", "print(\"Y_test shape: \", Y_test.shape)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "OEmJMYi01d9s" }, "source": [ "## Classifier choice (Random Forest Classifier) and Training\n", "\n", "Come funziona la cross validation?\n", "https://911-code.com/come-e-scikit-imparare-cross_val_predict-precisione-punteggio-calcolato.html" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 35 }, "colab_type": "code", "id": "O62wZtNPuTw2", "outputId": "9f0e320f-4b58-4ed5-db58-6c9576a369c6" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "accuracy train set split in 3 set: [0.98885 0.9863 0.98635]\n" ] } ], "source": [ "clf = RandomForestClassifier(random_state=42)\n", "clf.fit(X_train, Y_train)\n", "accuracy_train = cross_val_score(clf, X_train, Y_train, cv=3, scoring=\"accuracy\")\n", "print(\"accuracy train set split in 3 set: \", accuracy_train)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "GCk-z8jY1y7k" }, "source": [ "### Perfomance Metrics Analysis on Train Set" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "3gpd_NjnvUk2" }, "outputs": [], "source": [ "Y_train_pred = cross_val_predict(clf, X_train,Y_train, cv=3)\n", "conf_matrix = confusion_matrix(Y_train, Y_train_pred)" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 161 }, "colab_type": "code", "id": "WFoaeCLyuzT2", "outputId": "a55a8431-ad8f-4f6a-da53-23c250ee513d" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Confusion matrix Train Set: \n", "[[54362 217]\n", " [ 309 5112]]\n", "Accuracy 0.99\n", "Precision 0.99\n", "Recall 0.87\n", "F1 Score 0.92\n", "Default Threshold chosen: 0.51\n" ] } ], "source": [ "print(\"Confusion matrix Train Set: \")\n", "print(conf_matrix)\n", "res_precision_train_score = precision_score(Y_train, Y_train_pred) \n", "res_recall_train_score = recall_score(Y_train, Y_train_pred) \n", "res_f1_train_score = f1_score(Y_train, Y_train_pred)\n", "print(\"Accuracy \", np.round(np.mean(accuracy_train),2) )\n", "print(\"Precision \", np.round(res_precision_train_score,2) )\n", "print(\"Recall \", np.round(res_recall_train_score,2) )\n", "print(\"F1 Score \", np.round(res_f1_train_score,2) )" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "NqUhgDS_rBDJ" }, "source": [ "* Recall: Quanto l'algoritmo sbaglia nel dare come risultato che il numero è 5.\n", "* Precision: Quanto l'algoritmo sbaglia nel dare come risultato che il numero non è 5.\n", "\n", "Esiste un modo di allenare il classificatore in modo da avere una recall più alta del 93%? Cioè voglio essere sicuro che l'algoritmo quando veda 5 lo riconsca sempre piuttosto che quando veda NON 5 dica è non 5." ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "MXMlaDl7iFdX" }, "source": [ "## Test Classifier" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 283 }, "colab_type": "code", "id": "wtyD4cNBiK3L", "outputId": "a7e5774a-25cc-46c0-b372-7e3862361b8f" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Prediction (Not-5, Yes-5): [[0.98 0.02]]\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAANh0lEQVR4nO3df6hc9ZnH8c/H1P4RI2I2l8vFatIV\nQUXctAy60tBkKVti8EeqYCoiWQikiEILBVfcYEVE4m/2D6mkqzS71t81ehFpqlKQIlTHXzEmrGZD\npIZrchPFWhCv2mf/uCflqnfO3Mw5M2eS5/2Cy8ycZ845D0M+OTPnO2e+jggBOPod03QDAAaDsANJ\nEHYgCcIOJEHYgSS+McidLVq0KJYsWTLIXQKp7NmzRwcOHPBstUpht71S0n9KmifpvyJiY9nzlyxZ\nona7XWWXAEq0Wq2OtZ7fxtueJ+keSedLOlPS5bbP7HV7APqrymf2cyTtiojdETEl6WFJF9fTFoC6\nVQn7SZL+POPxe8WyL7G93nbbdntycrLC7gBU0fez8RGxKSJaEdEaGRnp9+4AdFAl7HslnTzj8beK\nZQCGUJWwvyzpNNvftv1NST+WNF5PWwDq1vPQW0R8bvsaSVs1PfR2f0S8VVtnAGpVaZw9Ip6R9ExN\nvQDoI74uCyRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEgP9KWn0\n5o477iitf/LJJx1r27ZtK1338ccf76mnQ6666qrS+nnnndexduWVV1baNw4PR3YgCcIOJEHYgSQI\nO5AEYQeSIOxAEoQdSIJx9iGwZs2a0vpjjz3Wt33bs87uO2f33ntvaf25557rWFu+fHnpuqecckpP\nPWF2HNmBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnG2QegyXH0008/vbS+cuXK0vru3btL6+Pj46X1\nXbt2daw98MADpetef/31pXUcnkpht71H0seSvpD0eUS06mgKQP3qOLL/S0QcqGE7APqIz+xAElXD\nHpJ+b/sV2+tne4Lt9bbbttuTk5MVdwegV1XDviwivivpfElX2/7+V58QEZsiohURrZGRkYq7A9Cr\nSmGPiL3F7X5JWySdU0dTAOrXc9htH2f7+EP3Jf1Q0va6GgNQrypn40clbSmuh/6GpAcj4ne1dHWE\nabfbpfUtW7ZU2v5ZZ51VWi8b6160aFHpugsWLCitT01NldbPPffc0vobb7zRsXbw4MHSdVGvnsMe\nEbsl/VONvQDoI4begCQIO5AEYQeSIOxAEoQdSIJLXGswMTFRWo+I0nq3obWtW7eW1sfGxkrrVXSb\nLnrnzp09b/uCCy7oeV0cPo7sQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AE4+w1uPDCC0vrZT+nLEnH\nH398aX3hwoWH3VNdHnnkkdJ6t0tgMTw4sgNJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoyzD8DixYub\nbqGj22+/vbT+9ttvV9p+2U9Nd/sZatSLIzuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJME4+1Hu6aef\nLq3fcMMNpfVPP/20tD46Olpa37hxY8fa/PnzS9dFvboe2W3fb3u/7e0zli20/aztd4rbE/vbJoCq\n5vI2/teSVn5l2XWSno+I0yQ9XzwGMMS6hj0iXpD0wVcWXyxpc3F/s6TVNfcFoGa9nqAbjYhDE5y9\nL6njBzfb6223bbcnJyd73B2AqiqfjY/pWQs7zlwYEZsiohURrZGRkaq7A9CjXsO+z/aYJBW3++tr\nCUA/9Br2cUlri/trJT1VTzsA+qXrOLvthyStkLTI9nuSfiFpo6RHba+T9K6ky/rZJHrXbrdL693G\n0btZs2ZNaX358uWVto/6dA17RFzeofSDmnsB0Ed8XRZIgrADSRB2IAnCDiRB2IEkuMT1KLB6dedL\nE7Zu3Vpp22vXri2t33zzzZW2j8HhyA4kQdiBJAg7kARhB5Ig7EAShB1IgrADSTDOfgSYmJgorb/4\n4osda90uYe3260EbNmworS9YsKC0juHBkR1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkmCc/QhwySWX\nlNYPHDjQ87avuOKK0vqpp57a87YxXDiyA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EASjLMPgfHx8dL6\na6+91vO2V6xYUVq/6aabet42jixdj+y277e93/b2GctutL3X9uvF36r+tgmgqrm8jf+1pJWzLL87\nIpYWf8/U2xaAunUNe0S8IOmDAfQCoI+qnKC7xva24m3+iZ2eZHu97bbt9uTkZIXdAaii17D/UtKp\nkpZKmpB0Z6cnRsSmiGhFRKvbjxsC6J+ewh4R+yLii4j4m6RfSTqn3rYA1K2nsNsem/HwR5K2d3ou\ngOHQdZzd9kOSVkhaZPs9Sb+QtML2UkkhaY+kn/SxxyPewYMHS+u33HJLaX1qaqrnfS9durS0zu++\n59E17BFx+SyL7+tDLwD6iK/LAkkQdiAJwg4kQdiBJAg7kASXuA7AnXd2/IKhJOmll16qtP3Vq1d3\nrHEJKw7hyA4kQdiBJAg7kARhB5Ig7EAShB1IgrADSTDOPgB33XVXX7d/zz33dKxxCSsO4cgOJEHY\ngSQIO5AEYQeSIOxAEoQdSIKwA0kwzn4UKPup6mOPPXaAnXzdCSec0LHWrbfPPvustP7RRx/11JMk\nffjhh6X1u+++u+dtz8W8efM61m699dbSdefPn9/TPjmyA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAS\njLMfBc4+++ymW+josssu61gbGxsrXXffvn2l9Ycffrinnobd6OhoaX3Dhg09bbfrkd32ybb/YHuH\n7bds/7RYvtD2s7bfKW5P7KkDAAMxl7fxn0v6eUScKemfJV1t+0xJ10l6PiJOk/R88RjAkOoa9oiY\niIhXi/sfS9op6SRJF0vaXDxts6TOcxABaNxhnaCzvUTSdyT9SdJoREwUpfclzfpBw/Z6223b7cnJ\nyQqtAqhizmG3vUDSbyX9LCL+MrMWESEpZlsvIjZFRCsiWiMjI5WaBdC7OYXd9rGaDvpvIuKJYvE+\n22NFfUzS/v60CKAOXYfebFvSfZJ2RsTM30Qel7RW0sbi9qm+dHgUWLVqVWn9ySefHFAng/foo482\ntu+yS2iPOabaV0wuuuii0nqr1ep528uWLet53TJzGWf/nqQrJb1p+/Vi2fWaDvmjttdJeldS5wFV\nAI3rGvaI+KMkdyj/oN52APQLX5cFkiDsQBKEHUiCsANJEHYgCS5xHYAnnniitH7bbbeV1qempups\n50t27NhRWu/nZaTr1q0rrS9evLjS9i+99NKOtTPOOKPSto9EHNmBJAg7kARhB5Ig7EAShB1IgrAD\nSRB2IAnG2YfAtdde23QLHT344INNt4CacGQHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeS\nIOxAEoQdSIKwA0kQdiAJwg4kQdiBJLqG3fbJtv9ge4ftt2z/tFh+o+29tl8v/sonIQfQqLn8eMXn\nkn4eEa/aPl7SK7afLWp3R8Qd/WsPQF3mMj/7hKSJ4v7HtndKOqnfjQGo12F9Zre9RNJ3JP2pWHSN\n7W2277d9Yod11ttu225PTk5WahZA7+YcdtsLJP1W0s8i4i+SfinpVElLNX3kv3O29SJiU0S0IqI1\nMjJSQ8sAejGnsNs+VtNB/01EPCFJEbEvIr6IiL9J+pWkc/rXJoCq5nI23pLuk7QzIu6asXxsxtN+\nJGl7/e0BqMtczsZ/T9KVkt60/Xqx7HpJl9teKikk7ZH0k750CKAWczkb/0dJnqX0TP3tAOgXvkEH\nJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IwhExuJ3Zk5Le\nnbFokaQDA2vg8Axrb8Pal0Rvvaqzt8URMevvvw007F/bud2OiFZjDZQY1t6GtS+J3no1qN54Gw8k\nQdiBJJoO+6aG919mWHsb1r4keuvVQHpr9DM7gMFp+sgOYEAIO5BEI2G3vdL2/9reZfu6JnroxPYe\n228W01C3G+7lftv7bW+fsWyh7Wdtv1PczjrHXkO9DcU03iXTjDf62jU9/fnAP7PbnifpbUn/Kuk9\nSS9Lujwidgy0kQ5s75HUiojGv4Bh+/uS/irpvyPirGLZbZI+iIiNxX+UJ0bEvw9JbzdK+mvT03gX\nsxWNzZxmXNJqSf+mBl+7kr4u0wBetyaO7OdI2hURuyNiStLDki5uoI+hFxEvSPrgK4svlrS5uL9Z\n0/9YBq5Db0MhIiYi4tXi/seSDk0z3uhrV9LXQDQR9pMk/XnG4/c0XPO9h6Tf237F9vqmm5nFaERM\nFPfflzTaZDOz6DqN9yB9ZZrxoXntepn+vCpO0H3dsoj4rqTzJV1dvF0dSjH9GWyYxk7nNI33oMwy\nzfjfNfna9Tr9eVVNhH2vpJNnPP5WsWwoRMTe4na/pC0avqmo9x2aQbe43d9wP383TNN4zzbNuIbg\ntWty+vMmwv6ypNNsf9v2NyX9WNJ4A318je3jihMnsn2cpB9q+KaiHpe0tri/VtJTDfbyJcMyjXen\nacbV8GvX+PTnETHwP0mrNH1G/v8k/UcTPXTo6x8lvVH8vdV0b5Ie0vTbus80fW5jnaR/kPS8pHck\nPSdp4RD19j+S3pS0TdPBGmuot2Wafou+TdLrxd+qpl+7kr4G8rrxdVkgCU7QAUkQdiAJwg4kQdiB\nJAg7kARhB5Ig7EAS/w+xTgZkgsfn/QAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "tags": [] }, "output_type": "display_data" } ], "source": [ "test_img = X_test[766] # 5\n", "test_img = X_test[1] # 2\n", "Y_pred = clf.predict_proba([test_img])\n", "test_img = test_img.reshape(28, 28)\n", "plt.imshow(test_img, cmap = mpl.cm.binary,interpolation=\"nearest\")\n", "print(\"Prediction (Not-5, Yes-5): \", Y_pred)" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 35 }, "colab_type": "code", "id": "sSwSgRHk7v-j", "outputId": "9a8518b2-508c-4546-f313-98709b29df17" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Default Threshold chosen: 0.51\n" ] } ], "source": [ "# Calculation default threshold\n", "Y_proba = cross_val_predict(clf, X_train,Y_train, cv=3, method=\"predict_proba\")\n", "Y_scores = Y_proba[:, 1]\n", "precisions, recalls, thresholds_pr = precision_recall_curve(Y_train, Y_scores)\n", "threshold_chosen =thresholds_pr[np.argmax(precisions >= res_precision_train_score)]\n", "print(\"Default Threshold chosen: \", threshold_chosen )\n" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "XpJek8wqiMio" }, "source": [ "## Come definiamo la a threshold sulla probabilità per dire che un numero è o non è 5?" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "vs_zGaUYtKgq" }, "source": [ "### Recall - Precision threshold analysis" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 53 }, "colab_type": "code", "id": "5Ur9TnbVniYd", "outputId": "e1a0145f-cdf1-46d4-aa16-f508e468e84f" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Y_proba shape (60000, 2)\n", "Example Probabilities [Not-5, 5]: [0.11 0.89]\n" ] } ], "source": [ "Y_proba = cross_val_predict(clf, X_train,Y_train, cv=3, method=\"predict_proba\")\n", "print(\"Y_proba shape \",Y_proba.shape)\n", "print(\"Example Probabilities [Not-5, 5]: \",Y_proba[0])" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "FEV8oWeio_pB" }, "source": [ "Consideriamo solo la senconda colonna perchè ci interessa analizzare quando l'lagoritmo predice 5 se l'ha predetto correttamente oppure no." ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 53 }, "colab_type": "code", "id": "MnrdmvuZn7Jh", "outputId": "f52edc82-2c82-4359-b81c-74c087ed96ea" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Y_scores shape: (60000,)\n", "Example Score [5]: 0.89\n" ] } ], "source": [ "# Prendiamo solo la seconda colonna \n", "Y_scores = Y_proba[:, 1]\n", "print(\"Y_scores shape: \",Y_scores.shape)\n", "print(\"Example Score [5]: \",Y_scores[0])" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 279 }, "colab_type": "code", "id": "PN5CXfwzplcT", "outputId": "6b7363d7-b237-439e-df9e-a4a4deb09b9a" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3dd3hU1dbH8e/KpJGEJEAklFClhxYI\nTQSpCoggvdjAwrWAhXsV0Hu96MWGgKCgoi8CKtUKKgiCBKUXBekQkZKAAqElQIAk+/3jDBhKyAQy\nOZnM+jzPPM4pM+e3JzhrTttbjDEopZTyXj52B1BKKWUvLQRKKeXltBAopZSX00KglFJeTguBUkp5\nOV+7A+RURESEKV++/DXXOXXqFMHBwXkTKB/RdnsXb203eG/bb6Td69evP2KMuelqyzyuEJQvX551\n69Zdc524uDhatGiRN4HyEW23d/HWdoP3tv1G2i0ie7NapoeGlFLKy2khUEopL6eFQCmlvJwWAqWU\n8nJaCJRSysu5rRCIyEcickhENmexXETkbRGJF5HfRKSeu7IopZTKmjv3CKYA7a6xvD1Q2fkYALzn\nxixKKaWy4Lb7CIwxP4lI+Wus0hn42Fj9YK8SkXARKWmMOeiOPMv2LWPh7wtxiAOHjwNfH1/8Hf4X\nH5nnB/sFE+IfQoh/CCULlyQqNApfH4+75UIptzMGzp0DX19wOODkSUhKspaJQEYGpKVBuXIQEGAt\n++uvv5dfUKkS+PnB4cPW8rQ0SE//e73ata1tbNsGO3fCpk0RHD9uLROBjh3Bxwc2bYK9ey99f19f\nuOMO6/mCBbBli5X53DlrO8HBMGSItXz6dNi+/dI2Fi0KTz9tPZ88Gf7449LlJUrA449bzydOhMTE\nS5eXKwcPPWQ9f/ttOHLk0uWVK8N991nPR42yPsPMataEnj2v/Oxzk53fbqWB/ZmmE5zzrigEIjIA\na6+ByMhI4uLirvnGKSkpV6wzc/9MJu6eeF1BffAhIiCCML8wQnxDLj5CfUMJ9bMe4X7hhPuFE+oX\nSiFHIYIcQQQ6AnGI47q2eT2u1m5v4KntNgZSUx2cOuXA3z+D0NA00tKErVtDOXfOh9RUB6mpPpw7\n50OVKslUqnSKU6ccfP99CYwRzpwpzqef7ub8eR8aNUqiRo1kDh0KYOrU8qSmWq8/c8aBMdCnz34a\nNjzK778HM25cZTIyhPR0ISNDyMiARx/dTf36x9i8OZSxY6uQkcEl6wwdup1atU6wcmVR53Lh7Fkf\nTp92kJ7uw7vvrqd69WS++64Eo0ZVu6KtkyevoXz503z+eRQTJlS6YvmsWSspXvwsH39cjsmTK1yx\n/JtvlhESksb771dk1qyyQM1Lli9aFIfDAW+9VZm5c0tfsiwgIJ3vv/8ZgNGjq/HDDyUuWV6kyDka\nNVoBwLvv1mTFimKXLC9V6gx1664B4J136rBhQ/glyytVSqFGjfUAjB1bjx07Cl+yvFatE9x88wYA\nxoxpwL59QZcsb9jwKGXKbAJg5MjGHDkScMnyFi0OU7z4VsB9/9bFnQPTOPcIvjXG1LzKsm+B140x\ny5zTi4Ehxphr3jYcGxtrbuTO4gyTQXpGOmkZaZzPOM+59HOcTTtLukm/OP/0+dOknEvh5NmTJCYn\nsu/EPvae2EvS6SSOpx7nxNkTHDtzjKQzSaSmpWbdfoTwwHCKFipK0UJFCQsMIzwwnPCAcCJDIikZ\nUpKShUtSunBpokKjiAyJvKE9D73bMm8cOwa7d1uPhATrF1yNGtCjh7W8a1c4ehROn4YzZyA1FXr3\nhv/9D86ehYoVITkZUlKsYgAwbBi8+qr1umLFrtzmiBHwwgvWr92r9bDy9tswaBDs2AGtWlm/coOD\nISjI+kU8bBi0awebN8OTT1q/3i/8ivfxsZY3aQLr18PLL1+63NcXnnsOatWCtWvhvfesX9vBwRAS\nYv33gQcgKgri42HZMiuTMX+/vkMHCA+3fs1v2PB3uy+sd9dd1vts3Wo9fH2tXBe0b2/tMezfb+01\n/PLLOurXj734PjExVqa9e63lF+YbY81v0MCaPnnS2ksJCAB/fyufJ7nBO4vXG2Nir7bMzj2CRKBM\npuko5zy38hEffBw++Dn8KEShG36/0+dPk3Q6iUOnDnH49GGOnjlKyrkUUs6lcCL1BEfPHOVo6lGO\nnjnKidQTJJ5M5FjqMQ6dOkSGybgiW1hA2MWCERUaRYXwClQsUpHiwcUJ8Q+hsH9hwgPDKR5cnJuC\nb8Lf4X/DbfBWxli76Re+OKKjrfkTJ1pfRgcPwoED1n9jYuDzz63ldepYX0iZ3Xvv34Xgwq5/sWJQ\nqBAEBsLNN1vz/P2tL8XgYChcGEJDrUf9+tby0FBYtMj6orrwZR4QAEWKWMujoqz3dzhg+fKfadOm\nGf7+fx8GqVr1ykMTmdWsCT/+mPXy+vVhzpyslzdo8PeX6tVUqmQ9slKlivXISo0a1iMrZcpYj5Mn\nU4iJuXJ5uXLWIyuhoVkv82Z2FoK5wEARmQk0Ak646/yAOwX5BREUFkSZsDLZr5xJekY6h08f5mDy\nQRKTE0k8mUhiciLHzhzj+NnjHE89zv4T+1m6ZynJ55KzfJ/wwHBKhJSgREgJHKcdzDs/j6jQKMqE\nlqFS0UrcXPRmgvyCsny9NzhxwvpyvPAF85//wPz5sGvX38dj69SxfqmCdRx461YoWRJKlYJGjaBx\n47/f7+WXISzM+mVftqz15ZL5l+VPP2WdRQQ+/DDr5b6+0Lp11ssdjr/3GIKD0wkIyHpdpVzltkIg\nIjOAFkCEiCQA/wX8AIwx7wPzgA5APHAa6O+uLPmRw8dx8Qs8puRVfto4GWM4euYoSWeSSD6bTPK5\nZI6dOcbh04c5dOoQf6X8xZ+n/uRg8kF2nNzBz6t/5lz6uUveIyo0imoR1ahWrBrVIqpR46YaRBeP\npnhwcXc30+2MgWPH/Fi2DG691Zr30Ufw3XfWicldu6xf9UWL/n0S89Qp68u0cWPrRF2JEtaX/gVL\nl3LNL9h+/dzWHKVs4c6rhvpks9wAT7hr+wWFiFAsqBjFgq5y4PgycXFx3HbbbSSdSWLv8b3sOrqL\nXUm72Hl0JzuO7GDqxqmX7F1EBEVQqWglKhapSMXwitQpUYdGpRsRFRqFZL6kw2aHD1tXizRsaB1m\n+eIL+OAD2LfPepw+3RSA48etX+oJCdax6LAwaNsWqlWzHhkZ1nHnMWOuvT39la28jV4TWcCICBFB\nEUQERVC/VP1LlhljOJB8gC2Ht7Dl0Ba2Ht7K7uO7WbF/BbM2zyLdWNfrlQwpSUzJGKJviib6pmhq\nR9Ymuni0W89HnDplHRYJCIBffrEOn2zZYh2iufBLfssW6/BOcrJ1UrVGDesE6Pnz8XTqVInAQGu9\nF1+0Hkop12gh8CIiQunQ0pQOLc3tN99+ybKzaWfZ+NdGViesZnXian776zcW7V508TCTn48f0cWj\nqVuiLrWL16ZOiTrULVGXooWK5jjHqVOwfDmsWWM91q6FP/+EefOsq0MOHoQZM6yTt127Wl/41av/\nfRKwX79LD8/ExSXQosU1zlAqpa5JC4ECIMA3gIalG9KwdEMGMQiAtIw04o/Gs/HPjfz656/8+uev\nzN81nykbplx8XbWIatwSdQu3lLmFFuVbULFIxUsOKxljXdL444/WCdmmTeH33/++wad6det51ap/\nX23Svr11iWY+OjqlVIGmhUBlydfH1zrJHFGNXjV7XZz/V8pfbDq0iXUH1rF8/3K+3vE1H234CIAy\noWVoXq45kT7R/L66GivmVuPw9kqQ4ce//mUVgho1rEskY2Ot4/iX89GuEJXKU1oIVI5FhkQSGRJJ\nm4ptAOvcw7Yj21i4M44VB+KI2xNHYvI0CAR6ggM/KoZVYX9UTUYur0dsqVjqN61HWGD4NbejlMob\nWgjUdTPGOoH73XfC3Lk1iI+vwYEDj+NwwPc/JmOK7eAI29h6eCtbDm9hdeJqZm2ZdfH10TdF06xs\nM5qVa0btyNpUCK9AsL/3DUiulN20EKjr8tln8Oyzf3fwFRMDTzxhdacQHAztWhUGYp2PvyWdTmL9\nwfWsSVzD8v3LmbZpGu+vf//i8sjgSBpHNaZndE/uqnIXhQMu7bdFKZX7tBAolxw9at2odfvtVk+Q\nJUpY3RU8/7zVZUJUlGvvUyyoGLfffPvFq5bSMtLY9NcmdiTtYPex3cQfjWfB7wuYs2MOgb6BNC/X\nnAalGhBbKpbGUY0pEVIimy0opXJKC4G6pk2brA7Npk2zOlBzOKxC0KyZ9bhRvj6+xJSMueTu6gyT\nwYr9K5i9ZTY/7f2J15e9fvEeh9qRtbm94u20vbktjaMaExqgnccodaO0EKirMgbuvhvmzrU6Trv3\nXhg40CoC7uYjPtxa9lZuLWv1GXHm/Bk2/LmBpXuXsvD3hYxbPY5RK0fhIz7UKl6Lco5y7A3fS6Oo\nRlQpVgUf0cuOlMoJLQTqoowM63r/1q2ta/jr14dbboFHHrH66rFLIb9CNCnThCZlmjD01qGknEth\nxf4VFx+L9i5i7py5ABQJLMLtN99O+0rtaVepHZEhkfYFV8pDaCFQGGN10vbvf8PGjVana82b599u\nGkL8Qy45z7B4yWJKRJdgdeJqlu1bxvz4+RevTqoTWYfWFVrTpmIbbi17q558VuoqtBB4MWPg++9h\n+HCrq4eKFeHTT62bvjyJQxxEF48mung0D8Y8SIbJYOOfG5kfP5/FfyxmwtoJjFk1Boc4aFC6AS3K\nteC+OvdR46ZrdHyvlBfRQuDFUlOtPnsKFbI6eXvgAWsUKE/nIz4XT0A/3+x5zpw/w/L9y4nbE8eS\nPUsYtXIUbyx/gx7RPfh3s39TK7KW3ZGVspUWAi9jjHUPQNeuVgFYtMjq58e/AA90VsivEG0qtrl4\nJ/SR00cYu2osb69+m9lbZtOuUjseqfcId1W5Cz9HAaiESuWQXl7hRfbtg44doVcvmDnTmlerVsEu\nAlcTERTBiFYj2PP0Hl5q8RKb/tpEt9ndKPNWGQbNG8TcHXM5kXrC7phK5RktBF4gNRVeecUanGXJ\nEnjnHbjnHrtT2a9ooaK8eNuL7Hl6D9/0+YbGUY2Z9OskOs/sTLGRxWg1tRWzNs+6YsQ3pQoaPTTk\nBfr2ha++gm7dYPToaw/u7Y18fXzpWKUjHat05GzaWVYlrOKH3T8wfdN0en/Rm+LBxelXpx+9avYi\npkRMvhq9TancoHsEBdThw38PzD50KCxcCJ9/rkUgOwG+AdxW/jZGtBpB/JPxzL9nPo2jGjNm1Rjq\nf1Cfyu9U5rkfnmNB/AJOnTtld1ylcoUWggLoyy+tAV+ef96abtjQGrtX5YyP+NCuUjvm9J7Dn//8\nk/+76/+4uejNvLXqLdpNa0f4G+E0n9ycyb9O5vT503bHVeq6aSEoQNLShMGDrUNAFSvCY4/Znajg\nKBZUjIfqPcSCexdwbMgxFty7gH81+RdHTh/hwbkPUnpMaZ6a/xRrEtdgjLE7rlI5ooWggDhwAJ5+\nui5vvWX1CfTzz9aYvyr3Xbiz+bU2r7Hl8S0s7beU9pXa896692j0f42o+HZFnvvhObYd3mZ3VKVc\nooWggDh2DJKS/Jkxw7oqKCDA7kTeQURoXq4507tN59Czh5jSeQo1bqrBW6veosa7NWg5taVeeaTy\nPS0EHu7wYesmseho+OSTNfTubXci7xUeGM4DdR/gu77fkTg4kddbv87e43vp/UVvSo8pzbMLn2XH\nkR12x1TqCloIPNjWrVa30CNHWtO+vnpsOr8oHlycIbcOuXjlUfNyzRm7eizVJlSj4/SO/HLwF7sj\nKnWRFgIPtWULtGhhPb/rLlujqGu4cOXRFz2/YP8z+xnRcgQr9q+g/gf16Ta7Gxv/3Gh3RKW0EHii\nLVugZUvw9bW6jK6hnWh6hBIhJXih+Qv88dQf/Pe2//LD7z9Qd2Jd2nzchnm75pFhMuyOqLyUFgIP\nc+qUdU+Ary/ExUGVKnYnUjkVFhjG8BbD2fv0Xl5v/Trbj2znzul3Uvu92ny57Uu9/FTlOS0EHiY4\n2BpDWIuA5ytSqAhDbh3CH0/9waddPiUtI41us7sR+2EsX2//mrSMNLsjKi+hhcBDbN8O8+dbz7t3\n1yJQkPg5/Lin9j1sfnwzU++eyrEzx+gyqwtl3irDkB+GsDNpp90RVQGnhcAD7NwJrVrBP/5h9SSq\nCiZfH1/ur3M/OwbuYE7vOTQs3ZDRK0dTbXw1us/urlcaKbdxayEQkXYiskNE4kVk6FWWlxWRJSLy\nq4j8JiId3JnHEx09ap0TSEuz9ggCA+1OpNzNz+FHp6qdmNN7DgmDE3i+2fMs2r2I+h/U587pd+oe\ngsp1bisEIuIAJgDtgRpAHxG5/PqWfwOzjTExQG/gXXfl8UTGwMMPW91HfPutdhnhjUqElGBEqxHs\nfXovr7Z6leX7llP7vdr8b+n/OJehdyur3OHOPYKGQLwxZrcx5hwwE+h82ToGCHU+DwMOuDGPx5k3\nzxpH4PXXrR5ElfcKCwxjWLNhbHtiG3dXu5sX417k4XUPs3j3YrujqQLAnYWgNLA/03SCc15mw4F7\nRSQBmAcMcmMej9OhA3z9NTzzjN1JVH5RsnBJZnafyfx75pNu0mnzSRt6ftaThJMJdkdTHkzcdc2y\niHQH2hljHnZO3wc0MsYMzLTOYGeG0SLSBJgE1DTm0jtrRGQAMAAgMjKy/swLA+5mISUlhZCQkFxt\nT146c8bBsWN+lCqVszPDnt7u6+Wt7T568ijfHP2G6fun44MPfcr2oWdUTwIdBf9Ekrf+zW+k3S1b\ntlxvjIm96kJjjFseQBNgQabpYcCwy9bZApTJNL0bKH6t961fv77JzpIlS7JdJz+7/35jwsONSUrK\n2es8vd3Xy9vbvfvobtN1VlfDcEzUmCjz6cZPTVp6mr3h3Mzb/+bXA1hnsvhedeehobVAZRGpICL+\nWCeD5162zj6gNYCIVAcCgcNuzJTvTZkCH38MTz4JRYvanUZ5ggpFKvBFzy9Y2m8pxYOLc+9X9xL1\nVhRPzX+KlftX6p3KKltuKwTGmDRgILAA2IZ1ddAWEXlZRDo5V/sn8IiIbARmAP2MF/+r3boVnnjC\n6kzuxRftTqM8TfNyzVn7yFq+6PkFTaKaMHH9RG756BaaTW7Grwd/tTueysd83fnmxph5WCeBM897\nMdPzrUBTd2bwFKmp0LMnhITA9OngcNidSHkiH/Gha/WudK3elROpJ5i+aTr/jfsvsR/GMqDeAEa0\nGkGxoGJ2x1T5jN5ZnE+cPw/NmsHUqVCypN1pVEEQFhjGYw0eY+egnQxqOIgPf/mQKuOr8MH6D0jP\nSLc7nspHtBDkE4ULw3vvQbt2didRBU14YDhj241lw6MbqFW8Fv/49h80ntRYu6xQF2khsFlGhtWH\n0OrVdidRBV3N4jVZ8sASpnWdRsLJBJpMasKkXybZHUvlA1oIbDZ5MnzwgTXYjFLuJiL0rdWXzY9t\n5rZyt/HwNw/z6LePcjbtrN3RlI20ENjo8GF49llo3hz697c7jfImxYKKMf+e+QxtOpSJ6yfSeFJj\nlu5ZancsZRMtBDZ66SU4eRLefx9E7E6jvI3Dx8FrbV7j615fk3Q6iRZTW9B1Vlfij8bbHU3lMS0E\nNtmxwyoA//gHVK9udxrlzTpX68z2gdsZ0XIEC39fSK33avHe2vf0RjQvooXAJuXLw5tvwn//a3cS\npSDIL4gXmr/AzkE7ua3cbTw+73G6zOrCkdNH7I6m8oAWApsEBFi9ihYvbncSpf5WqnAp5t0zj7fu\neIv58fOp834dfvzjR7tjKTfTQpDHMjLg7rutcQaUyo98xIenGz/N6odXExoQSpuP2zB00VDOpetA\nOAWVFoI8Nm0azJkDp07ZnUSpa6tboi7rHlnHI/Ue4Y3lb9D0o6ZsP7Ld7ljKDbQQ5KHTp2HYMIiN\nhb597U6jVPaC/YOZeNdEvuj5BbuP7SZmYgzjVo0j49IhQ5SH00KQh0aPhsREGDMGfPSTVx6ka/Wu\nbH5sM60rtObpBU/T+uPWHEjWkWULCv06yiOHDsEbb0C3blbnckp5mpKFS/JNn2+Y1GkSaxPXUm9i\nPZbtW2Z3LJULtBDkkZtugk8+sYqBUp5KRHgw5sGLJ5JbTm3J+DXj9Z4DD6eFII+IQJcucPPNdidR\n6sZFF49m7SNraV+pPYPmD6LLrC78mfKn3bHUddJCkAcGDYJXXrE7hVK5KywwjK97f83o20fzffz3\nRL8bzYxNM3TvwANpIXCzAwesriSO6A2aqgDyER8GNxnMhkc3ULloZfp+2Zden/fi6JmjdkdTOaCF\nwM3eew/S0629AqUKqmoR1Vj24DJebfUqX23/itrv1dY7kj2IFgI3Sk219gY6dYKKFe1Oo5R7+fr4\nMqzZMFY+tJJg/2DafNyG15e9bncs5QItBG40fbp1SOipp+xOolTeiS0Vyy8DfqF3zd4MWzyMl+Je\n0vMG+Zyv3QEKspgY+Ne/oEULu5MolbeC/YP5pMsnBPgGMHzpcM5nnOd/Lf+H6MAb+ZIWAjeKibEe\nSnkjh4+DSZ0m4Su+vPLzK5xNO8vItiO1GORDWgjcZPx4aN1aB51R3s1HfJh410T8Hf6MWjmKY6nH\neL/j+/j66FdPfqLnCNxgxw7rKqGZM+1OopT9fMSH8R3G85/m/2HSr5Po+VlPUtNS7Y6lMtFC4AZj\nx1oDzzzxhN1JlMofRISXW77M2DvG8tX2r7j9k9v1TuR8RAtBLjtyBKZMgfvu09HHlLrcU42fYnrX\n6aw7sI56E+uxfN9yuyMpclAIRKSQiFR1Z5iC4L33rPsHnnnG7iRK5U99avVh1cOrCPILosXUFkxY\nM8HuSF7PpUIgIncBG4DvndN1RWSuO4N5qrQ06N4datSwO4lS+VftyNqsG7CO9pXaM3D+QJ774Tkd\n7MZGrp66Hw40BOIAjDEbRKSCmzJ5tJdeAr13RqnshQeG81Wvr3jq+6d4c8WbHEw5yKROk/B3+Nsd\nzeu4WgjOG2NOXHb9r37dXWbzZoiOtrqcVkplz+Hj4J3271CqcCle+PEFDp06xJc9vyTYP9juaF7F\n1XMEW0SkL+AQkcoi8g6wwo25PM6uXVCrltW3kFLKdSLC882eZ1KnSSzavYg7Pr2D46nH7Y7lVVwt\nBIOAaOAsMAM4CTyd3YtEpJ2I7BCReBEZmsU6PUVkq4hsEZHprgbPb6ZNs/YE7rrL7iRKeaYHYx5k\nVvdZrElcQ6uprTh86rDdkbyGS4eGjDGngRecD5eIiAOYALQFEoC1IjLXGLM10zqVgWFAU2PMMRHx\nyAsujbEKQcuWEBVldxqlPFf3Gt0J8gui2+xu3DblNn7q/xMRQRF2xyrwXL1q6BsRmXvZ4xMReUpE\nArN4WUMg3hiz2xhzDpgJdL5snUeACcaYYwDGmEPX2xA7rVkD8fFw7712J1HK83Wo3IF5feex+9hu\n7ppxF6fPn7Y7UoEnrnQPKyLjgJuwDgsB9MI6PGSAUGPMfVd5TXegnTHmYef0fUAjY8zATOt8DewE\nmgIOYLgx5vurvNcAYABAZGRk/ZnZ9N2QkpJCSEhItu3KLRMm3MzcuaX44osVhISk59l2L5fX7c4v\ntN0F00+Hf2L41uHcUuwWXop+CYc4Li4r6G3Pyo20u2XLluuNMbFXXWiMyfYBrM1qHrAli9d0B/4v\n0/R9wPjL1vkW+ArwAyoA+4Hwa2WpX7++yc6SJUuyXSc3nT5tzIoVebrJq8rrducX2u6C653V7xiG\nYx779jGTkZFxcb43tP1qbqTdwDqTxfeqq5ePhohIWWPMPgARKQtcKEvnsnhNIlAm03SUc15mCcBq\nY8x54A8R2QlUBta6mCtfKFQImjSxO4VSBc/AhgPZf2I/I1eMpGxYWYbeetVrTtQNcrUQ/BNYJiK/\nA4L16/1xEQkGpmbxmrVAZeeNZ4lAb6DvZet8DfQBJotIBFAF2J2zJthr6FDrBPHAgdmvq5TKudfa\nvMb+k/sZtngYZcPK0rfW5V8j6ka5etXQPOcVPtWcs3YYYy70Izs2i9ekichAYAHW8f+PjDFbRORl\nrF2Uuc5lt4vIViAdeNYYk3QD7clTJ07AuHHw0EN2J1Gq4PIRHyZ3nsyB5AP0+7ofJUNKIuhdm7kp\nJ72PVgaqAnWAniJyf3YvMMbMM8ZUMcbcbIx5xTnvRWcRwHnoarAxpoYxppYxxqN68P/iC6uDufuu\nOFWulMpNAb4BfNXrKyoXq0yXWV3YlbzL7kgFiquXj/4XeMf5aAmMBDq5MZdH+PhjqFwZGja0O4lS\nBV+RQkWY13ceYYFhPLPxGVbs184NcourewTdgdbAn8aY/lh7BWFuS+UB9u6FpUvh/vu1byGl8kq5\n8HL83P9nivgXoe0nbVm0e5HdkQoEVwvBGWNMBpAmIqHAIS69IsjrpKTAnXfCPffYnUQp71I2rCzj\n6o7j5iI3c+f0O1n4+0K7I3k8VwvBOhEJBz4E1gO/ACvdlsoDREfDt99CBe2MW6k8V9S/KHH94qgW\nUY0en/Vg6+Gt2b9IZcmlQmCMedwYc9wY8z5W30EPOA8ReaX9+2HfPrtTKOXdihYqyjd9vqGQbyE6\nTu+ondTdAFdPFi++8NwYs8cY81vmed5m5EioVg1OnbI7iVLerWxYWeb0nsPBlIN0nd2Vs2ln7Y7k\nka5ZCEQkUESKAhEiUkREijof5YHSeREwv0lPh9mzrfMDwTp2hlK2axTViCmdp7Bs3zIenPugDnl5\nHbK7oewfWOMOlMI6N3Dh+piTwHg35sq3fv4ZDh2CXr3sTqKUuqBXzV78fux3XvjxBcqEluH1Nq/b\nHcmjXLMQGGPGAeNEZJAx5p08ypSvffaZ1bdQ+/Z2J1FKZTbs1mHsP7GfN5a/QVRoFAMbar8vrnK1\ni4l3ROQWoHzm1xhjPnZTrnzJGJgzRw8LKZUfiQjjO4znQMoBnpz/JKUKl6Jr9a52x/IILhUCEfkE\nuBnYgNUnEFhjEXhVIRCB9eshOdnuJEqpq3H4OJjRbQZtPm5D3y/6suDeBdxW/ja7Y+V7rvY+GgvU\ncPZp7dUiI62HUip/CvIL4raEeJcAAB3NSURBVJs+39BscjM6zezET/1+ok6JOnbHytdcvaFsM1DC\nnUHyu4wM6wTxQr2JUal8r1hQMRbcu4DQgFDaTWvHH8f+sDtSvuZqIYgAtorIgszjFrszWH6zcqV1\n2WiSx3SSrZR3KxNWhu/v+Z6zaWfpNLMTZ86fsTtSvuXqoaHh7gzhCT77DAICoGNHu5MopVwVXTya\n6d2m035ae5794VnGd/DKq96z5WoXE0uBPYCf8/larP6GvIIx1tgDd9wBhQvbnUYplRPtKrXjn03+\nyYS1E5i7w6sOZLjM1S4mHgE+ByY6Z5XGGmbSK6xbBwkJ0K2b3UmUUtfj1davUq9kPfrP6U/iycuH\nTleuniN4AmiKdUcxxphdQHF3hcpvzpyBW27Rw0JKeSp/hz8zus3gbNpZ7vvqPu2G4jKuFoKzxphz\nFyZExBfrPgKv0Lw5LF8ORYvanUQpdb2qFKvCuHbjWLJnCRPWTLA7Tr7iaiFYKiLPA4VEpC3wGfCN\n+2LlHydO6A1kShUUD8Y8SIfKHRiyaAg7k3baHSffcLUQDAUOA5uwOqKbB/zbXaHyk/feg+LF4ehR\nu5MopW6UiPDhXR8S6BtIv6/7kZ6Rnv2LvICrhaAQ8JExpocxpjvwkXNegffVV1Crlh4WUqqgKFW4\nFOM7jGdlwkpGrRhld5x8wdVCsJhLv/gLAQV+1OjERFizBu6+2+4kSqnc1KdmH7pV78aLcS/y21+/\n2R3Hdq4WgkBjTMqFCefzIPdEyj++dl4g26WLvTmUUrlLRHi/4/sULVSUe768h9S0VLsj2crVQnBK\nROpdmBCR+kCBv1/7q6+galWoXt3uJEqp3BYRFMFHnT5i86HNPL/4ebvj2MrVLiaeAj4TkQNYo5SV\nAAr8GF1jx1qjkSmlCqb2ldvzRIMneGvVW9xZ+U5aV2xtdyRbZFsIRMQH8AeqAVWds3cYY867M1h+\nULOm3QmUUu42su1IFv+xmAe+foBtT2yjcID39SOT7aEhY0wGMMEYc94Ys9n5KPBF4MMPYf58u1Mo\npdwtyC+IKZ2nkJicyCs/v2J3HFu4fNWQiHQTEcl+Vc+XlgZDh8LMmXYnUUrlhUZRjehXtx9jVo5h\nV9Iuu+PkOVcLwT+w7iY+JyInRSRZRE66MZetVq2ybiDTvoWU8h6vtX6NQN9ABi8cbHeUPOdqN9SF\njTE+xhg/Y0yoczrU3eHs8u234OsLt99udxKlVF4pEVKC/zT/D9/u/Jb5u7zruLCr3VCLiNwrIv9x\nTpcRkYYuvK6diOwQkXgRGXqN9bqJiBGRWNeju8+331odzYWF2Z1EKZWXnmr8FFWKVeHpBU9zNu2s\n3XHyjKuHht4FmgB9ndMpwDW77xMRh3Od9kANoI+I1LjKeoWxLk9d7WIWt7rQyZweFlLK+/g7/BnX\nbhw7k3by8tKX7Y6TZ1wtBI2MMU8AqQDGmGNYl5ReS0Mg3hiz29mF9Uyg81XW+x/wxoX3tltYGOzZ\nAwMH2p1EKWWHdpXa0b9uf15f/jqrE/LF71O3c/WGsvPOX/gGQERuArIb2aE0sD/TdALQKPMKzruV\nyxhjvhORZ7N6IxEZAAwAiIyMJC4u7pobTklJyXadrBgDnnpt1I2025Npu72Pu9veNbgr3/l/R4/p\nPfiw/ocEOALctq2ccFu7jTHZPoB7gLlYX+avADuAHtm8pjvwf5mm7wPGZ5r2AeKA8s7pOCA2uyz1\n69c32VmyZEm261xNcrIxZcoYM2PGdb3cdtfbbk+n7fY+edH2H37/wTAc88z3z7h9W666kXYD60wW\n36uuXjU0DXgOeA04CNxtjPksm5clAmUyTUc5511QGKgJxInIHqAxMNfOE8ZLlsD+/db4A0op79am\nYhsei32MsavGsnzfcrvjuNU1C4GIBIrI0yIyHrgNmGiMGW+M2ebCe68FKotIBRHxB3pj7VUAYIw5\nYYyJMMaUN8aUB1YBnYwx6667NTdo4UIICoKmTe1KoJTKT0a2HUlUaBSPfvco59MLbocK2e0RTAVi\nsUYmaw+4PIqDMSYNGAgsALYBs40xW0TkZRHpdJ153WrBAmjRAgLyx+FApZTNQvxDeKf9O2w+tJmx\nq8baHcdtsjtZXMMYUwtARCYBa3Ly5saYeVjDWmae92IW67bIyXvntj/+gF279GohpdSlOlfrTKeq\nnRi+dDg9o3tSLryc3ZFyXXZ7BBf3hZy/8AssERg0CDp0sDuJUiq/eaf9OwA8+f2TNidxj+wKQR1n\n30InRSQZqF1Q+xoqXx7efhsqVbI7iVIqvykbVpaXWrzE3B1z+XLbl3bHyXXXLATGGIex+ha60L+Q\nrymAfQ2dPw+rV0N6ut1JlFL51VONniKmRAyPf/c4SaeT7I6Tq1y9s7hAW70aGjf+e4xipZS6nJ/D\nj8mdJ5N0Jomnvn/K7ji5SgsB1tVCPj7QqpXdSZRS+VmdEnV4odkLTNs0jbk75mb/Ag+hhQDr/oFG\njaBIEbuTKKXyu+ebPU/tyNr849t/cPTMUbvj5AqvLwRJSbB2rY49oJRyjb/Dnymdp3D41GFe+alg\nDG3p9YXgxx+tjua0ECilXBVTMoae0T354JcPOHbmmN1xbpjXF4KOHeGHH6BhtsPsKKXU355r+hwp\n51J4b917dke5YV5fCAoVgjZtrKEplVLKVXVL1OWOm+9g3OpxpKbli+FUrptXF4J9++CFF6weR5VS\nKqeGNB3CoVOHmLphqt1RbohXF4Lvv4dXX4VTp+xOopTyRC3KtyC2VCyjVo4iPcNz70j16kKwaBGU\nLg1Vq9qdRCnliUSEIU2HEH803qO7nvDaQpCRAYsXW+cHPHVoSqWU/bpU60KlopUYs2qM3VGum9cW\ngg0b4OhRaNvW7iRKKU/m8HHwZMMnWZWwilUJq+yOc128thDs3g2hodC6td1JlFKern9Mf8ICwhi3\nepzdUa6L1xaC7t2tu4pLlLA7iVLK04X4h/BwvYf5bMtnJJxMsDtOjnltIQC9d0AplXsGNhyIwTB+\nzXi7o+SYVxaCxYuhVi3Yvt3uJEqpgqJ8eHm6Vu/KB+s/4NQ5z7om3SsLwY8/wrZt1qWjSimVW55u\n9DTHUo/x8caP7Y6SI15ZCOLioEEDKFzY7iRKqYLkljK30KBUA0avHE1ahucM8+51hSAlBdasgZYt\n7U6ilCpoRIRhtw7j92O/M3vLbLvjuMzrCsGKFZCWBi1a2J1EKVUQda7WmRo31eC1Za+RYTLsjuMS\nrysE4eHQpw/ccovdSZRSBZGP+DDs1mFsPrSZb3Z8Y3ccl3hdIWjYEKZPh5AQu5MopQqq3jV7U7FI\nRV75+RWMMXbHyZZXFYLUVPjjD7tTKKUKOl8fX4Y0HcLaA2tZtHuR3XGy5VWFYOlSqFjRumpIKaXc\n6YE6D1CqcCleXfaq3VGy5VWFYMkS8POzLh1VSil3CvAN4J9N/kncnrh83xmdVxWCuDjrHEFwsN1J\nlFLeYED9ARQtVJTXl71ud5Rr8ppCkJwM69bpZaNKqbwT4h/CoIaDmLNjDlsObbE7Tpa8phAsXw7p\n6XojmVIqbw1qOIggvyDeWP6G3VGy5DWFoGFDmDEDmjSxO4lSypsUCyrGgHoDmL5pOnuP77U7zlW5\ntRCISDsR2SEi8SIy9CrLB4vIVhH5TUQWi0g5d2UpWhR694agIHdtQSmlrm5wk8H4iA+jVoyyO8pV\nua0QiIgDmAC0B2oAfUSkxmWr/QrEGmNqA58DI92VRyml7FImrAz31r6XSb9OIul0kt1xruDOPYKG\nQLwxZrcx5hwwE+iceQVjzBJjzGnn5Cogyo15lFLKNoObDOZM2hkmrp9od5QruHOMrtLA/kzTCUCj\na6z/EDD/agtEZAAwACAyMpK4bO4IS0lJyXadgkjb7V28td3guW1vUKQBo5eNJvZ8LP4+/jl+vbva\nnS8GaxSRe4FY4LarLTfGfAB8ABAbG2taZHMNaFxcHNmtUxBpu72Lt7YbPLftI8qM4I5P7+DPYn9y\nf537c/x6d7XbnYeGEoEymaajnPMuISJtgBeATsaYs27Mo5RStmpbsS3RN0UzeuXofNUZnTsLwVqg\nsohUEBF/oDcwN/MKIhIDTMQqAofcmEUppWwnIgxuMpjf/vqNH//40e44F7mtEBhj0oCBwAJgGzDb\nGLNFRF4WkU7O1d4EQoDPRGSDiMzN4u2UUqpA6FurL8WDizN65Wi7o1zk1nMExph5wLzL5r2Y6Xkb\nd25fKaXym0DfQB6PfZzhS4fzx7E/qFCkgt2RvOfOYqWUyi8ejHkQQZi6cardUQAtBEoplefKhJWh\n7c1tmbJhSr4Y1zhfXD56o86fP09CQgKpqakAhIWFsW3bNptT5b3rbXdgYCBRUVH4+fm5IZVS6mr6\n1+1Pny/6sOSPJbSu2NrWLAWiECQkJFC4cGHKly+PiJCcnEzhwoXtjpXnrqfdxhiSkpJISEigQgX7\nj1Uq5S3urnY34YHhfLThI9sLQYE4NJSamkqxYsUQEbujeBwRoVixYhf3ppRSeSPQN5C+Nfvy5bYv\nOZ563NYsBaIQAFoEboB+dkrZ48GYB0lNS2Xm5pm25igwhUAppTxNvZL1qB1Zm8kbJtuaQwtBLnE4\nHNStW5eaNWvSo0cPTp8+nf2LsrFu3TqefPLJLJcfOHCA7t273/B2lFL2EBH61+3PmsQ1bD602bYc\nWghySaFChdiwYQObN2/G39+f999//5LlxhgyMnJ2mVhsbCxvv/12lstLlSrF559/fl15lVL5w721\n78XPx49Jv0yyLUOBLAQdOhSiRQsuebz7rrXs9GmuWNaiBUyZYi0/cuTKZTnVrFkz4uPj2bNnD1Wr\nVuX++++nZs2a7N+/n4ULF9KkSRPq1atHjx49SElJAWDt2rXccsst1KlTh4YNG5KcnExcXBwdO3YE\nYOnSpdStW5e6desSExNDcnIye/bsoWbNmoB1wvyxxx6jVq1axMTEsGTJEgCmTJlC165dadeuHZUr\nV+a5557LeYOUUm4TERTB3dXu5pPfPuFsmj39bhbIQmCntLQ05s+fT61atQDYtWsXjz/+OFu2bCE4\nOJgRI0awaNEifvnlF2JjYxkzZgznzp2jV69ejBs3jo0bN7Jo0SIKFSp0yfuOGjWKCRMmsGHDBn7+\n+ecrlk+YMAERYdOmTcyYMYMHHnjg4pVAGzZsYNasWWzatIlZs2axf/9+lFL5x0MxD5F0Jok5O+bY\nsv0CcR/B5ebNO5Pl9fRBQXCtcR0iIq69PCtnzpyhbt26gLVH8NBDD3HgwAHKlStH48aNAVi1ahVb\nt26ladOmAJw7d44mTZqwY8cOSpYsSYMGDQAIDQ294v2bNm3K4MGDueeee+jatStRUZcO5rZs2TIe\neughAKpVq0a5cuXYuXMnAK1btyYsLAyAGjVqsHfvXsqUKYNSKn9oU7ENZcPKMunXSfSM7pnn2y+Q\nhcAOF84RXC44OPjic2MMbdu2ZcaMGZess2nTpmzff+jQodx5553MmzePpk2bsmDBAgIDA13KFhAQ\ncPG5w+EgLS3NpdcppfKGw8dB/7r9eXnpy+w9vpdy4eXydPt6aCgPNW7cmOXLlxMfHw/AqVOn2Llz\nJ1WrVuXgwYOsXbsWsO4QvvzL+vfff6dWrVoMGTKEBg0asH379kuWN2vWjNmzZwOwc+dO9u3bR9Wq\nVfOgVUqp3NC/bn8AWy4l1UKQh2666SamTJlCnz59qF27Nk2aNGH79u34+/sza9YsBg0aRJ06dWjb\ntu0Vd/qOHTuWmjVrUrt2bfz8/Gjfvv0lyx9//HEyMjKoVasWvXr1YsqUKZfsCSil8rdy4eVoU7EN\nH/36EekZ6Xm7cWOMRz3q169vLrd169ZLpk+ePHnFOt7gRtp9+WfoSZYsWWJ3BFt4a7uNKbhtn7V5\nlmE4Zt7OeVddfiPtBtaZLL5XdY9AKaXyic5VOxMRFMGHv3yYp9vVQqCUUvlEgG8A/ev2Z+6OuRxM\nPphn29VCoJRS+cjD9R4m3aTn6UljLQRKKZWPVClWhZblW/LhLx/m2ehlWgiUUiqfGVB/AHuO72HR\n7kV5sj0tBEoplc90qdaFiKAIPlj/QZ5sTwtBLsncDfVdd93F8eO5O+LQlClTGDhwIADDhw9n1KhR\nufr+Sqn8I8A3gH51+jFnxxz+TPnT7dvTQpBLMndDXbRoUSZMmGB3JKWUBxtQfwDpGem8u/Zdt2+r\nwPU19PT3T7M+cT0OhyPX3rNuibqMbTfW5fWbNGnCb7/9dnH6zTffZPbs2Zw9e5YuXbrw0ksvAfDx\nxx8zatQoRITatWvzySef8M033zBixAjOnTtHsWLFmDZtGpGRkbnWFqWUZ6hcrDKdq3Vm/JrxPNf0\nOUL8Q9y2Ld0jyGXp6eksXryYTp06AbBw4UJ27drFmjVr2LBhA+vXr+enn35iy5YtjBgxgh9//JGN\nGzcybtw4AG699VZWrVrFr7/+Su/evRk5cqSdzVFK2WhI0yEcSz3m9kFrCtwewdh2Y0lOTs6yG2p3\nudANdWJiItWrV6dt27aAVQgWLlxITEwMACkpKezatYuNGzfSo0cPIiIiAChatCgACQkJ9OrVi4MH\nD3Lu3DkqVKiQp+1QSuUfjaMa07xcc8asGsPjDR5323Z0jyCXXDhHsHfvXowxF88RGGMYNmwYGzZs\nYMOGDcTHx18cN+BqBg0axMCBA9m0aRMTJ068ovM5pZR3ee6W59h3Yh8zN8902za0EOSyoKAg3n77\nbUaPHk1aWhp33HEHH3300cUhKRMTEzl06BCtWrXis88+IykpCYCjR48CcOLECUqXLg3A1KlT7WmE\nUirf6FC5AzWL12TkipFYfcflPi0EbhATE0Pt2rWZMWMGt99+O3379qVJkybUqlWL7t27k5ycTHR0\nNC+88AK33XYbderUYfDgwYB1aWiPHj2oX7/+xcNGSinvJSI8e8uzbD60mdVHV7tnG+6qMO4SGxtr\n1q1bd8m8bdu2Ub169YvTdpwjyA9upN2Xf4aeJC4ujhYtWtgdI895a7vB+9p+Pv08d8+6m1aFWvHP\nLv+8rvcQkfXGmNirLXPrHoGItBORHSISLyJDr7I8QERmOZevFpHy7syjlFKeyM/hx3d9v6N+kfpu\neX+3FQIRcQATgPZADaCPiNS4bLWHgGPGmErAW8Ab7sqjlFLq6ty5R9AQiDfG7DbGnANmAp0vW6cz\ncOGM6OdAaxGR69mYpx3iyk/0s1PKu7nzPoLSwP5M0wlAo6zWMcakicgJoBhwJPNKIjIAGAAQGRlJ\nXFzcJW8SEhJCQkICYWFhiAjp6ekkJyfnZls8wvW02xjDiRMnOHXq1BWfq6dISUnx2Ow3wlvbDd7b\ndne12yNuKDPGfAB8ANbJ4stPEp0/f56EhAQSExMBSE1NJTAwMK9j2u562x0YGEidOnXw8/NzQyr3\n87YThxd4a7vBe9vurna7sxAkAmUyTUc5511tnQQR8QXCgKScbsjPz++SO3Dj4uIu3snrTby13Uqp\nG+POcwRrgcoiUkFE/IHewNzL1pkLPOB83h340egBa6WUylNu2yNwHvMfCCwAHMBHxpgtIvIysM4Y\nMxeYBHwiIvHAUaxioZRSKg+59RyBMWYeMO+yeS9mep4K9HBnBqWUUtfmcXcWi8hhYG82q0Vw2ZVH\nXkLb7V28td3gvW2/kXaXM8bcdLUFHlcIXCEi67K6lbog03Z7F29tN3hv293Vbu10TimlvJwWAqWU\n8nIFtRB8YHcAm2i7vYu3thu8t+1uaXeBPEeglFLKdQV1j0AppZSLtBAopZSX8+hC4K0D37jQ7sEi\nslVEfhORxSJSzo6cuS27dmdar5uIGBEpEJcXutJuEenp/JtvEZHpeZ3RHVz4d15WRJaIyK/Of+sd\n7MiZ20TkIxE5JCKbs1guIvK283P5TUTq3fBGjTEe+cDqtuJ3oCLgD2wEaly2zuPA+87nvYFZdufO\no3a3BIKczx/zlnY71ysM/ASsAmLtzp1Hf+/KwK9AEed0cbtz51G7PwAecz6vAeyxO3cutb05UA/Y\nnMXyDsB8QIDGwOob3aYn7xHk6cA3+Ui27TbGLDHGnHZOrsLq+dXTufL3Bvgf1kh3qXkZzo1cafcj\nwARjzDEAY8yhPM7oDq602wChzudhwIE8zOc2xpifsPpey0pn4GNjWQWEi0jJG9mmJxeCqw18Uzqr\ndYwxacCFgW88mSvtzuwhrF8Pni7bdjt3kcsYY77Ly2Bu5srfuwpQRUSWi8gqEWmXZ+ncx5V2Dwfu\nFZEErD7NBuVNNNvl9DsgWx4xMI26PiJyLxAL3GZ3FncTER9gDNDP5ih28MU6PNQCa+/vJxGpZYw5\nbmsq9+sDTDHGjBaRJlg9Gdc0xmTYHczTePIeQU4GvuFGBr7JZ1xpNyLSBngB6GSMOZtH2dwpu3YX\nBmoCcSKyB+vY6dwCcMLYlb93AjDXGHPeGPMHsBOrMHgyV9r9EDAbwBizEgjE6pStoHPpOyAnPLkQ\neOvAN9m2W0RigIlYRaAgHC+GbNptjDlhjIkwxpQ3xpTHOjfSyRizzp64ucaVf+dfY+0NICIRWIeK\ndudlSDdwpd37gNYAIlIdqxAcztOU9pgL3O+8eqgxcMIYc/BG3tBjDw0ZLx34xsV2vwmEAJ85z43v\nM8Z0si10LnCx3QWOi+1eANwuIluBdOBZY4xH7/m62O5/Ah+KyDNYJ477FYAfeojIDKzCHuE8//Ff\nwA/AGPM+1vmQDkA8cBrof8PbLACfm1JKqRvgyYeGlFJK5QItBEop5eW0ECillJfTQqCUUl5OC4FS\nSnk5LQQqz4hIuohsEJHNIvKZiATl8PUpOVx/ioh0v8r8WBF52/m8n4iMdz5/VETuzzS/VKbX/J+I\n1MjJ9i/bZn9n2zeIyDkR2eR8/rqIDBeRf13ve19jm1dt/zXWL3+NHi/jCsDNeSoLHnsfgfJIZ4wx\ndQFEZBrwKFa3EDjnCdYlzW7tIsB5k9kVN5o5r9G+oB+wGWdHZsaYh29wm5OByQDOO59bGmOOOKeH\nZ/f6vPpslHfSPQJll5+BSs5foTtE5GOsL94yItLH+Yt5s4i8kflFIvKWs8/9xSJyk3PeIyKyVkQ2\nisgXl+1ptBGRdSKyU0Q6OtdvISLfXh7owi9z56/oWGCa81d7ocy/iEXkdhFZKSK/OPdsQpzzX5e/\nx4EYlcPPo4ZzG7tF5Enn+13ts8nptpuLyArn+3Z3risi8qbz890kIr2u8lkUEpGZIrJNRL4CCuWw\nPcqDaCFQeU6sfp/aA5ucsyoD7xpjooHzWN1ItwLqAg1E5G7nesFYd5VGA0ux7rgE+NIY08AYUwfY\nhtUHzQXlsbo0vhN4X0QCs8tnjPkca4/hHmNMXWPMmUzZI4B/A22MMfWc6w0WkWJAFyDaGFMbGJGT\nzwSoBtzhzPpfEfFzzs/82Zy6jm2XBG4FOgKvO+d1xfps6wBtgDflym6MHwNOG2OqY33O9XPYHuVB\ntBCovFRIRDZgfYHtw+oCBGCvs191gAZAnDHmsLPr8GlYA3UAZACznM8/xfqCA6gpIj+LyCbgHiA6\n0zZnG2MyjDG7sPrfqXaDbWiMNQjKcmdbHgDKYXVxngpMEpGuWLf+58R3xpizzsNFh4BI5/zMn831\nbPtrZ/u3ZnrPW4EZxph0Y8xfWEW1wWV5mmN9xhhjfgN+y2F7lAfRcwQqL108R3CBdeibU9f5fhf6\nR5kC3G2M2Sgi/XB2wHbZOllN55QAPxhj+lyxQKQhVido3YGBWHs1rsrcQ2w6f/+/mfmzuZ5tZ35f\nTx+USbmJ7hGo/GYNcJuIRIiIA6vP+aXOZT5YX3QAfYFlzueFgYPOwyn3XPZ+PUTER0Ruxhr2cIeL\nOZKd73u5VUBTEakEICLBIlLFeaw+zBgzD3gG67BLbsutbf8M9BIRh/M8S3Oszz2zn7A+Y0SkJlA7\nF9uh8hndI1D5ijHmoFgDlS/B+gX7nTFmjnPxKaChiPwb6/DJhZOc/wFWY3VBvJpLv8D3YX3JhQKP\nGmNSxbXRSqdgnVM4AzTJlO+wc69jhogEOGf/G6twzHGegxBgcE7a7Ypc3PZXWG3aiLWH9Jwx5k8R\nKZ9pnfeAySKyDeu8y/rcaofKf7T3UaWU8nJ6aEgppbycFgKllPJyWgiUUsrLaSFQSikvp4VAKaW8\nnBYCpZTycloIlFLKy/0//7NHtpX2fN8AAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "tags": [] }, "output_type": "display_data" } ], "source": [ "precisions, recalls, thresholds_pr = precision_recall_curve(Y_train, Y_scores)\n", "def plot_precision_recall_vs_threshold(precisions,recalls, thresholds):\n", " plt.plot(thresholds_pr, precisions[:-1], \"b--\", label=\"Precision\")\n", " plt.plot(thresholds_pr, recalls[:-1], \"g-\", label=\"Recall\")\n", " plt.legend()\n", " plt.grid()\n", "\n", "plot_precision_recall_vs_threshold(precisions,recalls, thresholds_pr)\n", "plt.xlabel(\"Probabilities Threshold\")\n", "plt.ylabel(\"Percentage\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "qnomlg_xp31E" }, "source": [ "Ora se scelgo di avere una precision di 0.96 quale sarà la mia recall e viceversa?" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 35 }, "colab_type": "code", "id": "sHynnXgkqoVp", "outputId": "31abf598-73f3-4d12-93b2-dfc5845fd503" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "True\n" ] } ], "source": [ "print(Y_train[0])" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 53 }, "colab_type": "code", "id": "vr5oDYaNqFV3", "outputId": "c6c2f462-87da-4a80-e5f6-11ba1b2e2c6c" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "precision: 0.96 recall: 0.94\n", "Threshold precision: 0.35\n" ] } ], "source": [ "# Set Precision\n", "chosen_precision = 0.96\n", "threshold_chosen =thresholds_pr[np.argmax(precisions >= chosen_precision)]\n", "Y_train_pred_new = (Y_scores >= threshold_chosen) # Set True if Y_scores > threshold_precision\n", "res_precision_train_score_new = precision_score(Y_train, Y_train_pred_new)\n", "res_recall_train_score_new = recall_score(Y_train, Y_train_pred_new)\n", "print(\"precision: \", np.round(res_precision_train_score_new,2), \"recall: \", np.round(res_recall_train_score_new,2))\n", "print(\"Threshold precision: \",threshold_chosen)" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 279 }, "colab_type": "code", "id": "vElqcZRmspSx", "outputId": "9fde4b6d-95af-44ac-c8f7-0b07c8c738c7" }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAf2klEQVR4nO3de5ydVX3v8c9375k9k8kkhBAYMIkk\naBDCxQtTwNKjQ6UaaA+0ai0Urag0tce0trX24LEvaml5+aqttqfHtD05grdWI/rq6WusOaUV2aAW\nNEEQuRiM4ZaA3G+TTDK33/njeWayM5nMDCZrX+b5vl+v/ZrnsuZ5fiuB/cta63nWUkRgZmbFVWp0\nAGZm1lhOBGZmBedEYGZWcE4EZmYF50RgZlZwbY0O4MVasmRJrFixYtoyu3btYv78+fUJqIm43sXi\nehfPodT9tttuezIijp7qXMslghUrVrBly5Zpy1SrVfr6+uoTUBNxvYvF9S6eQ6m7pAcPds5dQ2Zm\nBedEYGZWcE4EZmYF50RgZlZwTgRmZgWXLBFIulbS45LuOsh5SfpbSdsk3SnpNaliMTOzg0vZIvgM\nsGaa8+cDq/LPWuDvE8ZiZmYHkew9goi4WdKKaYpcBHwusnmwb5W0SNJxEfFoing2P/A037zviRSX\nbhoPPDjE94a2NjqMujuc9S6VRFmiXM5/lrJPW0m0l0tU2vJPzXZ7Ofv3VO2M7rXTu8fEsZrz7Lez\nXzkACTraSlTKZTras/u1t5VQTbknB8fY8cxuSpPiLJW0716x714x6T61MUpZfUslaCuVKJX2j7mt\nJNrK7kmeqxr5QtlS4OGa/R35sQMSgaS1ZK0Genp6qFar0154YGDggDKb7h/iy1uHDyng5hfw422N\nDqIBDk+9W3JljpturNut2gTtZegoi8426Mx/tkn7JbaSRElQFpTyT1lZsoksM4GghCiXsm6JUikr\nU1b+9xBQLolFHWJ+O1TKolKGSkmM7B3kwf4baBu/T2nfPbraRUdZs67TMV//Oid86lN0PP44e485\nhu2XX87j5513uP/oDpupvtsOh5Z4szgiNgAbAHp7e2OmN+umevuurw8+lia8plHUNy4PZ73HxoLR\nCEbH8k8Eo6PByFgwPDrG0MgYQ/nPvSPZz5GxMUT25aOa76D9vo40/mPf0anKKj84OhYT98nuNcrw\n6FhNefHDrT/kpJNOIiKLb2ws+zk6FvtdSzX3mnyf8eNZvfevf22cI6PB4PAog0Oj7B4aYdfQKLv2\njrBr7wgj4/fLy48GjI6NsXc0/zMc3XfNkoQEYxGMjWX1zGIeY2QsGBkNpOxaQ6OjDI9OlZ4F7Jni\neKarUqar0sa8SomFne0cs6CDJd0dLO6usLirQrmURXpK9V95xcc/QXnPIACdjz3G6r/+a1affDJc\neulBr99Iqf4fb2Qi2Aksr9lflh8za5hSSZQQ7eVGRzKz6q4f09e7fOaCLSoieGb3MC/sGZ5IQnuG\nx9j8vds58eRTGB2DkbGxiWQyNDLGc4PDPL1riN1Do+wZHuW5wWEef2EP9z76Ak/vGmKoJpl+6+//\nfCIJTNi9m4EP/BH/+ao3cMLR81m5pHsiccxljUwE/cA6SRuBs4DnUo0PmFnrkcTi+RUWz6/sd3zv\nw230nXrci75eRLB7aJSxyDqyFnzsySnLdT32KGs/f1u2XSlz5srFvPaEo1h65DxO7FnAy4/upjTH\nkkOyRCDpi0AfsETSDuBPgHaAiPgHYBNwAbAN2A28K1UsZmaSmN9R85X30pfCgwfOwza2fBn9687h\nR48NcOeOZ7lx6xNUt+570GRRVzvnvGwJ7zpnBb0rFtcj9ORSPjV0yQznA3hfqvubmU3r6qth7VrY\nvXvfsa4u2j76UU5ftojTly3iLWcs40+B5/cMs/OZQe7a+RybH3iar9/7OF/7waOctXIxbzrlWM5c\nuZhTlx7RsKocqpYYLDYzO+zGB4Q//GF46KGshXD11VMOFC/sbGfhce2cfNxCfrV3ObuHRvjidx/m\nmm9u56p/vQeAN796Ke/+uZWc8pKFE4PxrcKJwMyK69JLf6onhLoqbbzn51by7nNW8MTAXq755v18\n9pYH+Ofbd3LCkvl84I2v4ILTjm2ZhOA3RMzMfkqSOGZBJx+64GS+86HzuPpXTqWzvcz7vvA93vQ3\nN7Pxuw8xNtb8b6g4EZiZHQZHdLVz6VnH07/uHD72ltPpaCtzxT//gMs+s5nHXzj4ew/NwInAzOww\naiuXeNvPLKd/3Tn82S+fyne2P8Uv/e23+P7DzzY6tINyIjAzS0AS7zj7eP7lfedQaSvxrs9sZscz\nu2f+xQZwIjAzS+jk4xbyuXefyZ7hUf7y+uacFNKJwMwssROO7ubtZx/PV7//CLc9+HSjwzmAE4GZ\nWR289/Uv46WLu3jntZu5/aFnGh3OfpwIzMzqYPH8Cl9cezaL51d457XfZetPXmh0SBOcCMzM6uS4\nI+bxT5efRaWtzAe+fMd+0303khOBmVkdLV/cxUcuXM1dO5/nC999qNHhAE4EZmZ194unHcdZKxfz\nN/9xHy/safzKiU4EZmZ1Jon/ccHJPLVriM/dcuBU2PXmRGBm1gCvXL6I1514NJ/+9v0MDo02NBYn\nAjOzBll37st5cmCIf7y1sa2CpIlA0hpJWyVtk3TFFOePl3SDpDslVSUtSxmPmVkzOXPlYn7u5Uv4\n+5t+3NBWQbJEIKkMrAfOB1YDl0haPanYXwGfi4jTgauAj6aKx8ysGb339S/j6V1D3HTfEzMXTiRl\ni+BMYFtEbI+IIWAjcNGkMquBb+TbN05x3sxsTjv7hMUs6mrn3+56tGExpFyhbCnwcM3+DuCsSWW+\nD7wZ+J/ArwALJB0VEU/VFpK0FlgL0NPTQ7VanfbGAwMDM5aZi1zvYnG9545TFgX/cfcj3PCNZymX\nDr6qWaq6N3qpyj8EPinpMuBmYCdwQEdZRGwANgD09vZGX1/ftBetVqvMVGYucr2LxfWeOwYWP8K3\nv3A7R5zwSnpXLD5ouVR1T9k1tBNYXrO/LD82ISIeiYg3R8SrgQ/nx5p39QYzswT+y6qjaSuJG374\neEPunzIRbAZWSVopqQJcDPTXFpC0RNJ4DB8Crk0Yj5lZUzpiXjtnn3AU19/1EyLqP/9QskQQESPA\nOuB64F7guoi4W9JVki7Mi/UBWyXdB/QAV6eKx8ysmb3p1GPZ/uQu7ntsoO73TvoeQURsiogTI+Jl\nEXF1fuzKiOjPt78SEavyMpdHxN6U8ZiZNas3ru4B4Mat9e8e8pvFZmZNoGdhJycdu4Cbttb/fQIn\nAjOzJvG6E49my4NPMzQyVtf7OhGYmTWJU16ykOHR4P4nd9X1vk4EZmZN4hXHLgBg62P1XcbSicDM\nrEmcsKSbkmCbE4GZWTFV2kocd8Q8djwzWNf7OhGYmTWRpYucCMzMCm3pkfPY+awTgZlZYb1kUSeP\nPjfI6Fj9pppwIjAzayLHLuxkLOCpgfpNtOBEYGbWRHoWdgLw2PNOBGZmhTSeCH7y/J663dOJwMys\niRzVXQHg6V1uEZiZFdKCznYAXtgzUrd7OhGYmTWR7o5sBeHnnQjMzIqpXBLdHW28sGe4bvd0IjAz\nazJHdVd4cmCobvdLmggkrZG0VdI2SVdMcf6lkm6UdLukOyVdkDIeM7NW8JIj5vFIHd8uTpYIJJWB\n9cD5wGrgEkmrJxX7Y7K1jF9Ntrj936WKx8ysVbxk0Tx21nG+oZQtgjOBbRGxPSKGgI3ARZPKBLAw\n3z4CeCRhPGZmLWHpkfN47IU9DI/WZ6UyRaSZz0LSW4E1EXF5vv8O4KyIWFdT5jjg34EjgfnAeRFx\n2xTXWgusBejp6Tlj48aN0957YGCA7u7uw1WVluF6F4vrPXfdtGOYT981xF++bh5Hd+379/qh1P3c\nc8+9LSJ6pzrX9tOFedhcAnwmIj4u6bXA5yWdGhH7pcGI2ABsAOjt7Y2+vr5pL1qtVpmpzFzkeheL\n6z136b4n+PRd32Xl6lfRu2LxxPFUdU/ZNbQTWF6zvyw/Vus9wHUAEXEL0AksSRiTmVnTW9CZ/Rv9\nhb31eZcgZSLYDKyStFJShWwwuH9SmYeANwBIOpksETyRMCYzs6a3ME8EA3V6qSxZIoiIEWAdcD1w\nL9nTQXdLukrShXmxDwC/Ken7wBeByyLVoIWZWYvo7qjvNBNJxwgiYhOwadKxK2u27wHOSRmDmVmr\n6R5vEeytz9vFfrPYzKzJdLWXKQmeH2zxriEzM/vplEriyK4KT++uzzQTTgRmZk1o8fwKT9dpviEn\nAjOzJrR4foWndzkRmJkV1pLuDp6s0yplTgRmZk3ILQIzs4JbPL/Cs7uHGanDxHNOBGZmTWhRV/ZS\nWT2WrHQiMDNrQkfMyxLBc4PpXypzIjAza0LjieB5JwIzs2Jyi8DMrOAWOhGYmRXbRNfQHicCM7NC\ncteQmVnBdbSVKAl27x1Nfi8nAjOzJiSJrkobu4daPBFIWiNpq6Rtkq6Y4vxfS7oj/9wn6dmU8ZiZ\ntZLO9jKDw+kTQbIVyiSVgfXALwA7gM2S+vNVyQCIiN+vKf87wKtTxWNm1mq6KmUGh1r7zeIzgW0R\nsT0ihoCNwEXTlL+EbN1iMzMjSwT16BpSqrXiJb0VWBMRl+f77wDOioh1U5Q9HrgVWBYRB9Ra0lpg\nLUBPT88ZGzdunPbeAwMDdHd3H3olWozrXSyu99x31S2DdLWJP/yZTuDQ6n7uuefeFhG9U51Lunj9\ni3Ax8JWpkgBARGwANgD09vZGX1/ftBerVqvMVGYucr2LxfWe+zb86FaGRsbo6/tZIF3dU3YN7QSW\n1+wvy49N5WLcLWRmtp95zTZYLGkpcHzt70TEzdP8ymZglaSVZAngYuDXp7juScCRwC2zjcXMrAg6\nK02UCCT9BfBrwD3AeFQBHDQRRMSIpHXA9UAZuDYi7pZ0FbAlIvrzohcDGyPVYIWZWYua115m73D6\nhWlm2yL4ZeAVEfGiFtCMiE3ApknHrpy0/5EXc00zs6LobC/VpUUw2zGC7UB7ykDMzGx/89rLDNbh\n8dHZtgh2A3dIugGYaBVExO8micrMzCYGiyMCScnuM9tE0J9/zMysTjorZQD2jozR2V5Odp9ZJYKI\n+KykCnBifmhrRKSfG9XMrMDm5V/+g0OjjU8EkvqAzwIPAAKWS3rnDI+PmpnZIRj/8h8cHuXIhPeZ\nbdfQx4E3RsRWAEknkr0AdkaqwMzMiq5Szp7nGRpJ+wjpbJ8aah9PAgARcR9+isjMLKlKW/YVPTya\nNhHMtkWwRdKngH/M9y8FtqQJyczMYF8i2Ju4RTDbRPDbwPuA8cdFvwn8XZKIzMwM2JcIhpqhRZC/\nUfyJ/GNmZnXQUacxgmkTgaTrIuJtkn5ANrfQfiLi9GSRmZkV3ESLoMFdQ+/Pf/5S0ijMzOwA7eX6\nDBZP+9RQRDyabz4JPBwRDwIdwCuBR5JGZmZWcPVqEcz28dGbgc58TYJ/B94BfCZVUGZmBh11empo\ntolAEbEbeDPwdxHxq8Ap6cIyM7PxN4v3JJ6KetaJQNJryd4f+Fp+LN3EF2Zmtm+uoSZJBL8HfAj4\nv/kqYycAN870S5LWSNoqaZukKw5S5m2S7pF0t6QvzD50M7O5bV+LoDneI7gJuKlmfzv7Xi6bkqQy\nsB74BWAHsFlSf0TcU1NmFVmCOScinpF0zIuvgpnZ3DQ+RpC6RTDTewR/ExG/J+mrTP0ewYXT/PqZ\nwLY8aSBpI3AR2brH434TWB8Rz+TXe/xFxm9mNmeVSqKzvcTeRiYC4PP5z7/6Ka69FHi4Zn8HcNak\nMicCSPo22ZjDRyLi3yZfSNJaYC1AT08P1Wp12hsPDAzMWGYucr2LxfUuhjJjbHvgIarVx5LVfdpE\nEBG35ZtbgMGIGIOJbp+Ow3T/VUAfsAy4WdJpEfHspDg2ABsAent7o6+vb9qLVqtVZiozF7nexeJ6\nF8PCW27gqGOW0Nf3ymR1n+1g8Q1AV83+PODrM/zOTmB5zf6y/FitHUB/RAxHxP3AfWSJwczMyAaM\nUw8WzzYRdEbEwPhOvt01TXmAzcAqSSvzZS4v5sB1j/+FrDWApCVkXUXbZxmTmdmc15kvYJ/SbBPB\nLkmvGd+RdAYwON0vRMQIsA64HrgXuC5/9PQqSeODzNcDT0m6h+xx1A9GxFMvthJmZnNVZ3sp+Qtl\ns12P4PeAL0t6hGzN4mOBX5vplyJiE7Bp0rEra7YD+IP8Y2Zmk3S2ldnbJO8RbJZ0EvCK/NDWiBhO\nF5aZmQF0tJd4ZtdQ0nvMqmtIUhfw34H3R8RdwApJnprazCyxzrbmGSz+NDAEvDbf3wn8eZKIzMxs\nQkd7ib0jzTFY/LKI+BgwDJDPRKpkUZmZGZBNM9Es01APSZpHPs2EpJcBe5NFZWZmwPh7BM3x1NCf\nAP8GLJf0T8A5wGWpgjIzs0w9WgQzJgJJAn5ItijN2WRdQu+PiCeTRmZmZnS0lRufCCIiJG2KiNPY\ntyiNmZnVQUdbidGxSLqA/WzHCL4n6WeSRWFmZlMaX5wmZatgtmMEZwFvl/QAsIuseygi4vRUgZmZ\nWfb4KKRdt3i2ieBNySIwM7ODai9niSBl19BMK5R1Au8FXg78ALgmn0zOzMzqoDKeCEYOWCTysJlp\njOCzQC9ZEjgf+HiySMzM7ADt+brFQ41qEQCr86eFkHQN8N1kkZiZ2QEq5WwSh6GEg8UztQgmZhh1\nl5CZWf01fIwAeKWk5/NtAfPy/fGnhhYmi8zMzOqSCKZtEUREOSIW5p8FEdFWsz1jEpC0RtJWSdsk\nXTHF+cskPSHpjvxz+aFUxsxsrqk0wRjBT01SGVgP/ALZIvWbJfVHxD2Tin4pItalisPMrJXtaxE0\n7qmhQ3EmsC0itkfEELARuCjh/czM5pzxx0dTDhYnaxEAS4GHa/Z3kL2hPNlbJL0OuA/4/Yh4eHIB\nSWuBtQA9PT1Uq9VpbzwwMDBjmbnI9S4W17sYHn4hSwB33PkDTu7ek6TuKRPBbHwV+GJE7JX0W2Tv\nLfz85EIRsQHYANDb2xt9fX3TXrRarTJTmbnI9S4W17sYfvzEAHz7Jk486WS6n/1Rkrqn7BraCSyv\n2V+WH5sQEU9FxPgCN58CzkgYj5lZyxnvGtqbcN3ilIlgM7BK0kpJFeBioL+2gKTjanYvBO5NGI+Z\nWcuZ35F13OweSvcqV7KuoYgYkbQOuB4oA9dGxN2SrgK2REQ/8LuSLgRGgKfxqmdmZvvpqmTTUO8a\nGk22UnzSMYKI2ARsmnTsyprtDwEfShmDmVkr62grUS6JwaFR6Ehzj5RdQ2Zmdogk0VUpsyth15AT\ngZlZk5tfaWP33nQL0zgRmJk1ua4OtwjMzAptfqWN3UNuEZiZFda8Splde90iMDMrrPmVslsEZmZF\n1tXR5jECM7Mim18p+6khM7Mi66q4RWBmVmgdbaXGLVVpZmaNVy6JkRZdoczMzA6DtpIYGQsi0iQD\nJwIzsybXlq9JkKpN4ERgZtbkyqVs/ulUyxY7EZiZNbn2cpYIxhI1CZwIzMyaXLmUfVWnGi92IjAz\na3LjLYKWTASS1kjaKmmbpCumKfcWSSGpN2U8ZmataHyMYDRR31CyRCCpDKwHzgdWA5dIWj1FuQXA\n+4HvpIrFzKyVteddQ604RnAmsC0itkfEELARuGiKcn8G/AWwJ2EsZmYta6JFkCgRpFy8finwcM3+\nDuCs2gKSXgMsj4ivSfrgwS4kaS2wFqCnp4dqtTrtjQcGBmYsMxe53sXiehfHfY9k8wy9MLA7Sd1T\nJoJpSSoBnwAum6lsRGwANgD09vZGX1/ftOWr1SozlZmLXO9icb2LY9edj8Kd36NjXleSuqfsGtoJ\nLK/ZX5YfG7cAOBWoSnoAOBvo94Cxmdn+8p6hlnyzeDOwStJKSRXgYqB//GREPBcRSyJiRUSsAG4F\nLoyILQljMjNrORpPBK0211BEjADrgOuBe4HrIuJuSVdJujDVfc3M5p4sE6RqESQdI4iITcCmSceu\nPEjZvpSxmJm1qvEWQSp+s9jMrMmN54FEPUNOBGZmzU6JmwROBGZmTW6iRZDo+k4EZmZNTi38+KiZ\nmR0GStwkcCIwM2tySvz4qBOBmVmzc9eQmVmxTTwz5K4hM7NiKsldQ2ZmheanhszMCm6iReCuITOz\nYnKLwMys4NwiMDMrOA8Wm5kV3PgKZWNuEZiZFZMmuoZabIUyAElrJG2VtE3SFVOcf6+kH0i6Q9K3\nJK1OGY+ZWStq2TWLJZWB9cD5wGrgkim+6L8QEadFxKuAjwGfSBWPmVmrGh8jaMWuoTOBbRGxPSKG\ngI3ARbUFIuL5mt35pEt4ZmYtK/Vgcco1i5cCD9fs7wDOmlxI0vuAPwAqwM9PdSFJa4G1AD09PVSr\n1WlvPDAwMGOZucj1LhbXuzgefH4UgMHBPUnqnnTx+tmIiPXAekm/Dvwx8M4pymwANgD09vZGX1/f\ntNesVqvMVGYucr2LxfUujnseeR7+85t0dHYmqXvKrqGdwPKa/WX5sYPZCPxywnjMzFpSKf+mbsUx\ngs3AKkkrJVWAi4H+2gKSVtXs/iLwo4TxmJm1pJYdI4iIEUnrgOuBMnBtRNwt6SpgS0T0A+sknQcM\nA88wRbeQmVnRTTw+migTJB0jiIhNwKZJx66s2X5/yvubmc0NnmvIzKzQPPuomVnBjS9V6URgZlZQ\n43MNpeJEYGbW5CZaBK046ZyZmR26xA0CJwIzs2YnvDCNmVmhKfF7BE4EZmYF50RgZtbk/B6BmVnB\nyYvXm5kV28RDQx4jMDMrJncNmZkVnPCbxWZmhebHR83MCs6TzpmZFVxHe5lfPO04julK00WUNBFI\nWiNpq6Rtkq6Y4vwfSLpH0p2SbpB0fMp4zMxa0RHz2ll/6Ws4dUmatcSSJQJJZWA9cD6wGrhE0upJ\nxW4HeiPidOArwMdSxWNmZlNL2SI4E9gWEdsjYgjYCFxUWyAiboyI3fnurcCyhPGYmdkUlGp+a0lv\nBdZExOX5/juAsyJi3UHKfxL4SUT8+RTn1gJrAXp6es7YuHHjtPceGBigu7v7EGvQelzvYnG9i+dQ\n6n7uuefeFhG9U51Lunj9bEl6O9ALvH6q8xGxAdgA0NvbG319fdNer1qtMlOZucj1LhbXu3hS1T1l\nItgJLK/ZX5Yf24+k84APA6+PiL0J4zEzsymkHCPYDKyStFJSBbgY6K8tIOnVwP8GLoyIxxPGYmZm\nB5EsEUTECLAOuB64F7guIu6WdJWkC/Nifwl0A1+WdIek/oNczszMEkk6RhARm4BNk45dWbN9Xsr7\nm5nZzJI9NZSKpCeAB2cotgR4sg7hNBvXu1hc7+I5lLofHxFHT3Wi5RLBbEjacrDHpOYy17tYXO/i\nSVV3zzVkZlZwTgRmZgU3VxPBhkYH0CCud7G43sWTpO5zcozAzMxmb662CMzMbJacCMzMCq6lE8Es\nFr7pkPSl/Px3JK2of5SHX1EX/Jmp3jXl3iIpJM2JRwxnU29Jb8v/zu+W9IV6x5jCLP47f6mkGyXd\nnv+3fkEj4jzcJF0r6XFJdx3kvCT9bf7ncqek1xzyTSOiJT9AGfgxcAJQAb4PrJ5U5r8B/5BvXwx8\nqdFx16ne5wJd+fZvF6XeebkFwM1k61v0NjruOv19ryJb5OnIfP+YRsddp3pvAH47314NPNDouA9T\n3V8HvAa46yDnLwD+H9lSxmcD3znUe7Zyi2DGhW/y/c/m218B3iApzaKf9VPUBX9m8/cN8GfAXwB7\n6hlcQrOp928C6yPiGYCYGxM4zqbeASzMt48AHqljfMlExM3A09MUuQj4XGRuBRZJOu5Q7tnKiWAp\n8HDN/o782JRlIpsE7zngqLpEl85s6l3rPWT/emh1M9Y7byIvj4iv1TOwxGbz930icKKkb0u6VdKa\nukWXzmzq/RHg7ZJ2kM1p9jv1Ca3hXux3wIyaYmEaS2OmBX/mEkkl4BPAZQ0OpRHayLqH+shafzdL\nOi0inm1oVOldAnwmIj4u6bXA5yWdGhFjjQ6s1bRyi2A2C99MlJHURtZ8fKou0aXzYhf8uTDmxoI/\nM9V7AXAqUJX0AFnfaf8cGDCezd/3DqA/IoYj4n7gPrLE0MpmU+/3ANcBRMQtQCfZpGxz3ay+A16M\nVk4EMy58k++/M99+K/CNyEdbWlhRF/yZtt4R8VxELImIFRGxgmxs5MKI2NKYcA+b2fx3/i9krQEk\nLSHrKtpezyATmE29HwLeACDpZLJE8ERdo2yMfuA38qeHzgaei4hHD+WCLds1FBEjksYXvikD10a+\n8A2wJSL6gWvImovbyAZfLm5cxIfHLOtdu+APwEMRceFBL9oCZlnvOWeW9b4eeKOke4BR4IMR0dIt\n31nW+wPA/5H0+2QDx5fNgX/oIemLZIl9ST7+8SdAO0BE/APZeMgFwDZgN/CuQ77nHPhzMzOzQ9DK\nXUNmZnYYOBGYmRWcE4GZWcE5EZiZFZwTgZlZwTkRmE0iaVTSHZLukvRVSYsO8/Uvk/TJfPsjkv7w\ncF7f7MVyIjA70GBEvCoiTiV7/+R9jQ7ILCUnArPp3ULNhF6SPihpcz4P/J/WHP+N/Nj3JX0+P/Zf\n83Uwbpf0dUk9DYjfbEYt+2axWWqSymRTGFyT77+RbA6fM8nmgu+X9Dqy+av+GPjZiHhS0uL8Et8C\nzo6IkHQ58Edkb8OaNRUnArMDzZN0B1lL4F7gP/Ljb8w/t+f73WSJ4ZXAlyPiSYCIGJ9LfhnwpXyu\n+Apwf33CN3tx3DVkdqDBiHgVcDzZv/zHxwgEfDQfP3hVRLw8Iq6Z5jr/C/hkRJwG/BbZpGhmTceJ\nwOwg8lXefhf4QD6N+fXAuyV1A0haKukY4BvAr0o6Kj8+3jV0BPumB34nZk3KXUNm04iI2yXdCVwS\nEZ/Ppzu+JZ/VdQB4ez4r5tXATZJGybqOLiNbQevLkp4hSxYrG1EHs5l49lEzs4Jz15CZWcE5EZiZ\nFZwTgZlZwTkRmJkVnBOBmVnBORGYmRWcE4GZWcH9f9UC3jC1lr1JAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "tags": [] }, "output_type": "display_data" } ], "source": [ "plt.plot(recalls[:-1],precisions[:-1])\n", "plt.plot(res_recall_train_score,res_precision_train_score, 'ro')\n", "plt.xlabel(\"Recall\")\n", "plt.ylabel(\"Precision\")\n", "plt.grid()\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 53 }, "colab_type": "code", "id": "CrpoSnJXrMSe", "outputId": "3c856fbb-73d9-4e54-9b52-2868493d1795" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "precision: 0.99 recall: 0.87\n", "Threshold recall: 0.51\n" ] } ], "source": [ "# Set Precision\n", "chosen_recall = 0.87\n", "threshold_chosen = thresholds_pr[np.argmax(recalls <= chosen_recall)]\n", "Y_train_pred_new = (Y_scores >= threshold_chosen) # Set True if Y_scores > threshold_precision\n", "res_precision_train_score_new = precision_score(Y_train, Y_train_pred_new)\n", "res_recall_train_score_new = recall_score(Y_train, Y_train_pred_new)\n", "print(\"precision: \", np.round(res_precision_train_score_new,2), \"recall: \", np.round(res_recall_train_score_new,2))\n", "print(\"Threshold recall: \",threshold_chosen)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "cgUOq2jB14kj" }, "source": [ "### Roc Curve threshold analysis" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "jll_EoUtyQVK" }, "source": [ "| Risultato Esatto | Risulato Predetto | |\n", "|------------|---------------------------|--|\n", "| | Negativo | Positivo |\n", "|Negativo | TN | FP |\n", "|Positivo | FN | TP |\n", "\n", "\n", "* False Positive Rate $FPR = \\frac{FP}{FP+TN}$\n", "* True Positive Rate (or recall) $TPR = \\frac{TP}{TP+FN}$\n", "* True Negative rate (or specificity): $TNR = \\frac{TN}{FP+TN}$\n" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 351 }, "colab_type": "code", "id": "39-qRUCZSJXj", "outputId": "27bbf478-191b-44c8-9436-8a4df1bc4ec3" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Binary classifier ROC AUC score: 0.9983436731328145\n", "Threhsold chosen : 0.34\n", "[[54362 217]\n", " [ 309 5112]]\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3dd3wU5fb48c8htEsRERCR3iGF3jTS\nBEEQQRAUpQi/AIqCePGKKCqKgIIFBUGaFBHFckVRERUuEfCLKJcmVZAuXDqB0JOc3x+zCSEk2U3Z\nbDZ73q/XvLIz88zMebLJnp1nZp5HVBVjjDGBK5evAzDGGONblgiMMSbAWSIwxpgAZ4nAGGMCnCUC\nY4wJcLl9HUBaFS9eXCtUqJCubc+dO0fBggUzN6BszuocGKzOgSEjdf7vf/97XFVLJLfO7xJBhQoV\nWLt2bbq2jYyMpEWLFpkbUDZndQ4MVufAkJE6i8i+lNZZ05AxxgQ4SwTGGBPgLBEYY0yAs0RgjDEB\nzhKBMcYEOK8lAhGZJSJHRWRzCutFRCaKyC4R2SQi9bwVizHGmJR584xgDnB3KuvbAVVd0wDgfS/G\nYowxJgVee45AVVeISIVUinQCPlSnH+xfReRGESmlqoe9FVNWi4sDEWcC2L8fDh2CM2fg7Fnn55kz\ncP48lC4NvXs75S5dgueeg5iYq1NsLKg6U//+cPvtTtklS2D+fOd1/Pr4nsXz5oU+fa7G89RTcPDg\nteXip44dISLCKbd5Mwwbdv3+4s2YAeXKOa/HjYOlS68tE/86LAzeecd5ffEitGt3fZl4zz8Pbdo4\nrz//HCZNSv53mjevc7x4vXrBviR3R0dF1aFIEejaFZ580lm2cSMMHpz8PgE+/BDin1McPRp+/DH5\ncrVqwXvvOa8vXYK77kp5nyNGQNu2mVOneCnVKb7OOalOyUlcp3nzyvPSS8mX89c6pfQ+xcaeo2HD\nY9x3X8r7yQhfPlBWGjiQaP6ga9l1iUBEBuCcNVCyZEkiIyPTdcDo6Og0basKUVF5OHkyLydP5uXM\nmdycO5eb8uXPU6tWFAC7dhVi6tRKnDuXm/Pngzh/PjcXLgRx+XIurlzJxZw5v1G+/HkAXn21Jv/5\nT8lkjxUWdppy5TYAcOlSLiZMaJZiXKVKbePy5SMAfPNNGT76qEqy5fLmjaVr16t1XriwIfv3J/9U\nYp48B6lceRcAGzcW4fvv66Z4/OXLr9Zp2bIaLF16S7Lljh07TWSkU6eLF3MRGZlynW6/fRt58zp1\n+vnnMqxcmXKdIiNXJsyvXNmQffuS1ulGAG6++SC1ajl12rSpCCtXplynyMjfqFDBqdOKFTVYuTL5\nOp0+fbVOly7lYuXKlOv088/byJcvs+qEmzrdmAPrdL3Eddq9uworVyZfzl/rlPz79B+gP5s2FeTO\nO0en+/MvNeLNgWlcZwTfqmpoMuu+BV5X1VWu+WXAs6qa6mPDDRo0UG88WRwTA7kTpcXmzeHXX+Hy\n5evLDhwIU6Y4r3//HRo1SvmY69ZBXdffwCuvwOLFcMMNULiw8/OGG6BAAahUCQYMcMrFxsK77zrx\nxE+5cl09u2jWDKq4/la3boW1a6+edcSXEYGgIChZ8mqdlyxxzkQSl4mfKld2vsEDnDwJq1dfXyZe\neDgUKuS8/uMP+N//rh47nggUKQINGlyt04oV1/5uEpevUQNucf39HzwIu3cn//vMlQvuuOPq/Nq1\nzhlVYuvXr6du3bqULu3UCyAqyvlmlpL69SH+yf3t2+Ho0eTL3XAD1KlztU6//JLyPqtXh5IlM6dO\n8VKqU3ydc1KdkpO4Th9++BsVKiT/z+evdUr8Pp09e5r333+G776bSenSVRg7diblymlGniz+r6o2\nSHalqnptAioAm1NYNw14KNH8DqCUu33Wr19f02v58uXXzMfFqX7/vWr//qolSqieO3d1XZs2TqPI\njTeqBgertmyp2rWrakSE6ocfXi135ozqkiWqq1erbt6sun+/6okTqufPq8bGpjvUTJO0zoHA6hwY\ncnKdY2JitGbNmporVy4dNmyYnj9/XlUzVmdgrabwuerLpqFFwCARWQA0BqI0C68PHDoEXbrAmjVX\nl23fDvVc9y7NmeN8q3DXv1PhwlfbF40xJiNOnDjBTTfdRFBQEGPGjKFs2bI0aJD8l/jM5M3bRz8B\nVgPVReSgiESIyGMi8piryGJgN7ALmAE87q1Ykrp0Ce67z0kCpUrBqFHw3/9eTQLgLA+wjg2NMT6i\nqnz00UdUq1aNmTNnAtC5c+csSQLg3buGHnKzXoEnvHX81Dz/vNO2X768085XvLgvojDGGDhw4ACP\nPfYYixcvpkmTJoSHh2d5DAH3ZPHRozB5svP6888tCRhjfOeTTz4hJCSEyMhI3nnnHVatWkVwcHCW\nx+F34xFkVPHi8M03sH49NGzo62iMMYGsaNGiNG7cmOnTp1OxYkWfxRFwiSBXLufhktQeMDHGGG+I\niYlhwoQJXL58mREjRnD33XfTtm1bJPG91D4QUE1DNy9d6jzClyuX8zP+kVxjjPGyjRs30qRJE4YN\nG8amTZvib5v3eRKAQEoE8+dTdfxbzjPhqs7PAQMsGRhjvOrSpUu8+OKLNGjQgAMHDvD555+zYMGC\nbJEA4gVOIhgxgjxXLl677Px5p6MRY4zxkp07dzJu3Dgefvhhtm7dSteuXbNVEoBAukawf3/alhtj\nTDpFR0fz9ddf06NHD0JDQ9m+fTuVKlXydVgpCpwzgvjuMj1dbowx6fDTTz8RFhZGr1692LZtG0C2\nTgIQSIlgzBjOyz+uXVagAIwZ45t4jDE5yqlTp4iIiKBNmzbkzZuXn3/+mZo1a/o6LI8ETiLo0YMn\n87/PXsqjIs5jxdOnQ48evo7MGOPnYmNjCQ8PZ+7cuTz33HNs3LiRpk2b+josjwXMNYLYWPjgwiPM\nkke4csXpotkYYzLi+PHjCZ3EjR07lnLlylGvnv+NuhswZwQxMRAefpzmzS0JGGMyRlX58MMPr+kk\n7r777vPLJAABlAjy5YPRozezfLmvIzHG+LN9+/bRrl07HnnkEWrWrEmzZimPfuYvAiYRGGNMRn30\n0UeEhoayatUqJk2axMqVK6lRo4avw8qwgLlGcOkSnDqVh7NnncFkjDEmrUqUKEF4eDjTpk2jfPny\nvg4n0wTMGcHPP0OXLuF06eLrSIwx/uLKlSu8/vrrvPrqqwC0bduW77//PkclAQigRBAd7fyMH3Td\nGGNSs379eho3bsxzzz3H1q1bs1UncZktYBLBhQvOzwIFfBuHMSZ7u3jxIs8//zwNGzbk0KFD/Pvf\n/+aTTz7JkQkgXsAkgkuXnJ958/o2DmNM9rZr1y7efPNNevfuzbZt2+gSAO3JAXOx+OxZ56c1DRlj\nkoqOjmbhwoX06tWL0NBQduzY4dMRw7JaqolARDqmtl5VF2VuON5z+bLzM18+38ZhjMlefvjhBwYM\nGMCBAwdo0KABNWvWDKgkAO7PCLqlsk4Bv0kEV644P/Pk8W0cxpjs4cSJEwwdOpQPP/yQGjVqsHLl\nSr/pJC6zpZoIVLVXVgXibfffD7Gxm+ncOdTXoRhjfCy+k7hdu3YxYsQIXnjhBfLnz+/rsHzGXdPQ\nk6mtV9WJmRuO91SvDk2bHifU8oAxAevYsWMUK1aMoKAgxo0bR/ny5alTp46vw/I5d3cNlXAzGWNM\ntqeqzJ49m2rVqjFjxgwAOnXqZEnAxV3T0ItZFYi3/ec/MH9+efLkgfBwX0djjMkqe/fuZcCAAfz0\n0080bdqUli1b+jqkbMej20dFJB/QBwgBEhrSVHWAd8LKfD/+CLNmVaRqVUsExgSKefPmMXDgQESE\nKVOm8Oijj5IrV8A8PuUxT38jHwIVgA7AGqAycNFLMXlFbKzz0/4GjAkcJUuWpFmzZmzZsoWBAwda\nEkiBpw+UVVPVB0XkHlX9QEQ+BFZ6M7DMFp8IbFAaY3KuK1euMH78eGJjY3nppZdo06YNbdq08XVY\n2Z6n6dF1Fz6nRaQmUBi42TsheYclAmNytnXr1tGwYUNeeOEFduzYkdBJnHHP00TwgYgUBUYCPwB/\nAm95LSoviItzftqZoTE5y4ULFxg+fDiNGjXiyJEjLFy4kPnz5+foTuIym0cfi6o6TVVPqepyVS2n\nqsVVdbK77UTkbhHZISK7RGR4MuvLichyEVkvIptEpH16KuGJ+C8H9rdhTM6ye/du3n77bfr06cPW\nrVu57777fB2S3/EoEYjIqyJyY6L5oiLyipttgoDJQDsgGHhIRIKTFHsB+ExV6wLdgSlpCT4tChWC\nG2+8bN1QG5MDnDlzhiVLlgAQEhLCzp07mTlzJkWLFvVxZP7J04aSDqp6On5GVU8B97rZphGwS1V3\nq+plYAHQKUkZBW5wvS4CHPIwnjR7/XVYuPD/iIjw1hGMMVlh8eLFhIaG8sYbb7Bt2zaAHDdiWFbz\n9K6hIBHJ6/pAR0TyA+569i8NHEg0fxBonKTMy8CPIjIYKAi0Tm5HIjIAGADO7WCRkZEehn2t6Ojo\ndG/rr6zOgSEQ6hwVFcXkyZP56aefKF++POPGjePIkSMcOXLE16FlGa+9z6rqdgKeB34GHnFNPwPP\nudmmKzAz0Xwv4L0kZYYCT7te3wZsBXKltt/69etrei1fvjzd2/orq3NgyOl1jomJ0WrVqmnu3Ln1\npZde0osXL+b4OicnI3UG1moKn6senRGo6lgR2cTVb+zjVfU7N5v9DZRNNF/GtSyxCOBu1zFWu840\nigNHPYkrLZ59FubMacKECfDww5m9d2OMNxw5coQSJUoQFBTEm2++Sfny5alVq5avw8px0nIz5Qbg\na1V9ClgmIgXdlP8dqCoiFUUkL87F4KTjF+wHWgG4nk/IDxxLQ0weO3YMjh7NnzB2sTEm+1JVPvjg\nA6pXr8706dMBuPfeey0JeImndw39P5wP8ZmuReWAr1PbRlVjgEE4zx1sw7k7aIuIjEo08tnTQH8R\n2Qh8AvRxncJ4jd0+akz2tnv3blq3bk2/fv2oU6cOrVsne+nQZCJPLxY/iXMX0BoAVf1TRNw+Wayq\ni4HFSZa9lOj1ViBLuoCzhwyNyf7mzp3L448/TlBQEFOnTqV///7WP1AW8DQRXFTVy/FP6rmeEfCr\n79b2QJkx2d+tt97KnXfeyfvvv0+ZMmV8HU7A8DQR/CIiw4D8ItISeAL41nthZT5LBMZkP5cvX+b1\n118nLi6Ol19+mbvuuou77rrL12EFHE/PuYYBZ4HtwBBgGTDCW0F5kyUCY7KH33//nfr16zNy5Eh2\n795tncT5kNszAlcz0GxV7Q287/2QvOPeeyEubh9hYfYEojG+dP78eV566SUmTJhAqVKlWLRoEffe\n666jAuNNbhOBqsaKSCURyaOqV9yVz666doXixfdQr54lAmN8ac+ePUyaNIn+/fszbtw4ihQp4uuQ\nAp6n1wj+AlaKyNfAufiFqjrRK1EZY3KUqKgovvzyS/r27UtISAi7du2ibNmy7jc0WcLTawT7gZ+A\nAkCJRJPf2LgR1qy5ib+TPttsjPGq7777jpCQEPr168f27dsBLAlkM552MfGitwPxtjffhI8+qsWt\nt0KvXr6Oxpic79ixYzz11FN8/PHHhIaG8uWXX1KjRg1fh2WS4WnTkN+zGxKMyTqxsbHccccd7Nmz\nh1deeYXhw4eTN6+7DouNrwRMIohnt48a4z3/+9//uPnmmwkKCuKtt96iQoUKhIaG+jos44Y9u22M\nybC4uDimTZtGtWrVmDZtGgAdOnSwJOAnPHmOIC/QHmgK3ApcADYD36nqDu+Gl3msacgY79i1axf9\n+/cnMjKSO++8k7Zt2/o6JJNGqZ4RiMiLOB3NtQQ2AnNxeiHNDbwjIktExC9SvnUxYUzmmz17NmFh\nYaxbt44ZM2awdOlSKlWq5OuwTBq5OyPYpKqvprBuvIiU4trBZ7Kt+ERgHRkak3nKlStH27ZtmTx5\nMqVLl/Z1OCadUk0EqprimAMiUlpV/wYOZ3pUXvDee3Dffavp0OE2X4dijN+6dOkSr732GnFxcYwa\nNYpWrVrRqlUrX4dlMsjt92MRaSgi94lIcdd8iIh8iGtsAn9RrBiULHmJgu7GVTPGJGvNmjXUr1+f\nV155hf3791sncTmIu2sErwHzgR7AEhF5GViOc72gmtejM8b43Llz5xg6dCi33XYbUVFRfPvtt8yZ\nMwexC245hrtrBJ2A2qp6QURuAg4AYaq62/uhZa433oBFi0J4/XUIz5Ix0YzJGfbt28eUKVN47LHH\neP3117nhhht8HZLJZO4SwUVVvQCgqidF5E9/TAIAa9bAqlUlOHTI15EYk/2dPn2aL774gn79+hEc\nHMyuXbtsxLAczF0iqCQiX7peC1Ax0Tyq2sVrkXmJnc0ak7qvv/6agQMHcvToUe644w5q1KhhSSCH\nc5cI7k8y/563AjHG+NbRo0d58skn+fTTT6lVqxaLFi2yTuIChLvbR5eJSBhQGdiiqjuzJixjTFaK\njY0lPDyc/fv3M3r0aIYNG0aePHl8HZbJIqkmAhF5HogA1gENRWSUqs7KksiMMV536NAhbrnlFoKC\ngnj33XepUKECwcHBvg7LZDF3zxH0AGqpajegITDQ+yEZY7wtLi6O999/nxo1ajB16lQA2rdvb0kg\nQLm7RnBJVc8BqOoxEfHbDhrq1oUDB05QsmQxX4dijE/9+eef9O/fnxUrVtC6dWvatWvn65CMj6X1\nrqHK/nrX0IgREB7+B02btvB1KMb4zAcffMCgQYPInz8/s2bNok+fPvZgmLG7howJJBUqVKBdu3ZM\nnjyZUqVK+Tock024SwQPq2pElkTiZVFRcPp0Hi5dgnz5fB2NMVnj0qVLvPqq04Hw6NGjrZM4kyx3\nbf51sySKLBARAZ07h/PNN76OxJis8X//93/UqVOHMWPGcPjwYeskzqTI3RlBAddzBMk2IqrqpswP\nyTvsf8AEiujoaEaMGMGkSZMoW7YsS5YssVHDTKrcJYLSwGSSTwQKNEttYxG5G3gXCAJmqurryZR5\nAHjZtb+Nqvqw+7DTzkYoM4Fi//79TJs2jSeeeIKxY8dSuHBhX4dksjl3iWCXqqb6YZ8SEQnCSSJ3\nAQeB30VkkapuTVSmKvAcEK6qp0Tk5vQcK21xefsIxmS9s2fPMn36dAYMGEBwcDC7d+/m1ltv9XVY\nxk+4Hbw+AxrhJJLdACKyAKdb662JyvQHJqvqKQBVPerFeIzJkRYuXEi/fv2IioqiefPmVK9e3ZKA\nSRN3ieD5DOy7NM74BfEOAo2TlKkGICK/4DQfvayqS5LuSEQGAAMASpYsSWRkZJqDOX48BCjB5s2b\nuemm42ne3l9FR0en6/flzwKlzidPnmTixIn8/PPPVKpUiddee43Dhw9z+LBfjB6bYYHyPifmrTq7\nSwSPuh42+UlVYxKvEJHywCPAwQz0P5QbqAq0AMoAK0QkTFVPJy6kqtOB6QANGjTQFi1apPlAxYs7\nP0NDQ0nH5n4rMjKS9Py+/Fkg1Dk2NpYaNWpw4MABxo4dS8OGDWndurWvw8pSgfA+J+WtOrtLBE8A\nTwOTReQIcAzID1TE+bY/WVX/ncK2fwNlE82XcS1L7CCwRlWvAHtE5E+cxPB7mmrhgWHDoG7dzTRp\nEprZuzYmyxw8eJBbb72VoKAgJk6cSMWKFalRo0bAfTM2mSvV5whU9W9VHaqqlYBewBs4zUX1VPXO\nVJIAOB/mVUWkoojkBboDi5KU+QrnbAARKY7TVOSVEdCaNIGmTY9jTafGH8XFxTFp0iRq1KjB+++/\nD0C7du1svACTKTy+WKyqu4BdaSgfIyKDgB9w2v9nqeoWERkFrFXVRa51bURkKxALPKOqJ9JUA2Ny\nuO3bt9OvXz9++eUX2rZtS4cOHXwdkslhvHnXEKq6GFicZNlLiV4rMNQ1edWnn8IPP5SnVCmoXt3b\nRzMmc8ycOZNBgwZRoEAB5s6dS69evayTOJPp/LZb6bT6+GOYPbsi27f7OhJjPFe5cmXuvfdetm3b\nRu/evS0JGK/w+IzA1c5fztVEZIzxgosXLzJq1CgAxo4dS8uWLWnZsqWPozI5nUdnBCJyD/AH8JNr\nvo6ILPRmYMYEml9++YU6derw2muvcezYMeskzmQZT5uGRuE8DHYaQFU3AFW8FZQxgeTs2bMMHjyY\npk2bcunSJX744QdmzJhhzUAmy3iaCK4kfcgLp5M4Y0wGHTx4kJkzZzJ48GD++OMP2rRp4+uQTIDx\n9BrBNlcvoblEpCLwJPCr98IyJmc7ceIEn332GQMHDqRmzZrs3r3bRgwzPuPpGcEgoD4QB3wJXAKG\neCsob7jhBihS5DJ58/o6EhPIVJUvvviC4OBgnnzySXbs2AFgScD4lKeJoK2qPquqdV3TcKCdNwPL\nbPPmwVdf/R/t/Cpqk5McPnyY+++/n27dulG2bFnWrl1LdXuoxWQDniaCF5JZNiIzAzEmJ4uNjaVp\n06Z8//33jB8/nl9//ZXatWv7OixjADfXCESkLXA3UFpE3k606gacZiJjTCoOHDhA6dKlCQoKYvLk\nyVSsWJFq1ar5OixjruHujOAosBm4CGxJNP2InzUNRUTAgw824ccffR2JCQSxsbFMnDjxmk7i2rZt\na0nAZEupnhGo6npgvYjMV9WLWRSTVxw/DkeP5ufCBV9HYnK6bdu2ERERwerVq2nXrh333nuvr0My\nJlWeXiMoLSILRGSTiPwZP3k1MmP80PTp06lTpw5//vkn8+bN47vvvqNcuXK+DsuYVHmaCOYAswHB\naRL6DPjUSzF5hT2tb7JC1apV6dy5M1u3bqVnz572dLDxC54mggKq+gOAqv6lqi/gZ9cI4tn/pclM\nFy5c4Nlnn2X48OEAtGzZkgULFnDzzTf7ODJjPOdpIrgkIrmAv0TkMRG5FyjsxbgynZ0RmMy2YsUK\nateuzfjx44mKirJO4ozf8jQR/BMoiNO1RDjQH/h/3grKm+yMwGTUmTNnePzxx2nevDmxsbEsW7aM\n999/35qBjN/yqK8hVV3jenkWZ+xiRKS0t4LyhgcegBtv3EfVquV9HYrxc4cOHWLOnDkMHTqUUaNG\nUbBgQV+HZEyGuE0EItIQKA2sUtXjIhICPAvcCZTxcnyZplcvKFt2DzVqWCIwaXf8+HE+++wzHn/8\ncWrUqMGePXsoWbKkr8MyJlOk2jQkIq8B84EewBIReRlYDmwE7MkYk+OpKp9++inBwcE89dRT/Pmn\nc9e0JQGTk7g7I+gE1FbVCyJyE3AACFPV3d4PLXOtXQtr1txEcDDYDR3GE4cOHWLgwIEsWrSIBg0a\nsGzZMnsy2ORI7hLBRVW9AKCqJ0XkT39MAgAjR8LixbUIDYV77vF1NCa7i42NpVmzZvz999+8+eab\nDBkyhNy5PR7i2xi/4u4vu5KIfOl6LUDFRPOoahevRWaMD+zbt48yZcoQFBTElClTqFSpElWq2Kis\nJmdzlwjuTzL/nrcCMcaXYmNjeffdd3nhhRcYP348gwYNsiEjTcBw1+ncsqwKxBhf2bx5MxEREfz2\n22906NCB++67z9chGZOlPH2gzJgcaerUqdSrV4/du3fz8ccfs2jRIsqU8Zu7oo3JFJYITECK7w6i\nZs2adOvWja1bt/LQQw/Z08EmIKXpNggRyaeql7wVjDHedv78eV566SWCgoIYN24czZs3p3nz5r4O\nyxif8uiMQEQaicgfwE7XfG0RmeTVyDLZ7NmwYMFq7rzT15EYX4mMjKRWrVq89dZbREdHWydxxrh4\n2jQ0EegAnABQ1Y1AS28F5Q033wwlS17iH//wdSQmq0VFRfHoo4/SsqXzJ/uf//yHyZMnWzOQMS6e\nJoJcqrovybLYzA7GGG84fPgwH330Ef/617/YtGlTQkIwxjg8TQQHRKQRoCISJCJPAW6HqhSRu0Vk\nh4jsEpHhqZS7X0RURBp4GE+ajR4NL74Ywtq13jqCyU6OHTvGpElO62WNGjXYu3cvb7zxBgUKFPBx\nZMZkP54mgoHAUKAccARo4lqWIhEJAibjjGQWDDwkIsHJlCsMDAHWJF2XmVavhlWrSnDkiDePYnxN\nVVm6dCk1a9bk6aefTugkrkSJEj6OzJjsy9NEEKOq3VW1uGvqrqrH3WzTCNilqrtV9TKwAKcTu6Re\nBcYBFz0P25jrHThwgHvvvZcxY8ZQpUoV1q9fb53EGeMBT28f/V1EduAMWP+lqp71YJvSOL2VxjsI\nNE5cQETqAWVV9TsReSalHYnIAGAAON3/RkZGehj2VSdOhAHF2LRpEwULnkzz9v4qOjo6Xb8vfxMb\nG0vv3r05efIk/fr1o3v37hw7diwg6g6B8z4nZnXOPJ6OUFZZRG4HugOviMgGYIGqLkjvgV1jIL8N\n9PHg+NOB6QANGjTQFi1apPl4xYo5P2vVqkU6NvdbkZGRpOf35S/27t1L2bJlCQoKYu7cuVSqVIn9\n+/fn6DonJ6e/z8mxOmcej58sVtX/U9UngXrAGZwBa1LzN1A20XwZ17J4hYFQIFJE9uJcd1jkzQvG\nJueIiYnhzTffpGbNmkyZMgWA1q1bU6lSJR9HZoz/8eiMQEQK4bTvdwdqAl8Dt7vZ7HegqohUxEkA\n3YGH41eqahRQPNExIoF/qard12NStWnTJiIiIli7di2dOnXi/vuTdpJrjEkLT68RbAa+Acar6kpP\nNlDVGBEZBPwABAGzVHWLiIwC1qrqonRFnE7168Px4ycoUaJYVh7WZLIpU6YwZMgQihYtyqeffkq3\nbt3swTBjMsjTRFBJVePSunNVXQwsTrLspRTKtkjr/tNi1Ci4884/aNTIq4cxXqKqiAihoaF0796d\nCRMmULx4cfcbGmPcSjURiMhbqvo08G8Rua5jFhuhzHjbuXPneOGFF8idOzdvvPEGzZo1o1mzZr4O\ny5gcxd0Zwaeun34/MllUFJw+nYdLlyBfPl9HYzyxbNky+vfvz549exg8eHDCWYExJnOleteQqv7m\nellTVZclnnAuGvuNhx+Gzp3DWbrU15EYd06fPk2/fv1o3bo1uXPnZsWKFUycONGSgDFe4unto/8v\nmWURmRmIMfGOHDnCggULeGRgbgIAAB78SURBVPbZZ9m4cSNNmzb1dUjG5GjurhE8iHPbZ0UR+TLR\nqsLAaW8GZgJL/If/kCFDqF69Onv37rWLwcZkEXfXCH7DGYOgDE4HcvHOAuu9FZQJHKrK/PnzGTJk\nCNHR0bRv356qVataEjAmC6WaCFR1D7AHsJZ1k+n279/PY489xvfff89tt93GBx98QNWqVX0dljEB\nx13T0M+q2lxETgGJbx8VQFX1Jq9GZ3KsmJgYWrRowdGjR5k4cSKPP/44QUFBvg7LmIDkrmkofign\nO083mWL37t2UL1+e3LlzM2PGDCpXrkyFChV8HZYxAc3d7aPxTxOXBYJUNRa4DXgUKOjl2DLViBEw\natRm6tf3dSSBKSYmhnHjxhEcHMzkyc7lplatWlkSMCYb8PT20a9whqmsDMwGqgIfey0qL7j9dmja\n9Di33OLrSALPhg0baNy4McOHD6d9+/Z069bN1yEZYxLxNBHEqeoVoAswSVX/iTPwjDGpeu+992jY\nsCF///03X3zxBV9++SWlSpXydVjGmEQ8HqpSRLoBvYBvXcvyeCck71iwAObNK8/Onb6OJDCoOvcW\n1KpVix49erB161brLtqYbCotTxa3xOmGerdrjIFPvBdW5ps3D2bNqohrLHPjJdHR0QwZMoRnnnFG\nHm3WrBlz5szhppvsBjNjsiuPEoGqbgaeBNaKSA3ggKqO8Wpkxu/8+OOPhIaGMmnSJK5cuZJwVmCM\nyd48SgQi0hTYBXwAzAL+FJFwbwZm/MepU6fo27cvbdu2JX/+/KxYsYJ3333XOokzxk942jQ0AWiv\nquGqejtwD/Cu98Iy/uTo0aN88cUXPPfcc2zYsIE77rjD1yEZY9LA0xHK8qrq1vgZVd0mInm9FJPx\nA//73//45JNP+Oc//5nQSVyxYjYMqDH+yNMzgnUiMlVE7nBN72OdzgUkVWXu3LkEBwfz3HPPsdN1\nG5YlAWP8l6eJ4DFgNzDMNe3GebrYb9xwAxQpcpm8dh6Tbnv37uXuu++mT58+BAcHs2HDBuskzpgc\nwG3TkIiEAZWBhao63vsheccnn0Bk5P/RokULX4fil2JiYmjZsiXHjx9n8uTJPPbYY+TK5en3CGNM\nduau99HncUYiWwc0FJFRqjorSyIz2cKuXbuoWLEiuXPnZtasWVSqVIny5cv7OixjTCZy95WuB1BL\nVbsBDYGB3g/JZAdXrlxh7NixhISEJHQS17JlS0sCxuRA7hLBJVU9B6Cqxzwon2316QMPPtiEZct8\nHUn2t27dOho1asSIESPo1KkTDz74oK9DMsZ4kbtrBJUSjVUsQOXEYxerahevRZbJjh2Do0fzc/Gi\nryPJ3iZOnMjQoUMpUaIEX375JZ07d/Z1SMYYL3OXCJL2EvaetwIxvqWqiAh169ald+/evPXWWxQt\nWtTXYRljsoC7MYutISWHO3v2LM899xz58uXjrbfeomnTpjRt2tTXYRljspDftvmbjFuyZAmhoaFM\nmTIFVbVO4owJUJYIAtCJEyd45JFHaNeuHQULFuSXX37h7bfftk7ijAlQaUoEIpLPW4GYrHPixAkW\nLlzIiy++yPr167ntttt8HZIxxoc87Ya6kYj8Aex0zdcWkUkebHe3iOwQkV0iMjyZ9UNFZKuIbBKR\nZSLitZvUH3wQevbcR5Uq3jpC9nb48GHefPNNVJVq1aqxb98+Ro0aRb58ltuNCXSenhFMBDoAJwBU\ndSPOiGUpEpEgYDLQDggGHhKR4CTF1gMNVLUW8AXgtS4seveGiIg9VK/urSNkT6rKrFmzqFmzJi++\n+CK7du0CsDuCjDEJPE0EuVR1X5JlsW62aQTsUtXdqnoZWAB0SlxAVZer6nnX7K9AGQ/jMR7Ys2cP\nzzzzDBEREdSuXZuNGzdaJ3HGmOt4Oh7BARFpBKjrm/5gwN3ov6WBA4nmDwKNUykfAXyf3AoRGQAM\nAChZsiSRkZEehn3Vjh2FOXz4H5w69QtFi15J8/b+JjY2lp49exIVFcU///lPOnTowKFDhzh06JCv\nQ/O66OjodP2N+DOrc2DwWp3jbxtMbQJuxvlGf9w1LQCKu9mmKzAz0Xwv4L0UyvbEOSPI5y6W+vXr\na3q0b68Kqt9+m67N/caff/6pMTExqqq6fPly/fTTT30cUdZbvny5r0PIclbnwJCROgNrNYXPVU8H\nrz+qqt1Vtbhr6q6qx91s9jdQNtF8Gdeya4hIa2AE0FFVL3kSj7nelStXGD16NKGhobz3nvMAeIsW\nLbj55pt9HJkxJrvzqGlIRGYA1z1tpKoDUtnsd6CqiFTESQDdgYeT7LcuMA24W1WPehq0udbatWuJ\niIhg06ZNdO/enYceesjXIRlj/Iin1wiWJnqdH+jMte3/11HVGBEZBPwABAGzVHWLiIzCOUVZBLwB\nFAI+dz3MtF9VO6axDgHt3XffZejQodxyyy18/fXXdOxovz5jTNp4lAhU9dPE8yIyD1jlwXaLgcVJ\nlr2U6HVrz8I0Samrk7gGDRoQERHB+PHjufHGG30dljHGD3l6RpBURaBkZgZiPHPmzBmeffZZ8ufP\nz4QJEwgPDyc8PNzXYRlj/JinTxafEpGTruk08BPwnHdDM0ktXryYkJAQpk+fTu7cua2TOGNMpvBk\n8HoBanP1jp849cNPoNmzYfny1dx5p//1q3P8+HGeeuop5s+fT0hICF988QWNG6f2SIYxxnjO7RmB\n60N/sarGuia/SwIAN98MJUte4h//8HUkaXfq1Cm++eYbRo4cybp16ywJGGMylafXCDaISF1VXe/V\naEyCv//+m/nz5/PMM89QtWpV9u3bZxeDTbpduXKFgwcPcjEHjdVapEgRtm3b5uswspQndc6fPz9l\nypQhT548Hu831UQgIrlVNQaoC/wuIn8B53DGL1ZVrefxkXxs9Gj44YcQJkyABg18HU3KVJWZM2fy\nr3/9iytXrtClSxeqVKliScBkyMGDBylcuDAVKlTIMeNOnD17lsKFC/s6jCzlrs6qyokTJzh48CAV\nK1b0eL/uzgh+A+oBfn9z+urVsGpVCY4c8XUkKfvrr7/o378/y5cvp0WLFsyYMYMqgdpvtslUFy9e\nzFFJwCRPRChWrBjHjh1L03buEoEAqOpf6Q3MeCYmJoZWrVpx8uRJpk2bRr9+/ciVywaQM5nHkkBg\nSM/77C4RlBCRoSmtVNW303xEc40dO3ZQuXJlcufOzdy5c6lcuTJlylhv3MaYrOPuK2cQThcQhVOY\nTDpdvnyZV155hbCwMCZPngxA8+bNLQkYY7Kcu0RwWFVHqeoryU1ZEmEO9Ntvv1G/fn1efvllunXr\nRo8ePXwdkjFetXfvXkJDQ69b3q9fP7Zu3eqDiJJXoUIFjh9317GyZxYtWsTrr78OwLFjx2jcuDF1\n69Zl5cqVtG/fntOnT2fKcTKDu0RgjYqZ7J133uG2225LeDZg/vz5FC9e3NdhmQAjkvI0ffrVctOn\np142o2bOnElwcNIRbNMuJiYm48Fkso4dOzJ8uDNU+7JlywgLC2P9+vU0bdqUxYsXp+lOwNhYdwNC\nZoy7RNDKq0fPQvXrQ6NGJyhRwjfHj38Or1GjRvTv358tW7bQoUMH3wRjjA/ExMTQo0cPatasSdeu\nXTl//jwtWrRg7dq1ABQqVIgRI0ZQu3ZtmjRpwhHXLX7ffPNNwrfp1q1bJywfO3YsvXr1Ijw8nF69\netGsWTM2bNiQcLw77riDjRs3JhtLdHQ0ffv2JSwsjFq1avHvf//7ujL33Xcf9evXT+jWBZwP5D59\n+hAaGkpYWBgTJkwAYOLEiQQHB1OrVi26d+8OwJw5cxg0aBAbNmxg2LBhfP3119SpU4cLFy5cc+bx\n0Ucf0ahRI+rUqcOjjz6a8KFfqFAhnn76aWrXrs3q1asz/PtPVUoj1mTXKb0jlKn6ZkSj06dP64AB\nA3TIkCFZfmxVG8UpULir89atW7MmkBTs2bNHAV21apWqqvbt21ffeOMNbd68uf7++++qqgrookWL\nVFX1mWee0VdffVVVVU+ePKlxcXGqqjpjxgwdOnSoqqoOHz5c69Wrp+fPn1dV1Tlz5iT8n+3YsUNT\n+6wYNmzYNf+TJ0+eVFXV8uXL67Fjx1RV9cSJE6qqev78eQ0JCdHjx4/r2rVrtXXr1gnbnTp1SlVV\nS5UqpRcvXrxm2ezZs/WJJ5647nXi42zdulU7dOigly9fVlXVgQMH6ty5cxN+H0lHGDxz5kyKdUos\nufebjI5QZtLnm2++ITg4mJkzZ5IvXz7rJM4EtLJlyyb0lNuzZ09Wrbq2J/u8efMmnCXXr1+fvXv3\nAs7DcG3btiUsLIw33niDLVu2JGzTsWNH/uHqN6Zbt258++23XLlyhVmzZtGnT58UY1m6dClPPPFE\nwnzRokWvKzNx4sSEs5MDBw6wc+dOKlWqxO7duxk8eDBLlizhhhtuAKBWrVr06NGDjz76iNy5Pe/U\nedmyZfz3v/+lYcOG1KlTh2XLlrF7924AgoKCuP/++z3eV0YETCKIioLTp/NwKQsGwzx27BgPP/ww\nHTt2pFixYvz666+MGzfO7uM2AS3p33/S+Tx58iQsCwoKSmj3Hzx4MIMGDeKPP/5g2rRp13STUbBg\nwYTXBQoU4K677uLrr7/ms88+y9BNGJGRkSxdupTVq1ezceNG6taty8WLFylatCgbN26kRYsWTJ06\nlX79+gHw3Xff8cQTT7Bu3ToaNmzo8TULVeWRRx5hw4YNbNiwgR07dvDyyy8DTlcRQUFB6a5DWgRM\nInj4YejcOZylS92XzaioqCgWL17MK6+8wtq1a2nYsKH3D2pMNrd///6Etu6PP/6YO+64w6PtoqKi\nKF26NABz585NtWy/fv148sknadiwYbLf8uPdddddCbdtg9OxY9JjFi1alAIFCrB9+3Z+/fVXwOkJ\nOC4ujvvvv5/Ro0ezbt064uLiOHDgAC1btmTcuHFERUURHR3tUd1atWrFF198wdGjzki9J0+eZN++\nfR5tm5kCJhF424EDB3jttddQVapUqcK+fft46aWXyJs3r69DMyZbqF69OpMnT6ZmzZqcOnWKgQMH\nerRd/G3W9evXd3uHXf369bnhhhvo27dvquVeeOEFTp06RWhoKLVr12b58uXXrL/77ruJiYmhZs2a\nDB8+nCZNmgBOZ5AtWrSgTp069OzZk9dee43Y2Fh69uxJWFgYdevW5cknn/T4jqDg4GBGjx5NmzZt\nqFWrFnfddReHDx/2aNtMldLFg+w6pfdicfv2qqD67bfp2jxFsbGx+v7772vhwoW1QIECunPnzsw9\nQAbZhdPAkN0vFntDchdO//77b61atarGxsb6ICLvs4vF2dDOnTu58847GThwII0aNeKPP/6wTuKM\n8ZEPP/yQxo0bM2bMGOunK43SO2ZxwIuJieGuu+7i9OnTfPDBB/Tt29cuBhvjQ71796Z3797XLJs9\nezbvvvvuNcvCw8OvuT5gLBGk2bZt26hatSq5c+dm3rx5VK5cmVtvvdXXYRljktG3b1+31wuMXSz2\n2KVLlxg5ciS1atXivffeA6Bp06aWBIwxfi9gzghGjIAmTTZTv/71HV+58+uvvxIREcHWrVvp1asX\nvXr18kKExhjjGwGTCG6/HS5fPs4tt6Rtu7feeotnnnmGMmXKsHjxYtq1a+edAI0xxkesaSgFcXFx\nANx222089thjbN682ZKAMSZHCphEsGABzJtXnp07Uy93+vRpIiIiGDJkCAC33347U6ZMSehTxBjj\nexUqVODEiRNZfsycOlZBwDQNzZsHixdXpEsXqFo1+TJfffUVjz/+OEePHmXYsGGoqt0SanKkFi1a\nXLfsgQce4PHHH+f8+fO0b9/+uvV9+vShT58+HD9+nK5du16zLjIy0uNjJzzEFMD3+nfs2JGOHTsC\nV8cqmDlzJuDchJIWsbGxGe6TKHDfiUSOHj3KAw88QOfOnSlZsiS//fYbY8eOtSRgTCbZu3cv1atX\np3fv3oSGhhIREUGDBg0ICQlh5MiRCeUqVKjAyJEjqVevHmFhYWzfvh2AEydO0KZNG0JCQujXr981\nPfm+/fbbhIaGEhoayjvvvJNwvBo1atCnTx+qVatGjx49WLp0KeHh4VStWpXffvstxVgDcqyClB45\nzq6TN7qY2Llzp9544406ZsyYhH7BcwrrbiEwZPcuJvbs2aMioqtXr1bVq339x8TEaPPmzXXjxo2q\n6vTTP3HiRFVVnTx5skZERKiq6uDBg/WVV15RVdVvv/1WAd2zZ4+uXbtWQ0NDNTo6Ws+ePavBwcG6\nbt063bNnjwYFBemmTZs0NjZW69Wrp3379tW4uDj96quvtFOnTinGmp3HKpg6daqqJj9WQWLWxYSH\n9u/fz5gxYxI6idu/fz/PP/88efLk8XVoxuRI5cuXT+i87bPPPqNevXrUrVuXLVu2XDNucZcuXYBr\nxyRYsWIFPXv2BOCee+5J6Fl01apVdO7cmYIFC1KoUCG6dOnCypUrAahYsSJhYWHkypWLkJAQWrVq\nhYgQFhaWsN/kZOexCuLjzuyxCryaCETkbhHZISK7RGR4MuvzicinrvVrRKSCN+MB526gKVOmEBIS\nwtixY/nrr78AKFy4sLcPbUxAix87YM+ePbz55pssW7aMTZs2cc8991wzxkC+fPmAa8ckSI/4/QDk\nypUrYT5XrlwZ2q8vxyp4/vnngcwfq8BriUBEgoDJQDsgGHhIRJKOUh0BnFLVKsAEYJy34nHs4Lnn\nWvDEE09w2223sWXLFuskzpgsdubMGQoWLEiRIkU4cuQI33//vdttmjVrxscffwzA999/nzB+QNOm\nTfnqq684f/48586dY+HChWm+2JpUdh6rYP/+/RmqW0q8eddQI2CXqu4GEJEFQCdga6IynYCXXa+/\nAN4TEXG1Z2WqQoViEGnLvn1RzJ49m0ceecQuBhvjA7Vr16Zu3brUqFHjmuErUzNy5EgeeughQkJC\nuP322ylXrhwA9erVo0+fPjRq1AhwBqapW7duqk0/7rzwwgs88cQThIaGEhQUxMiRIxOaq8AZq2Dq\n1KnUrFmT6tWrXzNWQd++fROeQUo8VkFUVBSqmu6xCuLi4siTJw/jx48nJCQk3XVLiXjhM9fZsUhX\n4G5V7eea7wU0VtVBicpsdpU56Jr/y1XmeJJ9DQAGAJQsWbL+ggUL0hXTmjVrqFKlCsWKFUvX9v4o\nOjqaQoUK+TqMLGV1vl6RIkVy3NlvZtw26W88rfOuXbuIioq6ZlnLli3/q6oNkivvF88RqOp0YDpA\ngwYNNLl7oD2VkW39UWRkpNU5ALir87Zt23LcdbCzZ8/muDq542md8+fPT926dT3erzcTwd9A2UTz\nZVzLkitzUERyA0WArH1c0BgTkGysgqu8mQh+B6qKSEWcD/zuwMNJyiwCHgFWA12B/3jj+oAxBntS\nPomcOlZBej5CvXbXkKrGAIOAH4BtwGequkVERolIR1exD4BiIrILGApcd4upMSbj8ufPz4kTJ9L1\nIWH8h6py4sQJ8ufPn6btvHqNQFUXA4uTLHsp0euLQDdvxmCMgTJlynDw4EGOHTvm61AyzcWLF9P8\ngefvPKlz/vz5KVOmTJr26xcXi40xGZMnTx4qVqzo6zAyVWRkZJouiOYE3qpzwHYxYYwxxmGJwBhj\nApwlAmOMCXBee7LYW0TkGLAvnZsXBzJniCH/YXUODFbnwJCROpdX1RLJrfC7RJARIrI2pUescyqr\nc2CwOgcGb9XZmoaMMSbAWSIwxpgAF2iJYLqvA/ABq3NgsDoHBq/UOaCuERhjjLleoJ0RGGOMScIS\ngTHGBLgcmQhE5G4R2SEiu0Tkuh5NRSSfiHzqWr9GRCpkfZSZy4M6DxWRrSKySUSWiUh5X8SZmdzV\nOVG5+0VERcTvbzX0pM4i8oDrvd4iIh9ndYyZzYO/7XIislxE1rv+vtv7Is7MIiKzROSoawTH5NaL\niEx0/T42iUi9DB9UVXPUBAQBfwGVgLzARiA4SZnHgamu192BT30ddxbUuSVQwPV6YCDU2VWuMLAC\n+BVo4Ou4s+B9rgqsB4q65m/2ddxZUOfpwEDX62Bgr6/jzmCdmwH1gM0prG8PfA8I0ARYk9Fj5sQz\ngkbALlXdraqXgQVApyRlOgFzXa+/AFqJf4/Y4bbOqrpcVc+7Zn/FGTHOn3nyPgO8CowDLmZlcF7i\nSZ37A5NV9RSAqh7N4hgzmyd1VuAG1+siwKEsjC/TqeoK4GQqRToBH6rjV+BGESmVkWPmxERQGjiQ\naP6ga1myZdQZQCcK8OcR7T2pc2IRON8o/JnbOrtOmcuq6ndZGZgXefI+VwOqicgvIvKriNydZdF5\nhyd1fhnoKSIHccY/GZw1oflMWv/f3bLxCAKMiPQEGgDNfR2LN4lILuBtoI+PQ8lquXGah1rgnPWt\nEJEwVT3t06i86yFgjqq+JSK3AfNEJFRV43wdmL/IiWcEfwNlE82XcS1LtoyI5MY5nTyRJdF5hyd1\nRkRaAyOAjqp6KYti8xZ3dS4MhAKRIrIXpy11kZ9fMPbkfT4ILFLVK6q6B/gTJzH4K0/qHAF8BqCq\nq4H8OJ2z5VQe/b+nRU5MBL8DVUWkoojkxbkYvChJmUXAI67XXYH/qOsqjJ9yW2cRqQtMw0kC/t5u\nDG7qrKpRqlpcVSuoagWc6yIdVXWtb8LNFJ78bX+FczaAiBTHaSranZVBZjJP6rwfaAUgIjVxEkHO\nGZPzeouA3q67h5oAUap6OCM7zHFNQ6oaIyKDgB9w7jiYpapbRGQUsFZVFwEf4Jw+7sK5KNPddxFn\nnId1fgMoBHzuui6+X1U7+izoDPKwzjmKh3X+AWgjIluBWOAZVfXbs10P6/w0MENE/olz4biPP3+x\nE5FPcJJ5cdd1j5FAHgBVnYpzHaQ9sAs4D/TN8DH9+PdljDEmE+TEpiFjjDFpYInAGGMCnCUCY4wJ\ncJYIjDEmwFkiMMaYAGeJwHiNiMSKyIZEU4VUylZIqbfFNB4z0tVT5UZXNwvV07GPx0Skt+t1HxG5\nNdG6mSISnMlx/i4idTzY5ikRKZCOY70jIs2SHDf+PenqWh7/Xm0Wkc/jj5Nk+TcicqNreQkRWZLW\nWEz2ZInAeNMFVa2TaNqbRcftoaq1cToWfCOtG6vqVFX90DXbB7g10bp+qro1U6K8GucUPIvzKSBN\niUBEigFNXB2ZJT5u/HvyhWtZ/HsVClwGHktm+UngCQBVPQYcFpHwtMRjsidLBCZLub75rxSRda7p\n9mTKhIjIb65voptEpKprec9Ey6eJSJCbw60Aqri2bSVOf/V/iNPfez7X8tfl6jgNb7qWvSwi/3J9\nW24AzHcd8x+ub9QNXGcNCR/erjOH99IZ52oSdRomIu+LyFpxxhN4xbXsSZyEtFxElruWtRGR1a7f\n4+ciUiiZfd8PpPWb+8r431tqceI8xdwjjfs22ZAlAuNN/0jUBLHQtewocJeq1gMeBCYms91jwLuq\nWgfng/igq+uAB4Fw1/JY3H8I3Qv8ISL5gTnAg6oahvNE/UDXt+XOQIiq1gJGJ97Y9W15LVe/QV9I\ntPrfrm3jPQgsSGecd+N8qMYboaoNgFpAcxGppaoTcbpXbqmqLcXpPuIFoLXrd7kWGJrMvsOB/yZZ\nNj/R+3JNr7vi9L3VDvgjyfIgnG4cEj+xvRZo6qZuxg/kuC4mTLZywfVhmFge4D1Xm3gsTl84Sa0G\nRohIGeBLVd0pIq2A+sDvri4y/oGTVJIzX0QuAHtxuiSuDuxR1T9d6+fiNHG8hzNOwQci8i3wracV\nU9VjIrJbnL5edgI1gF9c+01LnHlxuv5I/Ht6QEQG4Px/lsIZbGVTkm2buJb/4jpOXpzfW1KluL7f\nnR7J9Ln0DxHZ4Hq9EqcblsTLSwPbgJ8SbXOURM1mxn9ZIjBZ7Z/AEaA2zhnpdQPGqOrHIrIGuAdY\nLCKP4ozGNFdVn/PgGNd80InITckVcvVj0wjnm25XYBBwZxrqsgB4ANgOLFRVFedT2eM4cb6tvwFM\nArqISEXgX0BDVT0lInNwOlFLSoCfVPUhN8e4kML215VLJmknLHddPP4BJ9HFn8Xld+3f+DlrGjJZ\nrQhw2NVXfC+cjsSuISKVgN2u5pCvcZpIlgFdReRmV5mbxPNxl3cAFUQkvt27F/Czq029iKouxklQ\ntZPZ9ixOl9bJWYgzWtRDOEmBtMbp6hztRaCJiNTAGWnrHBAlIiVxmmmSi+VXIDy+TiJSUESSO7va\nRvLt/WniGt3uSeBpV/MROGdzGb7Ty/ieJQKT1aYAj4jIRpzmlHPJlHkA2OxqkgjFGZZvK06b+I8i\nsgmnicKj4flU9SJOD42fi8gfQBwwFedD9VvX/laRfBv7HGBq/MXiJPs9hfNBW15Vf3MtS3OcrmsP\nb+H0FLoRZ8zh7cDHOM1N8aYDS0RkueuunT7AJ67jrMb5fSb1Ha5uqTNKVdfjNFHFn4W0dO3f+Dnr\nfdSYHE5EVgEdMnuUMhFZAXSKHx/Z+C9LBMbkcCLSGKetP+kF54zsswTOnVFfuS1ssj1LBMYYE+Ds\nGoExxgQ4SwTGGBPgLBEYY0yAs0RgjDEBzhKBMcYEuP8P0cys6+mwIAwAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "tags": [] }, "output_type": "display_data" } ], "source": [ "fpr, tpr, thresholds_roc = roc_curve(Y_train, Y_scores)\n", "bin_roc_auc = roc_auc_score(Y_train, Y_scores)\n", "print(\"Binary classifier ROC AUC score: \", bin_roc_auc)\n", "\n", "chosen_tpr = 0.94 # same as recall\n", "threshold_chosen =thresholds_roc[np.argmax(tpr >= chosen_tpr)]\n", "print(\"Threhsold chosen :\", threshold_chosen)\n", "Y_train_pred_new = (Y_scores >= threshold_chosen)\n", "\n", "conf_matrix = confusion_matrix(Y_train, Y_train_pred_new)\n", "print(conf_matrix)\n", "TN = conf_matrix[0,0]\n", "FP = conf_matrix[0,1]\n", "FN = conf_matrix[1,0]\n", "TP = conf_matrix[1,1]\n", "chosen_fpr = FP/(FP+TN)\n", "chosen_tpr_check = TP/(TP+FN)\n", "#print(TN,FP,FN,TP)\n", "\n", "plt.plot(fpr, tpr, \"b--\", linewidth=2, label=\"binary_classifier\")\n", "plt.plot([0, 1], [0, 1], 'k--', label=\"random_classifier\") # random classifier\n", "plt.plot(chosen_fpr, chosen_tpr,'ro')\n", "plt.xlabel(\"False Positive Rate (FPR)\")\n", "plt.ylabel(\"True Positive Rate (TPR) recall\")\n", "plt.grid()\n", "plt.legend()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "DEfldCX12ZhR" }, "source": [ "## Test set analysis and Results" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 283 }, "colab_type": "code", "id": "oIXgU63N6gqL", "outputId": "56c8759f-0dd2-40d4-9c48-79079053305e" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Prediction (Not-5, Yes-5): [ True]\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAOTUlEQVR4nO3df6xU9ZnH8c+ztvgLoijXm6vg3m7R\nGLPJUjJBYhVZm61iQrAaCfzRsEa8jRFDtSZrNBF/JRLd0hCzaXK7GGDDSmpaI3/gCpKqaaKEkbDK\nD38tQQteYYiagjEC8uwf99hc8J7vXOac+XF53q/kZmbOM2fOkxM+nJnznTNfc3cBOP39XbsbANAa\nhB0IgrADQRB2IAjCDgTxvVZubMKECd7b29vKTQKh7NmzRwcPHrThaoXCbmY3Slou6QxJ/+nuS1PP\n7+3tVbVaLbJJAAmVSiW31vDbeDM7Q9J/SJol6UpJ883sykZfD0BzFfnMPk3Sh+6+292PSForaU45\nbQEoW5GwXyLpL0Me782WncDM+sysambVWq1WYHMAimj62Xh373f3irtXurq6mr05ADmKhH2fpElD\nHk/MlgHoQEXCvkXSZWb2AzMbI2mepHXltAWgbA0Pvbn7MTNbJOllDQ69PevuO0rrDECpCo2zu/t6\nSetL6gVAE/F1WSAIwg4EQdiBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEgCDsQBGEHgiDs\nQBCEHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EARhB4Io\nNIsrOsPBgwdza8uWLUuuu3Tp0kLbdvdk3cwKvX673H777cn6ihUrWtRJeQqF3cz2SDok6RtJx9y9\nUkZTAMpXxpH9n909/9ACoCPwmR0IomjYXdIGM3vLzPqGe4KZ9ZlZ1cyqtVqt4OYANKpo2K9x96mS\nZkm628xmnPwEd+9394q7V7q6ugpuDkCjCoXd3fdltwckvSBpWhlNAShfw2E3s3PNbNy39yX9VNL2\nshoDUK4iZ+O7Jb2QjaN+T9J/u/v/lNLVKPPuu+8m67Nnz07Wi57LOH78eG7tyy+/TK7b7HHw0TrO\nvnr16mR9+vTpyfqdd95ZZjulaDjs7r5b0j+V2AuAJmLoDQiCsANBEHYgCMIOBEHYgSC4xHWEUsNr\ns2bNSq778ccfl90Omiw1nClJR48ebVEn5eHIDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBMM4+Qq+8\n8kpujXH00eeqq65K1q+44opkff78+WW20xIc2YEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCMbZR+i2\n227Lra1duza57htvvFF2OycYO3Zsbq3eTx4XVW/K5ocffji3duaZZ5bdzohNnjw5WR8/fnyLOmkd\njuxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EATj7CPU3d2dW3v++eeT61599dXJetHr4ZcsWZJbu+++\n+wq9Nk4fdY/sZvasmR0ws+1Dll1gZhvN7IPs9vT7BgJwmhnJ2/iVkm48adkDkja5+2WSNmWPAXSw\numF399clfXbS4jmSVmX3V0m6ueS+AJSs0RN03e4+kN3/VFLuB1oz6zOzqplVa7Vag5sDUFThs/E+\neCVE7tUQ7t7v7hV3r3R1dRXdHIAGNRr2/WbWI0nZ7YHyWgLQDI2GfZ2kBdn9BZJeLKcdAM1Sd5zd\nzJ6TNFPSBDPbK2mJpKWSfm9md0j6SNLcZjbZ6Xp6epL1s88+u6nbHzNmTG5t165dhV77oosuStYv\nvPDCQq+P1qkbdnfP+zX8n5TcC4Am4uuyQBCEHQiCsANBEHYgCMIOBMElrqeBxYsXN+21p06dmqzf\nddddyfott9ySWzv//PMb6gmN4cgOBEHYgSAIOxAEYQeCIOxAEIQdCIKwA0Ewzt4Ct956a7L+1FNP\nJevHjh0rs51TsnXr1mR94cKFyfozzzyTW7vnnnuS686bNy9ZP+ecc5J1nIgjOxAEYQeCIOxAEIQd\nCIKwA0EQdiAIwg4EYYMTurRGpVLxarXasu2NFqkplyXpiSeeaFEnp67evx8za/i1L7/88mR948aN\nyfrEiRMb3vZoValUVK1Wh93pHNmBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAiuZ+8Ajz76aLJ+6aWX\nJutPP/10bu39999vqKeRaub3NN57771k/frrr0/WN2zYkFvr7e1tpKVRre6R3cyeNbMDZrZ9yLJH\nzGyfmW3L/m5qbpsAihrJ2/iVkm4cZvlv3H1K9re+3LYAlK1u2N39dUmftaAXAE1U5ATdIjN7O3ub\nPz7vSWbWZ2ZVM6vWarUCmwNQRKNh/62kH0qaImlA0q/znuju/e5ecfdKV1dXg5sDUFRDYXf3/e7+\njbsfl/Q7SdPKbQtA2RoKu5n1DHn4M0nb854LoDPUvZ7dzJ6TNFPSBEn7JS3JHk+R5JL2SPqFuw/U\n2xjXszfHV199lVur95vzX3/9dbL++OOPJ+tbtmxJ1jdv3pysN9O1116bW3v11Vdb10gLpa5nr/ul\nGnefP8ziFYW7AtBSfF0WCIKwA0EQdiAIwg4EQdiBILjEtQMMDNQdtUzq6emp/6Qc48aNS9aXL1+e\nrB86dKjhel9fX3Ldl156KVmvp96wYjQc2YEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCMbZW+CTTz5J\n1ufOnZusz5s3L1lftGjRKfdUlnrj9Kn6mjVrkutWKpVkfffu3ck6TsSRHQiCsANBEHYgCMIOBEHY\ngSAIOxAEYQeCYJy9BQ4fPpys7927N1l/6KGHkvWXX345t9bf359ct8i18EWdd955yfrs2bOT9XrX\n2u/cuTO39tprryXXve6665L10YgjOxAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EwTh7C9S7nv3zzz9P\n1uuN069fvz63dsMNNyTXXbhwYbI+ffr0ZL2exx57LLd25MiR5LpvvvlmoW2n9tsXX3xR6LVHo7pH\ndjObZGZ/MrOdZrbDzBZnyy8ws41m9kF2O7757QJo1Ejexh+T9Ct3v1LSdEl3m9mVkh6QtMndL5O0\nKXsMoEPVDbu7D7j71uz+IUm7JF0iaY6kVdnTVkm6uVlNAijulE7QmVmvpB9J2iyp292/naTsU0nd\nOev0mVnVzKq1Wq1AqwCKGHHYzWyspD9I+qW7/3Vozd1dkg+3nrv3u3vF3StdXV2FmgXQuBGF3cy+\nr8Ggr3H3P2aL95tZT1bvkXSgOS0CKEPdoTczM0krJO1y92VDSuskLZC0NLt9sSkdngZmzpyZrM+Y\nMSNZTw2t1bNjx45k/d577234tSVp8E1dvsF/PugEIxln/7Gkn0t6x8y2Zcse1GDIf29md0j6SFL6\nx88BtFXdsLv7nyXl/ff8k3LbAdAsfF0WCIKwA0EQdiAIwg4EQdiBILjEtQPU+7nnepep1htLj2rM\nmDG5tbPOOquFnXQGjuxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EATj7B2g3rTJfX19yfr999+fWzt6\n9GhDPZ0OUj+DXe+7C6cjjuxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EATj7KPAokWLkvWLL744t7Z1\n69bkuk8++WRDPXWCyZMnJ+srV65sTSOjBEd2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQjCRjC/9iRJ\nqyV1S3JJ/e6+3MwekXSnpFr21AfdPTmReKVS8Wq1WrhpAMOrVCqqVqvDzro8ki/VHJP0K3ffambj\nJL1lZhuz2m/c/d/LahRA84xkfvYBSQPZ/UNmtkvSJc1uDEC5Tukzu5n1SvqRpM3ZokVm9raZPWtm\n43PW6TOzqplVa7XacE8B0AIjDruZjZX0B0m/dPe/SvqtpB9KmqLBI/+vh1vP3fvdveLula6urhJa\nBtCIEYXdzL6vwaCvcfc/SpK773f3b9z9uKTfSZrWvDYBFFU37GZmklZI2uXuy4YsH/qTqD+TtL38\n9gCUZSRn438s6eeS3jGzbdmyByXNN7MpGhyO2yPpF03pEEApRnI2/s+Shhu3S46pA+gsfIMOCIKw\nA0EQdiAIwg4EQdiBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEgCDsQRN2fki51Y2Y1SR8N\nWTRB0sGWNXBqOrW3Tu1LordGldnb37v7sL//1tKwf2fjZlV3r7StgYRO7a1T+5LorVGt6o238UAQ\nhB0Iot1h72/z9lM6tbdO7Uuit0a1pLe2fmYH0DrtPrIDaBHCDgTRlrCb2Y1m9p6ZfWhmD7Sjhzxm\ntsfM3jGzbWbW1vmlszn0DpjZ9iHLLjCzjWb2QXY77Bx7bertETPbl+27bWZ2U5t6m2RmfzKznWa2\nw8wWZ8vbuu8SfbVkv7X8M7uZnSHpfUn/ImmvpC2S5rv7zpY2ksPM9kiquHvbv4BhZjMkHZa02t3/\nMVv2lKTP3H1p9h/leHf/tw7p7RFJh9s9jXc2W1HP0GnGJd0s6V/Vxn2X6GuuWrDf2nFknybpQ3ff\n7e5HJK2VNKcNfXQ8d39d0mcnLZ4jaVV2f5UG/7G0XE5vHcHdB9x9a3b/kKRvpxlv675L9NUS7Qj7\nJZL+MuTxXnXWfO8uaYOZvWVmfe1uZhjd7j6Q3f9UUnc7mxlG3Wm8W+mkacY7Zt81Mv15UZyg+65r\n3H2qpFmS7s7ernYkH/wM1kljpyOaxrtVhplm/G/aue8anf68qHaEfZ+kSUMeT8yWdQR335fdHpD0\ngjpvKur9386gm90eaHM/f9NJ03gPN824OmDftXP683aEfYuky8zsB2Y2RtI8Seva0Md3mNm52YkT\nmdm5kn6qzpuKep2kBdn9BZJebGMvJ+iUabzzphlXm/dd26c/d/eW/0m6SYNn5P9P0kPt6CGnr3+Q\n9L/Z34529ybpOQ2+rTuqwXMbd0i6UNImSR9IekXSBR3U239JekfS2xoMVk+bertGg2/R35a0Lfu7\nqd37LtFXS/YbX5cFguAEHRAEYQeCIOxAEIQdCIKwA0EQdiAIwg4E8f9zilrSPdBN3wAAAABJRU5E\nrkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "tags": [] }, "output_type": "display_data" } ], "source": [ "test_img = X_test[766] # 5\n", "#test_img = X_test[1] # 2\n", "Y_pred_proba = clf.predict_proba([test_img])\n", "Y_score = Y_pred_proba[:, 1]\n", "Y_pred = (Y_score >= threshold_chosen)\n", "test_img = test_img.reshape(28, 28)\n", "plt.imshow(test_img, cmap = mpl.cm.binary,interpolation=\"nearest\")\n", "print(\"Prediction (Not-5, Yes-5): \", Y_pred)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "P_jysF-gSNbz" }, "source": [ "## Considerations" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "Za6vJhBmQCI2" }, "source": [ "As a rule of thumb,you should prefer the PR curve whenever the positive class is rare orwhen you care more about the false positives than the false negatives,and the ROC curve otherwise. For example, looking at the previousROC curve (and the ROC AUC score), you may think that the classifieris really good. But this is mostly because there are few positives (5s)compared to the negatives (non-5s). In contrast, the PR curve makes itclear that the classifier has room for improvement (the curve could becloser to the top-right corner " ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "6FvAl_fxg5JE" }, "source": [ "## Extra: Cross validation imaplementation" ] }, { "cell_type": "code", "execution_count": 0, "metadata": { "colab": {}, "colab_type": "code", "id": "4wTF0w6Lg6g1" }, "outputs": [], "source": [ "# Custom implementation of cross validation\n", "# This code does the same of cross_val_score()\n", "from sklearn.model_selection import StratifiedKFold\n", "from sklearn.base import clone\n", "skfolds = StratifiedKFold(n_splits=3, random_state=42)\n", "for train_index, test_index in skfolds.split(X_train, Y_train_5):\n", " clone_clf = clone(sgd_clf)\n", " X_train_folds = X_train[train_index]\n", " Y_train_folds = Y_train_5[train_index]\n", " X_test_fold = X_train[test_index]\n", " Y_test_fold = Y_train_5[test_index]\n", " clone_clf.fit(X_train_folds, Y_train_folds)\n", " Y_pred = clone_clf.predict(X_test_fold)\n", " n_correct = sum(Y_pred == Y_test_fold)\n", " print(n_correct / len(Y_pred)) # prints 0.9502, 0.96565 and 0.96495" ] } ], "metadata": { "colab": { "collapsed_sections": [], "name": "binary_classification.ipynb", "provenance": [], "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.9" } }, "nbformat": 4, "nbformat_minor": 1 }