Module 6 Section 1 Lab 2 - Cross Validation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 6 Section 1 Lab 1 - Cross Validation\n",
    "Pengembangan model adalah langkah penting dalam siklus hidup *machine learning*. Kita akan melatih kumpulan data dengan berbagai jenis model yang tersedia. Kita perlu memastikan bahwa model apa pun harus berkinerja baik pada data masa depan. Biasanya, estimasi kesalahan model dilakukan setelah melatih model pada kumpulan data latih, yang dikenal dengan evaluasi residual. Hal itu tidak dapat dipercaya karena hanya berfungsi dengan baik pada data latih. Ada kemungkinan model tersebut *underfitting* atau *overfitting* ketika di uji pada data uji sebenarnya. Jadi, kita tidak bisa hanya melihat metrik evaluasi untuk memilih model dengan kinerja terbaik. Teknik evaluasi yang membantu mengetahui seberapa baik model kita dikenal sebagai Cross-Validation.\n",
    "\n",
    "**Cross Validation** adalah teknik *resampling* yang membuat model kita yakin tentang efesiensi dan performa pada data yang tidak terlihat. Ini merupakan metode untuk mengevaluasi model dengan melatih model pada subset dari kumpulan data input yang tersedia dan mengevaluasinya pada subset yang berbeda dari kumpulan data.\n",
    "\n",
    "Referensi: https://scikit-learn.org/stable/modules/cross_validation.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>Glucose</th>\n",
       "      <th>BloodPressure</th>\n",
       "      <th>SkinThickness</th>\n",
       "      <th>Insulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Age</th>\n",
       "      <th>Outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6</td>\n",
       "      <td>148</td>\n",
       "      <td>72</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>33.6</td>\n",
       "      <td>0.627</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>85</td>\n",
       "      <td>66</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>26.6</td>\n",
       "      <td>0.351</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>183</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>23.3</td>\n",
       "      <td>0.672</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>89</td>\n",
       "      <td>66</td>\n",
       "      <td>23</td>\n",
       "      <td>94</td>\n",
       "      <td>28.1</td>\n",
       "      <td>0.167</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>137</td>\n",
       "      <td>40</td>\n",
       "      <td>35</td>\n",
       "      <td>168</td>\n",
       "      <td>43.1</td>\n",
       "      <td>2.288</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Pregnancies  Glucose  BloodPressure  SkinThickness  Insulin   BMI  \\\n",
       "0            6      148             72             35        0  33.6   \n",
       "1            1       85             66             29        0  26.6   \n",
       "2            8      183             64              0        0  23.3   \n",
       "3            1       89             66             23       94  28.1   \n",
       "4            0      137             40             35      168  43.1   \n",
       "\n",
       "   DiabetesPedigreeFunction  Age  Outcome  \n",
       "0                     0.627   50        1  \n",
       "1                     0.351   31        0  \n",
       "2                     0.672   32        1  \n",
       "3                     0.167   21        0  \n",
       "4                     2.288   33        1  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "data = pd.read_csv('dataset/diabetes.csv')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(768, 9)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pisahkan kolom feature dan target\n",
    "X = data.iloc[:,:-1]\n",
    "y = data.iloc[:,8:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hold Out Validation - Train and Test Split\n",
    "Metode Holdout cukup mudah dipahami dan dikerjakan. Sampel data dibagi menjadi dua bagian: data latih dan data uji. Sebelum pembagian berlangsung, sampel data diacak sehingga sampel tercampur dan menghasilkan kumpulan data latih yang akurat. Biasanya, rasio data latih dengan data uji adalah 70:30 atau 80:20. Langkah selanjutnya adalah melatih model dengan data latih dan setelah dilatih, model diuji dengan data uji. Salah satu kelemahan metode ini adalah kumpulan data uji dapat berisi karakteristik penting dari keseluruhan pola atau tren data yang dapat terlewatkan. Dan mungkin saja kumpulan latih tidak mewakili keseluruhan sampel data. Metode ini juga dikenal sebagai pendekatan train/test split.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Akurasi pelatihan: 1.0\n",
      "Akurasi pengujian: 0.6875\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "Split arrays or matrices into random train and test subsets\n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html\n",
    "'''\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25)\n",
    "\n",
    "clf = DecisionTreeClassifier()\n",
    "clf.fit(X_train, y_train)\n",
    "\n",
    "train = clf.predict(X_train)\n",
    "test = clf.predict(X_test)\n",
    "\n",
    "acc_train = accuracy_score(y_train, train)\n",
    "acc_test = accuracy_score(y_test, test)\n",
    "\n",
    "print('Akurasi pelatihan:', acc_train)\n",
    "print('Akurasi pengujian:', acc_test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## K-Fold Cross Validation\n",
    "Jenis lain dari cross validation adalah K-Fold. Parameter jenis ini adalah $K$ yang mengacu pada jumlah subset atau lipatan yang diperoleh dari sampel data. Langkah pertama adalah melatih model menggunakan seluruh kumpulan data. Langkah kedua adalah membagi sampel data dalam jumlah $K$ himpunan bagian (misalkan, 10). Dari sini, himpunan bagian ini menjadi kumpulan data pengujian yang kemudian digunakan untuk menguji model satu per satu. Jenis ini dianggap sebagai metode validasi yang tidak bias dan inklusif karena melibatkan pelatihan dan pengujian hampir pada setiap subset.\n",
    "\n",
    "<img src='https://scikit-learn.org/stable/_images/sphx_glr_plot_cv_indices_006.png'></img>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.70689655 0.67241379 0.67241379 0.87931034 0.65517241 0.77586207\n",
      " 0.64912281 0.75438596 0.64912281 0.71929825]\n",
      "\n",
      "Rata-rata akurasi pelatihan pada K-Fold: 0.7133998790078644\n",
      "Akurasi pengujian: 0.6875\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "K-Folds cross-validator\n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html#sklearn.model_selection.KFold\n",
    "'''\n",
    "\n",
    "from sklearn.model_selection import KFold\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "clf = DecisionTreeClassifier()\n",
    "\n",
    "kfold = KFold(n_splits=10)\n",
    "kfold_train = cross_val_score(clf, X_train, y_train, cv=kfold)\n",
    "\n",
    "print(kfold_train)\n",
    "print('\\nRata-rata akurasi pelatihan pada K-Fold:', np.mean(kfold_train))\n",
    "print('Akurasi pengujian:', acc_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Repeated K-Fold Cross Validation\n",
    "Mengulangi K-Fold sebanyak $n$ kali. Dapat digunakan ketika perlu menjalankan K-Fold sebanyak $n$ kali sehingga menghasilkan pemisahan yang berbeda di setiap pengulangan."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.62068966 0.74137931 0.70689655 0.75862069 0.70689655 0.79310345\n",
      " 0.68421053 0.54385965 0.59649123 0.63157895 0.67241379 0.56896552\n",
      " 0.60344828 0.63793103 0.70689655 0.79310345 0.68421053 0.70175439\n",
      " 0.63157895 0.80701754]\n",
      "\n",
      "Rata-rata akurasi pelatihan pada Repeated K-Fold: 0.6795523290986085\n",
      "Akurasi pengujian: 0.6875\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "Repeated K-Fold cross validator\n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RepeatedKFold.html#sklearn.model_selection.RepeatedKFold\n",
    "'''\n",
    "\n",
    "from sklearn.model_selection import RepeatedKFold\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "clf = DecisionTreeClassifier()\n",
    "\n",
    "re_kfold = RepeatedKFold(n_splits=10, n_repeats=2)\n",
    "re_kfold_train = cross_val_score(clf, X_train, y_train, cv=re_kfold)\n",
    "\n",
    "print(re_kfold_train)\n",
    "print('\\nRata-rata akurasi pelatihan pada Repeated K-Fold:', np.mean(re_kfold_train))\n",
    "print('Akurasi pengujian:', acc_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stratified K-Fold Cross Validation\n",
    "Metode K-Fold bertingkat adalah metode lain yang melibatkan pembagian kumpulan sampel data dalam himpunan bagian atau lipatan $K$. Namun, untuk memastikan tidak ada pembagian data yang bias pada lipatan $K$, maka dilakukan proses stratifikasi untuk menata ulang data sedemikian rupa sehingga setiap lipatan mewakili seluruh data. Khususnya dalam tugas klasifikasi, subset harus merepresentasikan data dari kedua kelas atau semua kelas. Mungkin stratifikasi mengarah pada penempatan data dari semua kelas di semua himpunan bagian sedemikian rupa sehingga himpunan bagian sepenuhnya mewakili data dan mengarah pada performa model yang lebih tinggi.\n",
    "\n",
    "<img src='https://scikit-learn.org/stable/_images/sphx_glr_plot_cv_indices_009.png'></img>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.70689655 0.68965517 0.68965517 0.62068966 0.67241379 0.74137931\n",
      " 0.68421053 0.73684211 0.66666667 0.70175439]\n",
      "\n",
      "Rata-rata akurasi pelatihan pada Stratified K-Fold: 0.691016333938294\n",
      "Akurasi pengujian: 0.6875\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "Stratified K-Folds cross-validator\n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html#sklearn.model_selection.StratifiedKFold\n",
    "'''\n",
    "\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "clf = DecisionTreeClassifier()\n",
    "\n",
    "stra_kfold = StratifiedKFold(n_splits=10)\n",
    "stra_kfold_train = cross_val_score(clf, X_train, y_train, cv=stra_kfold)\n",
    "\n",
    "print(stra_kfold_train)\n",
    "print('\\nRata-rata akurasi pelatihan pada Stratified K-Fold:', np.mean(stra_kfold_train))\n",
    "print('Akurasi pengujian:', acc_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Leave-P-Out (LPO \\ LPOCV)\n",
    "Sampel data terdiri dari titik data ($n$). Jumlah total titik data ($n$) digunakan untuk memisahkan sekumpulan titik data yang digunakan untuk pengujian. Titik data ini disebut sebagai ($p$). Kumpulan data latih diperoleh dengan menghitung ($n-p$) dan model dilatih sesuai dengan itu. Setelah pelatihan selesai, $p$ titik data digunakan untuk cross validation. Semua kemungkinan kombinasi $p$ diuji pada model sehingga mendapatkan akurasi yang maksimal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "576"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'''\n",
    "Leave-P-Out cross-validator\n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.LeavePOut.html#sklearn.model_selection.LeavePOut\n",
    "'''\n",
    "\n",
    "from sklearn.model_selection import LeavePOut\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "clf = DecisionTreeClassifier()\n",
    "\n",
    "# tentukan nilai p=int. Semakin besar akan semakin lama komputasinya\n",
    "lpovc = LeavePOut(p=1)\n",
    "lpovc.get_n_splits(X_train)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0. 1. 1. 1. 0. 1. 0. 1. 1. 1. 0. 1. 1. 1. 1. 1. 0. 1. 1. 0. 0. 1. 1. 0.\n",
      " 1. 1. 1. 1. 1. 0. 1. 1. 0. 0. 1. 1. 1. 0. 1. 0. 1. 1. 1. 1. 1. 1. 0. 0.\n",
      " 1. 1. 1. 1. 0. 0. 0. 1. 1. 0. 1. 1. 1. 1. 1. 1. 1. 1. 1. 0. 1. 1. 1. 1.\n",
      " 1. 0. 1. 1. 1. 1. 1. 0. 1. 0. 0. 1. 0. 0. 1. 0. 1. 1. 1. 0. 1. 1. 0. 0.\n",
      " 1. 1. 0. 0. 1. 0. 1. 1. 1. 0. 0. 1. 1. 1. 0. 1. 1. 1. 1. 0. 0. 0. 0. 1.\n",
      " 0. 0. 0. 1. 1. 1. 1. 0. 1. 0. 1. 0. 0. 1. 1. 1. 0. 0. 1. 1. 1. 0. 0. 1.\n",
      " 1. 0. 1. 1. 0. 1. 0. 1. 1. 1. 1. 1. 0. 0. 1. 0. 1. 1. 1. 1. 1. 1. 1. 0.\n",
      " 1. 0. 1. 1. 1. 0. 1. 1. 1. 0. 1. 0. 1. 1. 0. 1. 0. 1. 1. 1. 1. 0. 1. 1.\n",
      " 1. 1. 1. 0. 0. 0. 0. 0. 1. 1. 0. 1. 1. 0. 1. 0. 0. 0. 1. 1. 1. 1. 1. 0.\n",
      " 1. 1. 0. 1. 1. 1. 1. 1. 1. 1. 1. 0. 1. 0. 1. 1. 0. 1. 1. 1. 1. 1. 1. 0.\n",
      " 1. 0. 1. 1. 0. 0. 1. 0. 1. 1. 1. 1. 1. 0. 1. 0. 1. 0. 0. 1. 1. 1. 0. 0.\n",
      " 1. 1. 1. 1. 1. 1. 0. 1. 1. 0. 1. 1. 0. 1. 1. 1. 1. 0. 1. 0. 0. 1. 0. 1.\n",
      " 1. 0. 1. 1. 1. 1. 1. 1. 0. 1. 1. 0. 0. 1. 1. 0. 1. 0. 1. 1. 1. 1. 1. 1.\n",
      " 0. 1. 0. 1. 0. 0. 1. 1. 1. 1. 1. 1. 1. 0. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.\n",
      " 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 0. 1. 0. 0. 0. 1. 0. 0. 1. 1. 1. 0. 1. 1.\n",
      " 0. 0. 1. 0. 0. 1. 1. 1. 0. 1. 1. 0. 1. 1. 1. 1. 0. 1. 1. 0. 1. 1. 0. 1.\n",
      " 0. 0. 1. 1. 1. 0. 0. 1. 0. 1. 0. 0. 1. 1. 0. 0. 1. 1. 0. 1. 1. 0. 1. 1.\n",
      " 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 0. 1. 1. 1. 1. 1. 1. 1. 1. 1. 0. 0.\n",
      " 1. 0. 0. 1. 1. 1. 1. 0. 0. 0. 1. 1. 0. 1. 0. 1. 1. 1. 0. 0. 1. 1. 0. 1.\n",
      " 0. 1. 1. 1. 1. 1. 0. 0. 1. 1. 1. 0. 0. 0. 0. 0. 0. 0. 1. 0. 1. 0. 1. 0.\n",
      " 1. 1. 1. 1. 1. 0. 1. 1. 0. 1. 0. 0. 0. 1. 1. 0. 1. 1. 1. 0. 0. 1. 1. 1.\n",
      " 1. 1. 1. 0. 0. 1. 1. 1. 0. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 0. 1. 1.\n",
      " 1. 1. 0. 0. 1. 1. 0. 0. 0. 1. 1. 0. 1. 0. 1. 1. 0. 1. 1. 1. 1. 0. 1. 0.\n",
      " 1. 1. 0. 1. 1. 0. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 0. 1. 1. 0. 0. 1. 1.]\n",
      "\n",
      "Rata-rata akurasi pelatihan pada LPOVC: 0.671875\n",
      "Akurasi pengujian: 0.6875\n"
     ]
    }
   ],
   "source": [
    "lpovc_train = cross_val_score(clf, X_train, y_train, cv=lpovc)\n",
    "\n",
    "print(lpovc_train)\n",
    "print('\\nRata-rata akurasi pelatihan pada LPOVC:', np.mean(lpovc_train))\n",
    "print('Akurasi pengujian:', acc_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Leave One Out (LOO / LOOCV)\n",
    "Varian dari metode LPOVC. Pada metode ini, $p$ dijaga menjadi $1 (p=1)$ dan titik data $n-p$ digunakan untuk melatih model. Jadi, ketika pelatihan selesai, titik data $p$ atau titik data tunggal digunakan untuk memvalidasi model. Salah satu kelemahan terbesar dari jenis ini adalah bahwa sebagian besar sampel data digunakan untuk melatih model, namun, hanya satu titik data yang digunakan untuk mengevaluasinya."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1. 1. 1. 1. 0. 1. 0. 1. 1. 1. 0. 1. 1. 1. 1. 1. 1. 1. 1. 0. 0. 1. 1. 0.\n",
      " 1. 1. 0. 1. 1. 0. 1. 1. 0. 0. 1. 1. 1. 1. 1. 0. 1. 1. 1. 1. 1. 1. 0. 0.\n",
      " 1. 1. 1. 1. 0. 0. 0. 1. 1. 0. 1. 1. 0. 1. 1. 1. 1. 1. 1. 0. 1. 1. 1. 1.\n",
      " 1. 0. 1. 1. 0. 1. 1. 0. 1. 0. 0. 1. 0. 0. 1. 0. 1. 1. 1. 1. 1. 1. 0. 1.\n",
      " 1. 1. 1. 0. 1. 0. 1. 1. 1. 0. 0. 1. 1. 1. 0. 1. 1. 1. 1. 0. 0. 0. 0. 1.\n",
      " 0. 1. 0. 1. 1. 1. 1. 0. 1. 0. 1. 0. 0. 1. 1. 1. 0. 0. 1. 1. 1. 0. 0. 1.\n",
      " 1. 0. 1. 1. 0. 1. 0. 1. 1. 1. 1. 1. 0. 0. 1. 0. 1. 1. 1. 1. 1. 1. 0. 0.\n",
      " 1. 0. 1. 1. 1. 0. 1. 0. 1. 0. 1. 0. 1. 1. 0. 1. 0. 1. 1. 1. 1. 1. 0. 1.\n",
      " 1. 1. 1. 0. 0. 0. 0. 0. 1. 1. 0. 1. 1. 1. 1. 0. 1. 0. 1. 1. 1. 1. 1. 0.\n",
      " 1. 1. 0. 1. 1. 1. 1. 1. 1. 1. 1. 0. 1. 0. 1. 1. 0. 1. 1. 1. 1. 1. 1. 0.\n",
      " 1. 0. 1. 1. 0. 1. 1. 0. 1. 1. 1. 1. 1. 0. 1. 0. 1. 0. 0. 1. 1. 1. 0. 0.\n",
      " 1. 1. 1. 1. 1. 1. 0. 1. 1. 0. 1. 1. 1. 1. 1. 1. 1. 0. 1. 0. 0. 1. 0. 1.\n",
      " 1. 0. 1. 1. 1. 1. 1. 1. 0. 1. 1. 0. 0. 1. 1. 1. 1. 0. 1. 1. 1. 1. 1. 1.\n",
      " 0. 1. 0. 1. 0. 0. 1. 1. 1. 1. 1. 1. 1. 0. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.\n",
      " 0. 1. 1. 1. 1. 1. 1. 0. 1. 1. 0. 1. 0. 0. 0. 1. 0. 0. 1. 1. 1. 0. 1. 1.\n",
      " 0. 0. 1. 0. 0. 1. 1. 1. 0. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 0. 1.\n",
      " 0. 0. 1. 1. 1. 0. 0. 1. 0. 1. 0. 0. 1. 1. 0. 0. 1. 1. 0. 1. 1. 0. 1. 1.\n",
      " 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 0. 1. 1. 0. 1. 1. 1. 1. 1. 1. 0. 0.\n",
      " 1. 0. 0. 1. 1. 1. 1. 0. 0. 0. 1. 1. 0. 1. 0. 1. 1. 1. 0. 0. 1. 1. 0. 1.\n",
      " 0. 1. 1. 1. 1. 1. 0. 1. 0. 1. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 1. 1.\n",
      " 1. 1. 1. 1. 1. 0. 1. 1. 1. 1. 0. 0. 0. 1. 1. 0. 1. 1. 1. 0. 0. 1. 1. 1.\n",
      " 1. 1. 1. 0. 0. 1. 1. 1. 0. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 0. 1. 1.\n",
      " 1. 1. 0. 0. 1. 1. 0. 0. 0. 1. 1. 0. 0. 0. 1. 1. 0. 1. 1. 1. 1. 0. 1. 0.\n",
      " 1. 1. 0. 1. 1. 0. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 0. 1. 1. 1. 0. 1. 1.]\n",
      "\n",
      "Rata-rata akurasi pelatihan pada LOOVC: 0.6857638888888888\n",
      "Akurasi pengujian: 0.6875\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "Leave-One-Out cross-validator\n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.LeaveOneOut.html#sklearn.model_selection.LeaveOneOut\n",
    "'''\n",
    "\n",
    "from sklearn.model_selection import LeaveOneOut\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "clf = DecisionTreeClassifier()\n",
    "\n",
    "loovc = LeaveOneOut()\n",
    "loovc_train = cross_val_score(clf, X_train, y_train, cv=loovc)\n",
    "\n",
    "print(loovc_train)\n",
    "print('\\nRata-rata akurasi pelatihan pada LOOVC:', np.mean(loovc_train))\n",
    "print('Akurasi pengujian:', acc_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Baca selanjutnya: https://www.di.ens.fr/willow/pdfs/2010_Arlot_Celisse_SS.pdf"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "b3ba2566441a7c06988d0923437866b63cedc61552a5af99d1f4fb67d367b25f"
  },
  "kernelspec": {
   "display_name": "Python 3.8.8 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}