examples/ml_agents/h2o_machine_learning_agent.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create AI Machine Learning Agent That Makes 30+ ML Models In 30 Seconds\n",
    "\n",
    "In this notebook, I will show you how to create an AI machine learning agent that can make 30+ machine learning models in 30 seconds. This AI agent will be able to make models for classification and regression tasks. \n",
    "\n",
    "The AI agent uses `h2o` library for AutoML. `h2o` is an open-source machine learning platform that is used for building machine learning models. It is easy to use and provides a lot of flexibility in building machine learning models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Want To Become A Full-Stack Generative AI Data Scientist?\n",
    "\n",
    "![Generative AI Data Scientist](../../img/become_a_generative_ai_data_scientist.jpg)\n",
    "\n",
    "I teach Generative AI Data Science to help you build AI-powered data science apps. [**Register for my next Generative AI for Data Scientists workshop here.**](https://learn.business-science.io/ai-register)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## H2O Machine Learning Agent\n",
    "\n",
    "In this notebook, we will create an AI Machine Learning Agent that makes 30+ machine learning models in 30 seconds. The AI Agent will use the H2O AutoML library to create the models, store the models on your computer, and provide you with a summary of the models, performance metrics, and full breakdown of the AI machine learning workflow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "from langchain_openai import ChatOpenAI\n",
    "import pandas as pd\n",
    "import h2o # pip install h2o\n",
    "import os\n",
    "import yaml\n",
    "\n",
    "from ai_data_science_team.ml_agents import H2OMLAgent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load The Customer Churn Dataset\n",
    "\n",
    "The Customer Churn dataset contains data on customers who have left the company. The dataset contains 21 columns and 7,043 rows. The target variable is `Churn` which is a binary variable that indicates whether the customer has left the company or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.microsoft.datawrangler.viewer.v0+json": {
       "columns": [
        {
         "name": "index",
         "rawType": "int64",
         "type": "integer"
        },
        {
         "name": "customerID",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "gender",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "SeniorCitizen",
         "rawType": "int64",
         "type": "integer"
        },
        {
         "name": "Partner",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "Dependents",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "tenure",
         "rawType": "int64",
         "type": "integer"
        },
        {
         "name": "PhoneService",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "MultipleLines",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "InternetService",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "OnlineSecurity",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "OnlineBackup",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "DeviceProtection",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "TechSupport",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "StreamingTV",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "StreamingMovies",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "Contract",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "PaperlessBilling",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "PaymentMethod",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "MonthlyCharges",
         "rawType": "float64",
         "type": "float"
        },
        {
         "name": "TotalCharges",
         "rawType": "object",
         "type": "string"
        },
        {
         "name": "Churn",
         "rawType": "object",
         "type": "string"
        }
       ],
       "conversionMethod": "pd.DataFrame",
       "ref": "c83e5dac-07eb-4ab1-9efe-067990ef197e",
       "rows": [
        [
         "0",
         "7590-VHVEG",
         "Female",
         "0",
         "Yes",
         "No",
         "1",
         "No",
         "No phone service",
         "DSL",
         "No",
         "Yes",
         "No",
         "No",
         "No",
         "No",
         "Month-to-month",
         "Yes",
         "Electronic check",
         "29.85",
         "29.85",
         "No"
        ],
        [
         "1",
         "5575-GNVDE",
         "Male",
         "0",
         "No",
         "No",
         "34",
         "Yes",
         "No",
         "DSL",
         "Yes",
         "No",
         "Yes",
         "No",
         "No",
         "No",
         "One year",
         "No",
         "Mailed check",
         "56.95",
         "1889.5",
         "No"
        ],
        [
         "2",
         "3668-QPYBK",
         "Male",
         "0",
         "No",
         "No",
         "2",
         "Yes",
         "No",
         "DSL",
         "Yes",
         "Yes",
         "No",
         "No",
         "No",
         "No",
         "Month-to-month",
         "Yes",
         "Mailed check",
         "53.85",
         "108.15",
         "Yes"
        ],
        [
         "3",
         "7795-CFOCW",
         "Male",
         "0",
         "No",
         "No",
         "45",
         "No",
         "No phone service",
         "DSL",
         "Yes",
         "No",
         "Yes",
         "Yes",
         "No",
         "No",
         "One year",
         "No",
         "Bank transfer (automatic)",
         "42.3",
         "1840.75",
         "No"
        ],
        [
         "4",
         "9237-HQITU",
         "Female",
         "0",
         "No",
         "No",
         "2",
         "Yes",
         "No",
         "Fiber optic",
         "No",
         "No",
         "No",
         "No",
         "No",
         "No",
         "Month-to-month",
         "Yes",
         "Electronic check",
         "70.7",
         "151.65",
         "Yes"
        ],
        [
         "5",
         "9305-CDSKC",
         "Female",
         "0",
         "No",
         "No",
         "8",
         "Yes",
         "Yes",
         "Fiber optic",
         "No",
         "No",
         "Yes",
         "No",
         "Yes",
         "Yes",
         "Month-to-month",
         "Yes",
         "Electronic check",
         "99.65",
         "820.5",
         "Yes"
        ],
        [
         "6",
         "1452-KIOVK",
         "Male",
         "0",
         "No",
         "Yes",
         "22",
         "Yes",
         "Yes",
         "Fiber optic",
         "No",
         "Yes",
         "No",
         "No",
         "Yes",
         "No",
         "Month-to-month",
         "Yes",
         "Credit card (automatic)",
         "89.1",
         "1949.4",
         "No"
        ],
        [
         "7",
         "6713-OKOMC",
         "Female",
         "0",
         "No",
         "No",
         "10",
         "No",
         "No phone service",
         "DSL",
         "Yes",
         "No",
         "No",
         "No",
         "No",
         "No",
         "Month-to-month",
         "No",
         "Mailed check",
         "29.75",
         "301.9",
         "No"
        ],
        [
         "8",
         "7892-POOKP",
         "Female",
         "0",
         "Yes",
         "No",
         "28",
         "Yes",
         "Yes",
         "Fiber optic",
         "No",
         "No",
         "Yes",
         "Yes",
         "Yes",
         "Yes",
         "Month-to-month",
         "Yes",
         "Electronic check",
         "104.8",
         "3046.05",
         "Yes"
        ],
        [
         "9",
         "6388-TABGU",
         "Male",
         "0",
         "No",
         "Yes",
         "62",
         "Yes",
         "No",
         "DSL",
         "Yes",
         "Yes",
         "No",
         "No",
         "No",
         "No",
         "One year",
         "No",
         "Bank transfer (automatic)",
         "56.15",
         "3487.95",
         "No"
        ],
        [
         "10",
         "9763-GRSKD",
         "Male",
         "0",
         "Yes",
         "Yes",
         "13",
         "Yes",
         "No",
         "DSL",
         "Yes",
         "No",
         "No",
         "No",
         "No",
         "No",
         "Month-to-month",
         "Yes",
         "Mailed check",
         "49.95",
         "587.45",
         "No"
        ],
        [
         "11",
         "7469-LKBCI",
         "Male",
         "0",
         "No",
         "No",
         "16",
         "Yes",
         "No",
         "No",
         "No internet service",
         "No internet service",
         "No internet service",
         "No internet service",
         "No internet service",
         "No internet service",
         "Two year",
         "No",
         "Credit card (automatic)",
         "18.95",
         "326.8",
         "No"
        ],
        [
         "12",
         "8091-TTVAX",
         "Male",
         "0",
         "Yes",
         "No",
         "58",
         "Yes",
         "Yes",
         "Fiber optic",
         "No",
         "No",
         "Yes",
         "No",
         "Yes",
         "Yes",
         "One year",
         "No",
         "Credit card (automatic)",
         "100.35",
         "5681.1",
         "No"
        ],
        [
         "13",
         "0280-XJGEX",
         "Male",
         "0",
         "No",
         "No",
         "49",
         "Yes",
         "Yes",
         "Fiber optic",
         "No",
         "Yes",
         "Yes",
         "No",
         "Yes",
         "Yes",
         "Month-to-month",
         "Yes",
         "Bank transfer (automatic)",
         "103.7",
         "5036.3",
         "Yes"
        ],
        [
         "14",
         "5129-JLPIS",
         "Male",
         "0",
         "No",
         "No",
         "25",
         "Yes",
         "No",
         "Fiber optic",
         "Yes",
         "No",
         "Yes",
         "Yes",
         "Yes",
         "Yes",
         "Month-to-month",
         "Yes",
         "Electronic check",
         "105.5",
         "2686.05",
         "No"
        ],
        [
         "15",
         "3655-SNQYZ",
         "Female",
         "0",
         "Yes",
         "Yes",
         "69",
         "Yes",
         "Yes",
         "Fiber optic",
         "Yes",
         "Yes",
         "Yes",
         "Yes",
         "Yes",
         "Yes",
         "Two year",
         "No",
         "Credit card (automatic)",
         "113.25",
         "7895.15",
         "No"
        ],
        [
         "16",
         "8191-XWSZG",
         "Female",
         "0",
         "No",
         "No",
         "52",
         "Yes",
         "No",
         "No",
         "No internet service",
         "No internet service",
         "No internet service",
         "No internet service",
         "No internet service",
         "No internet service",
         "One year",
         "No",
         "Mailed check",
         "20.65",
         "1022.95",
         "No"
        ],
        [
         "17",
         "9959-WOFKT",
         "Male",
         "0",
         "No",
         "Yes",
         "71",
         "Yes",
         "Yes",
         "Fiber optic",
         "Yes",
         "No",
         "Yes",
         "No",
         "Yes",
         "Yes",
         "Two year",
         "No",
         "Bank transfer (automatic)",
         "106.7",
         "7382.25",
         "No"
        ],
        [
         "18",
         "4190-MFLUW",
         "Female",
         "0",
         "Yes",
         "Yes",
         "10",
         "Yes",
         "No",
         "DSL",
         "No",
         "No",
         "Yes",
         "Yes",
         "No",
         "No",
         "Month-to-month",
         "No",
         "Credit card (automatic)",
         "55.2",
         "528.35",
         "Yes"
        ],
        [
         "19",
         "4183-MYFRB",
         "Female",
         "0",
         "No",
         "No",
         "21",
         "Yes",
         "No",
         "Fiber optic",
         "No",
         "Yes",
         "Yes",
         "No",
         "No",
         "Yes",
         "Month-to-month",
         "Yes",
         "Electronic check",
         "90.05",
         "1862.9",
         "No"
        ],
        [
         "20",
         "8779-QRDMV",
         "Male",
         "1",
         "No",
         "No",
         "1",
         "No",
         "No phone service",
         "DSL",
         "No",
         "No",
         "Yes",
         "No",
         "No",
         "Yes",
         "Month-to-month",
         "Yes",
         "Electronic check",
         "39.65",
         "39.65",
         "Yes"
        ],
        [
         "21",
         "1680-VDCWW",
         "Male",
         "0",
         "Yes",
         "No",
         "12",
         "Yes",
         "No",
         "No",
         "No internet service",
         "No internet service",
         "No internet service",
         "No internet service",
         "No internet service",
         "No internet service",
         "One year",
         "No",
         "Bank transfer (automatic)",
         "19.8",
         "202.25",
         "No"
        ],
        [
         "22",
         "1066-JKSGK",
         "Male",
         "0",
         "No",
         "No",
         "1",
         "Yes",
         "No",
         "No",
         "No internet service",
         "No internet service",
         "No internet service",
         "No internet service",
         "No internet service",
         "No internet service",
         "Month-to-month",
         "No",
         "Mailed check",
         "20.15",
         "20.15",
         "Yes"
        ],
        [
         "23",
         "3638-WEABW",
         "Female",
         "0",
         "Yes",
         "No",
         "58",
         "Yes",
         "Yes",
         "DSL",
         "No",
         "Yes",
         "No",
         "Yes",
         "No",
         "No",
         "Two year",
         "Yes",
         "Credit card (automatic)",
         "59.9",
         "3505.1",
         "No"
        ],
        [
         "24",
         "6322-HRPFA",
         "Male",
         "0",
         "Yes",
         "Yes",
         "49",
         "Yes",
         "No",
         "DSL",
         "Yes",
         "Yes",
         "No",
         "Yes",
         "No",
         "No",
         "Month-to-month",
         "No",
         "Credit card (automatic)",
         "59.6",
         "2970.3",
         "No"
        ],
        [
         "25",
         "6865-JZNKO",
         "Female",
         "0",
         "No",
         "No",
         "30",
         "Yes",
         "No",
         "DSL",
         "Yes",
         "Yes",
         "No",
         "No",
         "No",
         "No",
         "Month-to-month",
         "Yes",
         "Bank transfer (automatic)",
         "55.3",
         "1530.6",
         "No"
        ],
        [
         "26",
         "6467-CHFZW",
         "Male",
         "0",
         "Yes",
         "Yes",
         "47",
         "Yes",
         "Yes",
         "Fiber optic",
         "No",
         "Yes",
         "No",
         "No",
         "Yes",
         "Yes",
         "Month-to-month",
         "Yes",
         "Electronic check",
         "99.35",
         "4749.15",
         "Yes"
        ],
        [
         "27",
         "8665-UTDHZ",
         "Male",
         "0",
         "Yes",
         "Yes",
         "1",
         "No",
         "No phone service",
         "DSL",
         "No",
         "Yes",
         "No",
         "No",
         "No",
         "No",
         "Month-to-month",
         "No",
         "Electronic check",
         "30.2",
         "30.2",
         "Yes"
        ],
        [
         "28",
         "5248-YGIJN",
         "Male",
         "0",
         "Yes",
         "No",
         "72",
         "Yes",
         "Yes",
         "DSL",
         "Yes",
         "Yes",
         "Yes",
         "Yes",
         "Yes",
         "Yes",
         "Two year",
         "Yes",
         "Credit card (automatic)",
         "90.25",
         "6369.45",
         "No"
        ],
        [
         "29",
         "8773-HHUOZ",
         "Female",
         "0",
         "No",
         "Yes",
         "17",
         "Yes",
         "No",
         "DSL",
         "No",
         "No",
         "No",
         "No",
         "Yes",
         "Yes",
         "Month-to-month",
         "Yes",
         "Mailed check",
         "64.7",
         "1093.1",
         "Yes"
        ],
        [
         "30",
         "3841-NFECX",
         "Female",
         "1",
         "Yes",
         "No",
         "71",
         "Yes",
         "Yes",
         "Fiber optic",
         "Yes",
         "Yes",
         "Yes",
         "Yes",
         "No",
         "No",
         "Two year",
         "Yes",
         "Credit card (automatic)",
         "96.35",
         "6766.95",
         "No"
        ],
        [
         "31",
         "4929-XIHVW",
         "Male",
         "1",
         "Yes",
         "No",
         "2",
         "Yes",
         "No",
         "Fiber optic",
         "No",
         "No",
         "Yes",
         "No",
         "Yes",
         "Yes",
         "Month-to-month",
         "Yes",
         "Credit card (automatic)",
         "95.5",
         "181.65",
         "No"
        ],
        [
         "32",
         "6827-IEAUQ",
         "Female",
         "0",
         "Yes",
         "Yes",
         "27",
         "Yes",
         "No",
         "DSL",
         "Yes",
         "Yes",
         "Yes",
         "Yes",
         "No",
         "No",
         "One year",
         "No",
         "Mailed check",
         "66.15",
         "1874.45",
         "No"
        ],
        [
         "33",
         "7310-EGVHZ",
         "Male",
         "0",
         "No",
         "No",
         "1",
         "Yes",
         "No",
         "No",
         "No internet service",
         "No internet service",
         "No internet service",
         "No internet service",
         "No internet service",
         "No internet service",
         "Month-to-month",
         "No",
         "Bank transfer (automatic)",
         "20.2",
         "20.2",
         "No"
        ],
        [
         "34",
         "3413-BMNZE",
         "Male",
         "1",
         "No",
         "No",
         "1",
         "Yes",
         "No",
         "DSL",
         "No",
         "No",
         "No",
         "No",
         "No",
         "No",
         "Month-to-month",
         "No",
         "Bank transfer (automatic)",
         "45.25",
         "45.25",
         "No"
        ],
        [
         "35",
         "6234-RAAPL",
         "Female",
         "0",
         "Yes",
         "Yes",
         "72",
         "Yes",
         "Yes",
         "Fiber optic",
         "Yes",
         "Yes",
         "No",
         "Yes",
         "Yes",
         "No",
         "Two year",
         "No",
         "Bank transfer (automatic)",
         "99.9",
         "7251.7",
         "No"
        ],
        [
         "36",
         "6047-YHPVI",
         "Male",
         "0",
         "No",
         "No",
         "5",
         "Yes",
         "No",
         "Fiber optic",
         "No",
         "No",
         "No",
         "No",
         "No",
         "No",
         "Month-to-month",
         "Yes",
         "Electronic check",
         "69.7",
         "316.9",
         "Yes"
        ],
        [
         "37",
         "6572-ADKRS",
         "Female",
         "0",
         "No",
         "No",
         "46",
         "Yes",
         "No",
         "Fiber optic",
         "No",
         "No",
         "Yes",
         "No",
         "No",
         "No",
         "Month-to-month",
         "Yes",
         "Credit card (automatic)",
         "74.8",
         "3548.3",
         "No"
        ],
        [
         "38",
         "5380-WJKOV",
         "Male",
         "0",
         "No",
         "No",
         "34",
         "Yes",
         "Yes",
         "Fiber optic",
         "No",
         "Yes",
         "Yes",
         "No",
         "Yes",
         "Yes",
         "Month-to-month",
         "Yes",
         "Electronic check",
         "106.35",
         "3549.25",
         "Yes"
        ],
        [
         "39",
         "8168-UQWWF",
         "Female",
         "0",
         "No",
         "No",
         "11",
         "Yes",
         "Yes",
         "Fiber optic",
         "No",
         "No",
         "Yes",
         "No",
         "Yes",
         "Yes",
         "Month-to-month",
         "Yes",
         "Bank transfer (automatic)",
         "97.85",
         "1105.4",
         "Yes"
        ],
        [
         "40",
         "8865-TNMNX",
         "Male",
         "0",
         "Yes",
         "Yes",
         "10",
         "Yes",
         "No",
         "DSL",
         "No",
         "Yes",
         "No",
         "No",
         "No",
         "No",
         "One year",
         "No",
         "Mailed check",
         "49.55",
         "475.7",
         "No"
        ],
        [
         "41",
         "9489-DEDVP",
         "Female",
         "0",
         "Yes",
         "Yes",
         "70",
         "Yes",
         "Yes",
         "DSL",
         "Yes",
         "Yes",
         "No",
         "No",
         "Yes",
         "No",
         "Two year",
         "Yes",
         "Credit card (automatic)",
         "69.2",
         "4872.35",
         "No"
        ],
        [
         "42",
         "9867-JCZSP",
         "Female",
         "0",
         "Yes",
         "Yes",
         "17",
         "Yes",
         "No",
         "No",
         "No internet service",
         "No internet service",
         "No internet service",
         "No internet service",
         "No internet service",
         "No internet service",
         "One year",
         "No",
         "Mailed check",
         "20.75",
         "418.25",
         "No"
        ],
        [
         "43",
         "4671-VJLCL",
         "Female",
         "0",
         "No",
         "No",
         "63",
         "Yes",
         "Yes",
         "DSL",
         "Yes",
         "Yes",
         "Yes",
         "Yes",
         "Yes",
         "No",
         "Two year",
         "Yes",
         "Credit card (automatic)",
         "79.85",
         "4861.45",
         "No"
        ],
        [
         "44",
         "4080-IIARD",
         "Female",
         "0",
         "Yes",
         "No",
         "13",
         "Yes",
         "Yes",
         "DSL",
         "Yes",
         "Yes",
         "No",
         "Yes",
         "Yes",
         "No",
         "Month-to-month",
         "Yes",
         "Electronic check",
         "76.2",
         "981.45",
         "No"
        ],
        [
         "45",
         "3714-NTNFO",
         "Female",
         "0",
         "No",
         "No",
         "49",
         "Yes",
         "Yes",
         "Fiber optic",
         "No",
         "No",
         "No",
         "No",
         "No",
         "Yes",
         "Month-to-month",
         "Yes",
         "Electronic check",
         "84.5",
         "3906.7",
         "No"
        ],
        [
         "46",
         "5948-UJZLF",
         "Male",
         "0",
         "No",
         "No",
         "2",
         "Yes",
         "No",
         "DSL",
         "No",
         "Yes",
         "No",
         "No",
         "No",
         "No",
         "Month-to-month",
         "No",
         "Mailed check",
         "49.25",
         "97",
         "No"
        ],
        [
         "47",
         "7760-OYPDY",
         "Female",
         "0",
         "No",
         "No",
         "2",
         "Yes",
         "No",
         "Fiber optic",
         "No",
         "No",
         "No",
         "No",
         "Yes",
         "No",
         "Month-to-month",
         "Yes",
         "Electronic check",
         "80.65",
         "144.15",
         "Yes"
        ],
        [
         "48",
         "7639-LIAYI",
         "Male",
         "0",
         "No",
         "No",
         "52",
         "Yes",
         "Yes",
         "DSL",
         "Yes",
         "No",
         "No",
         "Yes",
         "Yes",
         "Yes",
         "Two year",
         "Yes",
         "Credit card (automatic)",
         "79.75",
         "4217.8",
         "No"
        ],
        [
         "49",
         "2954-PIBKO",
         "Female",
         "0",
         "Yes",
         "Yes",
         "69",
         "Yes",
         "Yes",
         "DSL",
         "Yes",
         "No",
         "Yes",
         "Yes",
         "No",
         "No",
         "Two year",
         "Yes",
         "Credit card (automatic)",
         "64.15",
         "4254.1",
         "No"
        ]
       ],
       "shape": {
        "columns": 21,
        "rows": 7043
       }
      },
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>customerID</th>\n",
       "      <th>gender</th>\n",
       "      <th>SeniorCitizen</th>\n",
       "      <th>Partner</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>tenure</th>\n",
       "      <th>PhoneService</th>\n",
       "      <th>MultipleLines</th>\n",
       "      <th>InternetService</th>\n",
       "      <th>OnlineSecurity</th>\n",
       "      <th>...</th>\n",
       "      <th>DeviceProtection</th>\n",
       "      <th>TechSupport</th>\n",
       "      <th>StreamingTV</th>\n",
       "      <th>StreamingMovies</th>\n",
       "      <th>Contract</th>\n",
       "      <th>PaperlessBilling</th>\n",
       "      <th>PaymentMethod</th>\n",
       "      <th>MonthlyCharges</th>\n",
       "      <th>TotalCharges</th>\n",
       "      <th>Churn</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7590-VHVEG</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>1</td>\n",
       "      <td>No</td>\n",
       "      <td>No phone service</td>\n",
       "      <td>DSL</td>\n",
       "      <td>No</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Month-to-month</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Electronic check</td>\n",
       "      <td>29.85</td>\n",
       "      <td>29.85</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5575-GNVDE</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>34</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>DSL</td>\n",
       "      <td>Yes</td>\n",
       "      <td>...</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>One year</td>\n",
       "      <td>No</td>\n",
       "      <td>Mailed check</td>\n",
       "      <td>56.95</td>\n",
       "      <td>1889.5</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3668-QPYBK</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>2</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>DSL</td>\n",
       "      <td>Yes</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Month-to-month</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Mailed check</td>\n",
       "      <td>53.85</td>\n",
       "      <td>108.15</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7795-CFOCW</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>45</td>\n",
       "      <td>No</td>\n",
       "      <td>No phone service</td>\n",
       "      <td>DSL</td>\n",
       "      <td>Yes</td>\n",
       "      <td>...</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>One year</td>\n",
       "      <td>No</td>\n",
       "      <td>Bank transfer (automatic)</td>\n",
       "      <td>42.30</td>\n",
       "      <td>1840.75</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9237-HQITU</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>2</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>Fiber optic</td>\n",
       "      <td>No</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Month-to-month</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Electronic check</td>\n",
       "      <td>70.70</td>\n",
       "      <td>151.65</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7038</th>\n",
       "      <td>6840-RESVB</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>24</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>DSL</td>\n",
       "      <td>Yes</td>\n",
       "      <td>...</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>One year</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Mailed check</td>\n",
       "      <td>84.80</td>\n",
       "      <td>1990.5</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7039</th>\n",
       "      <td>2234-XADUH</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>72</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Fiber optic</td>\n",
       "      <td>No</td>\n",
       "      <td>...</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>One year</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Credit card (automatic)</td>\n",
       "      <td>103.20</td>\n",
       "      <td>7362.9</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7040</th>\n",
       "      <td>4801-JZAZL</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>11</td>\n",
       "      <td>No</td>\n",
       "      <td>No phone service</td>\n",
       "      <td>DSL</td>\n",
       "      <td>Yes</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Month-to-month</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Electronic check</td>\n",
       "      <td>29.60</td>\n",
       "      <td>346.45</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7041</th>\n",
       "      <td>8361-LTMKD</td>\n",
       "      <td>Male</td>\n",
       "      <td>1</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>4</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Fiber optic</td>\n",
       "      <td>No</td>\n",
       "      <td>...</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>Month-to-month</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Mailed check</td>\n",
       "      <td>74.40</td>\n",
       "      <td>306.6</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7042</th>\n",
       "      <td>3186-AJIEK</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>66</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>Fiber optic</td>\n",
       "      <td>Yes</td>\n",
       "      <td>...</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Two year</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Bank transfer (automatic)</td>\n",
       "      <td>105.65</td>\n",
       "      <td>6844.5</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>7043 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      customerID  gender  SeniorCitizen Partner Dependents  tenure  \\\n",
       "0     7590-VHVEG  Female              0     Yes         No       1   \n",
       "1     5575-GNVDE    Male              0      No         No      34   \n",
       "2     3668-QPYBK    Male              0      No         No       2   \n",
       "3     7795-CFOCW    Male              0      No         No      45   \n",
       "4     9237-HQITU  Female              0      No         No       2   \n",
       "...          ...     ...            ...     ...        ...     ...   \n",
       "7038  6840-RESVB    Male              0     Yes        Yes      24   \n",
       "7039  2234-XADUH  Female              0     Yes        Yes      72   \n",
       "7040  4801-JZAZL  Female              0     Yes        Yes      11   \n",
       "7041  8361-LTMKD    Male              1     Yes         No       4   \n",
       "7042  3186-AJIEK    Male              0      No         No      66   \n",
       "\n",
       "     PhoneService     MultipleLines InternetService OnlineSecurity  ...  \\\n",
       "0              No  No phone service             DSL             No  ...   \n",
       "1             Yes                No             DSL            Yes  ...   \n",
       "2             Yes                No             DSL            Yes  ...   \n",
       "3              No  No phone service             DSL            Yes  ...   \n",
       "4             Yes                No     Fiber optic             No  ...   \n",
       "...           ...               ...             ...            ...  ...   \n",
       "7038          Yes               Yes             DSL            Yes  ...   \n",
       "7039          Yes               Yes     Fiber optic             No  ...   \n",
       "7040           No  No phone service             DSL            Yes  ...   \n",
       "7041          Yes               Yes     Fiber optic             No  ...   \n",
       "7042          Yes                No     Fiber optic            Yes  ...   \n",
       "\n",
       "     DeviceProtection TechSupport StreamingTV StreamingMovies        Contract  \\\n",
       "0                  No          No          No              No  Month-to-month   \n",
       "1                 Yes          No          No              No        One year   \n",
       "2                  No          No          No              No  Month-to-month   \n",
       "3                 Yes         Yes          No              No        One year   \n",
       "4                  No          No          No              No  Month-to-month   \n",
       "...               ...         ...         ...             ...             ...   \n",
       "7038              Yes         Yes         Yes             Yes        One year   \n",
       "7039              Yes          No         Yes             Yes        One year   \n",
       "7040               No          No          No              No  Month-to-month   \n",
       "7041               No          No          No              No  Month-to-month   \n",
       "7042              Yes         Yes         Yes             Yes        Two year   \n",
       "\n",
       "     PaperlessBilling              PaymentMethod MonthlyCharges  TotalCharges  \\\n",
       "0                 Yes           Electronic check          29.85         29.85   \n",
       "1                  No               Mailed check          56.95        1889.5   \n",
       "2                 Yes               Mailed check          53.85        108.15   \n",
       "3                  No  Bank transfer (automatic)          42.30       1840.75   \n",
       "4                 Yes           Electronic check          70.70        151.65   \n",
       "...               ...                        ...            ...           ...   \n",
       "7038              Yes               Mailed check          84.80        1990.5   \n",
       "7039              Yes    Credit card (automatic)         103.20        7362.9   \n",
       "7040              Yes           Electronic check          29.60        346.45   \n",
       "7041              Yes               Mailed check          74.40         306.6   \n",
       "7042              Yes  Bank transfer (automatic)         105.65        6844.5   \n",
       "\n",
       "     Churn  \n",
       "0       No  \n",
       "1       No  \n",
       "2      Yes  \n",
       "3       No  \n",
       "4      Yes  \n",
       "...    ...  \n",
       "7038    No  \n",
       "7039    No  \n",
       "7040    No  \n",
       "7041   Yes  \n",
       "7042    No  \n",
       "\n",
       "[7043 rows x 21 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\"data/churn_data.csv\")\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LLM Setup\n",
    "\n",
    "Run the code to set up your OpenAI API Key and set up key inputs for the LLM model creation and H2O AutoML model creation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ChatOpenAI(client=<openai.resources.chat.completions.completions.Completions object at 0x7f8f19ac6f50>, async_client=<openai.resources.chat.completions.completions.AsyncCompletions object at 0x7f8f19aed8a0>, root_client=<openai.OpenAI object at 0x7f8eb8460520>, root_async_client=<openai.AsyncOpenAI object at 0x7f8f19ac6fb0>, model_name='gpt-4o-mini', model_kwargs={}, openai_api_key=SecretStr('**********'))"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.environ['OPENAI_API_KEY'] = \"YOUR_OPENAI_API_KEY\"\n",
    "os.environ[\"OPENAI_API_KEY\"] = yaml.safe_load(open('../credentials.yml'))['openai']\n",
    "\n",
    "# Define constants for model, logging, and paths\n",
    "MODEL    = \"gpt-4o-mini\"\n",
    "LOG      = True\n",
    "LOG_PATH = \"logs/\"\n",
    "MODEL_PATH = \"h2o_models/\"\n",
    "\n",
    "# Initialize the language model\n",
    "llm = ChatOpenAI(model=MODEL)\n",
    "llm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the Agent\n",
    "\n",
    "Run the code to create the AI Machine Learning Agent that makes 30+ machine learning models in 30 seconds.\n",
    "\n",
    "- `model` - The LLM to use.\n",
    "- `log` and `log_path` - Set to `True` to log the Python function (pipeline) to a file in the `LOG_PATH` directory. (/logs by default)\n",
    "- `model_directory` - The directory to save the models. (/models by default)\n",
    "- `enable_mlflow` - Set to `True` to enable MLflow tracking. This is required to complete [the MLflowToolsAgent tutorial](/examples/ml_agents/mlflow_tools_agent.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAIrCAIAAABu+MUIAAAAAXNSR0IArs4c6QAAIABJREFUeJzs3WdYE9nbBvCTBgkh9N4FkWIDATsq9i6yiqLY0LXXVXct6Fqxt7Vhx7agWLA3RFRQFHRBFERFEOkk9JKQ9n6Y/Wd96WrITIbnd/GBTCYzT5Lh5kw7hyKVShEAADQ/Kt4FAABaCogbAICCQNwAABQE4gYAoCAQNwAABYG4AQAoCB3vApSYVILyvgjKS4WVpWKxSFrNl+BdUZOoMKlMNSpbk87RomsbqeBdDmhBKHDdzfeSilHSy9LPieUZ7yvNbFkqLBpbg6alryKoEuNdWlOV8kQVJSJVNVpeRpV1O/VW7dimrVl4FwXID+Lm+8Q9KEqKKTW3Y1m3U7d0VMO7nJ9VyhN+TqwozK0u5gp7jNA1tGTiXREgM4ibpvqSVHn3TG4Hd81uw3TxrkX+sj5VPbvBNbBg9v5FH+9aAGlB3DTJq/CigqzqvuP0VZhkPrj+JbkyIiTP53dLJpvMbxPgBeKmcfGRxVUVYlI2amqrKBEHb/8yZa0VQxUSB8gZxE0jHl3MV2FSe4zUw7sQhTq5Ns37N3N1LThxCeQJ/oM1JDGqhEqltLSsQQhNXGEZvD0D7yoA2UDc1Cs3jZ+fKeg9piUeOlVVo46YafowOB/vQgCpQNzU68nVgvY9NPGuAjdGVqr8SvHnxAq8CwHkAXFTt08J5RwdhoG5Kt6F4Kn7CL1nN7h4VwHIA+Kmbh9elfdU1CGbnJyc7OxsvF7eAG0Dho0T5+Pr8uZYOGiBIG7qwMupLuFWc3QUcV4mMzNz5MiRSUlJuLy8UUaWqimvy5pp4aClgbipQ9rbilbt2IpZl0gk+rFrEbBX/fDLm6hVW3b6uwoEF0sAeYDrbupw51Su6wAdfTM53y3N5/O3bt365MkThJCzs/OyZcukUunIkSNlMwwfPnzdunXV1dXHjh27d+9eXl6enp7esGHDZs2aRaPREELe3t42NjY2NjYhISF8Pv/UqVM+Pj41Xi7fmhFCkaEFVg5sq3ZKf4MYwB1cx1WHzI+VfccZyH2xp06dunnz5uzZs/X09G7evMlisdTU1DZt2uTv7z979mxXV1cdHR2EEI1Ge/HiRa9evczMzFJSUk6ePKmhoeHr64st5Pnz53w+f8+ePZWVlZaWlrVfLncqTGphvsAKQdyAnwVxU5NYKBUJpapq8t/NzM7OZrFYU6dOpdPpnp6e2ER7e3uEkJWVlZOTEzaFRqOdPn2aQqFgDzMzMyMiImRxQ6fTAwICWCxWfS+XOzUOrbxY1EwLBy0KHLupqaJUzNagNceShwwZwufzFyxY8OnTp4bnLCws3Lp1q6enZ9++fVNTU3k8nuypdu3aybJGMdQ0aBVlEDdADiBuapJKpEy1Zomb7t2779u3j8fjjR8/ftOmTSJR3X/DPB5v4sSJL1++nDNnzv79+x0cHMTi/zruUnDWIIRoNCqNRlHwSgEpwc5UTWoa9KL86mZaePfu3bt27RocHLxnzx5jY+Pp06fXnufy5cuFhYVBQUFGRkYIISMjoy9fvjRTPU1RXiIkd7cbQGFgM6qJoUpBCAkF8j9hV11djRCiUqkTJ07U19d///49QojJZCKECgoKZLMVFxdra2tjWYM9bODsYe2Xy11lmViNA/+WgBzAZlQHSwd2RYlIy4Ah38WGhIQ8fvx46NChBQUFBQUFjo6OCCFDQ0NTU9Nz586xWKySkpLx48e7urpevHjx8OHDHTt2jIiIiI6OlkgkxcXFWlpatZdZ++WqqvK+8UKKtPTl/FGAlglaN3XQ0GWkJsr/yn0zM7Pq6uo9e/aEhYWNHz9+0qRJCCEKhRIQEMBms3fu3Hnjxo3CwsK+ffvOmDEjNDR09erVQqEwKCjIysrqwoULdS6z9svlXnZidImFHZwFB3IAl/nVISeN/+wG95eFZngXgr+sT1Uv7hZ6zTfFuxBABrAzVQfjVkw6g1LNl6ow6z0j4+npWVxcXHt6hw4d3rx5U3u6pqbmtWvX5F1pTVFRUf7+/rWnS6VSqVRKpdbRmL13714D+1+56fw2nTjyLhO0UNC6qdubpyVF+dUNDEuQm5srkXzHOHZUKlV29Lf58Pn8OvenJBKJRCKh0+v472JsbCy7pLCG6ipJ0Ib0mVusm6FS0BJB3NTr1Lr0sYvNWnJ/vY8u5uubqbbr3nL7GAPyBYeK6+Xuqf/maQneVeCmvEhcWSaGrAFyBHFTr9ZObIlYGh9ZxwGaliB455d+PoZ4VwFIBeKmIT099dKTKj60vO7sLu3NHDrNmNkM96mClgyO3TTuwbk8C3s1O9eWcoLm0t7MfhMMteV9lSMA8O+rcQN8Db8kV764I/8r6IimtFB0bNXnbsN1IWtAc4DWTVPFRxbHPy7uPkKXlNeh8Cskz25w+RXifj6GzdHXDwAQN9+nvFgUfYPLL5e0asdu1Y7N0SbDOfKvKZW56YKEJ0XdR+g5dtXAuxxAZhA3342bXZ0UU/I5sYKpRjWxVlNVo6pp0DhaDJHoO676w5FEgsqLhJVlYgoFvYkqMWnFtHXWcOxKwiYbIBqImx/HzarOzxCUlwory8RUKqooFTfhRd8hOTnZxMREU1POV74w2TQVVQpbg66hy7CwV6MzoOssoCAQN8Q1d+7cKVOmdOnSBe9CAJAPOCgIAFAQiBsAgIJA3BCXsbExNpodAOQAcUNcOTk5347BAICyg7ghLjU1tTo7xAJAScHWTFyVlZXf1YMXAAQHcUNc2tracOwGkAnEDXEVFRXBsRtAJhA3xGVqalpn78IAKCmIG+LKysqqbxxxAJQRxA0AQEEgboiLzWbXNyQLAMoI4oa4Kioq4AZaQCYQN8SloaEBrRtAJhA3xFVaWgqtG0AmEDcAAAWBuCEuAwMDuO4GkAnEDXHl5+fDdTeATCBuAAAKAnFDXCYmJrAzBcgE4oa4srOzYWcKkAnEDQBAQSBuiAt2pgDJQNwQF+xMAZKBuAEAKAjEDXHBwC+AZCBuiAsGfgEkA3EDAFAQiBvignGmAMnA1kxcMM4UIBmIG+KCO8IByUDcEBfcEQ5IBuIGAKAgEDfEpampCdfdADKBuCGukpISuO4GkAnEDXEZGxvDoWJAJhA3xJWTkwOHigGZQNwQl6mpKRy7AWQCcUNcWVlZcOwGkAnEDXHp6OjATQyATCgwTiPRDBo0SEVFhUKhFBcXs1gs7HcVFZVLly7hXRoAPwVOfBAOi8XKzMzEfq+srEQIUSiU6dOn410XAD8L2uqEM2LECAqF8u0UMzOz8ePH41cRAPIBcUM43t7epqamsocUCmXQoEFaWlq4FgWAHEDcEA6Hwxk6dKjsobm5OTRtADlA3BCRj4+PlZUV1rQZOHAgNG0AOUDcEBHWwKHRaObm5uPGjcO7HADkg3BnpirLxNys6mpBS7+8zc1xRHSrVBcXF14Gg5dRjnc5eKJQkIYOQ8dIhUanNGF2QFwEuu5GUCl5GJKXk8a3sGfzq6DTTPAvJovGzeJTqcjOjePUG/YrlRhR4qaqXHzlQFZPTyMdYxW8awEEFXOzQEuf7jZQG+9CwA8iyrGb81u+DJpiClkDGtB1uH5xgfCfyGK8CwE/iBBx8zqiuL27jqoa3P0MGtF1uEFKbJlYRIgmOfhehIibvC98tibhDloDYpJIpEV5QryrAD+CEHEjqpZwtGE3CjSJngmzrAjiRikRIm4qy8USCTSPQZMIqsQEOb8Bvhch4gYA0BJA3AAAFATiBgCgIBA3AAAFgbgBACgIxA0AQEEgbgAACgJxAwBQEIgbAICCQNwAABQE4gYAoCAQN8rh1u0wj36uPB735xdVUlLs0c/12vUfHJMzNzcnJzf758sALRDEDfgOWdmZE3xHpqQk4V0IUEpkiBu4P1hhxCIRfNrghyllp1YlJcWeXv1nz1r08VNKdHSkra39X3uPI4SuXb90MfQcl5tvZGTSr+/gcd6TVFVVEUJ8Pv/sueOPHt0v4OYbGhoPHDBs4oRpNBotKflt4JG9KSlJTCare7dec+Ys0eBoIIT81y61MLfiC/j379+USqWdnDv/4uVz7vyJt+8SdLR1p02dPWDAUITQpct/P3kaMXDAsNNnjpaUFNvYtJnuNzc8/E50dCSdwRg4YNjMXxfQaDSsgOMnDj6MuFtdLTA3s/T2ntTXYyC2hIhH98eOmXjixEFeIdfW1n7Zb/4WFlbY2/z4KWX/gR0pKUm6Onrm5paNfixNr6eJ/g4OCrt2saystHVru6lTZpmYmE2ZNgYhtH7DivUIDRo0fMXv6xp+dwcP7fbyGv/4cXh5eZmjQ/tZsxbZtXFACMXERB09vj87O9PIyGTkiDFeo2FwmxZBKeMGc+7ciVGjxu7aGYj9CQWdPhp66ZzX6PGWltZfv6ZfuHgmMytj1YoNYrF41erFiW/jvUaPb23TJv3L56+ZX2g0Wnr656XLZltZ2fy+/M+S4qJTQYH5+bm7dh7GFh4ccnr06HG7dx2JiYk6FRQY8yJq7pzfpk+fFxwctHX7Ojs7RywUEhPj6TT6urXb8vJzd+3etPz3eSOGe+3ceTgmJiro9BELC6thQz0lEslq/yW5udkTJ0zT0tKJj4/buGkVn181dMgohFBy8tuLF88uXeovEol27968Zdufhw+eRghlZKQv+W2mpobWrzPm02j0M2ePNeUzaUo9Tfx4X71+eez4gX79Bndx6/4y9llVZaWujt7qVZs2B/hPmzrb2clVW1sHIdTwu0MICaurN67fWcDNDzp95Lels44fC9HU0Fq34Q8rS+ulv/mnpX3i8Qp+dBMASkaJ48bRsf2M6fOw37ncgvN/n/Rfvbl3r37YFF1d/T17t8yftywuLuaf+Ljly9bI/gAw586foFKp27cd4KhzEEIcjkbA1rUJCa87duyEELK0bLVw/nKEUBtb+9t3wuzt2o729EYIzZu79GnUo/iEV7I2yNo1W7S0tNu27fAy9llMTNSSxSspFIpdG4f792++fv1y2FDPJ08j3iT+E3z+hp6ePkKof7/BVVWVl68Ey+rZvGmPjo4uQsjLa/yhw3tKSks0NTQDj+6jUqgHDwRpaWkjhKhU6t59W5vysTRaTxM/3tzcbITQ6FHebdt2wFpz2KeBELKwsGrf3gmb0ui7mz1rsZqamgNCdm0cfSd7Xr16YdSosQKBwN2974D+Q5pYDCAHJY6bTp06y35/9eqFSCTaHOC/OcAfm4IdYuAW5L+Mfaaqqjpo4PAaL49PeOXs7IZlDULIza0bQijlQxIWN6oqqrI5VVRU6QwG9ruBgSG2N/fts//+wlBhMBgUyr9Dr+npG2CzxcREiUSiCb4jZS8Ri8VstrrsIZPJwn4xNDRGCPG4BaoqqrGxz0eOHINlDUKITm/qN9VoPU3UtUtPDkcjYMuaBfOXd+3as77ZGn13MoaGRhYWVsnv386Zvbht2w7nzp9gMlkjhnupqEC/sS2FEseN7K8UIcQr5CKEAjbvNdA3/HYeExOzokKenq5+7WMWFRXlWpr/jVjE4WhgraSGV4r99TblcCmF8u8YXkVFPF1dvd07A799llZXfDDoDISQWCLmFXJFIpGxkUmja2k6WT1NpKurd+CvkwcP7165enG7dh3X+m/R1zeoPVvT3x32IZeVlVIolK0Bfx0/cSDwyN7QS+dW/rEBi3hAekocN9/CwgJr59d4Sl2dU1jEq/0SPT2D0tIS2cOiokJs5uaorbi4yNDQGDtu3RRYDmIl4cjCwmrblr9e/xO79s9l27av27njUO15vuvdcQvyzS2sEELq6uqLF63w9p60Zu1S/zW/XQq91/QPBygvMpwIRwg5O7tRKJSrYRdkU6qqqmRPVVVVPYy4J3tKJBIhhNq27RCf8IrP52MTnzx5iBCSHZKQo06dOovF4us3/rusTlZbfdhstqmpeeTjcKFQ/kMO0OkMhFBZWWmjc1ZXVyOEOjm7de3q/uHje4SQqioT292TzdP0dxcf/yorO7OtYweEkEAgQAiZGJt6jR5fXlGOe7ACxSBJ68bM1Nxr9PjLV4JX+S/p2aMPj8cNu3ZxS8C+Nrb2A/oPDbt2ceu2P9+/f9faps3ntE+vXr84Gnjed4JfRMS9P1YuGDH8l/z83NNnjjo7uTp1dJF7bQP6D71x80rgkX05udltbO0/ffoQFf0o6OQlJpPZwKumTJ4ZsGXN/AXTBg8eSaVSL18Jllc9bDbb1MTsYug5TU2tEcO96pst+f279Rv+8BzlzWKpvXz5zN7OETt0ZWJsevHSOSaLVVpa4jV6fKPvbs/eABeXLtnZmZevBOvo6I72HCcUCqdM+6VP7wGtrGyuXQtVZ6tjR8oB6ZEkbhBC8+b+ZmBgePXqhdjY57q6eu49PfT1DBBCqqqqu3YGHju2/0H47Zu3rhgZmXj0GSgSiczMLLZvPXD0+P7tO9azWGoD+g+dPWux7MCqHDEYjB3bDh47vj8i4t7Nm1fMzCxGjhjT6KHfAf2HlJeXXbx49sjRfVaW1o6O7b9+/SKvklav3rz/wI579282EDcqDBVLi1Z//31KKpV2dHJZOP937ACQv3/A9h3rDxzcaWBg5NFnoJGRccPvTiQSBR7ZV10t6NjRZc6sxWw2u7Ss1NnJLfzhnYqK8latWgds3gtHi1uI7zt82Ewu7PraeYiBninsvZMKdpnfrRtP1NTU5LjYyAs5bbtxrNvXcfILEBx5WjctQXl5uc/Emmf0MbNmLho+bPT3LvDY8QPfHnaR0eBonj937YdqBKBeEDfKRE1N7eiRv+t8SoOj+QML9PaeNLyu/SkqhSTnEAChQNwoEyqVKt+LcTQ1NDU1fiSnmmLMLxPG/DKhmRYOlBH8EwMAKAjEDQBAQSBuAAAKAnEDAFAQiBsAgIJA3AAAFATiBgCgIBA3AAAFgbgBACgIxA0AQEEIETda+ir435YOlARTnc5QIcR2C74XIb42VTUqL5uPdxVAOWS8L9cxhr5KlBIh4qaVI7skvxrvKoASKOFWG1kw2RrfMTgfIA5CxI2lo5oahxp7l4t3IYDQJGIUeSG3z1h9vAsBP4gQvflhnt8qLC8W65sz9U2Z0N0KkKFSKaWFwrIiYcyt/KlrW7E1oWmjrAgUNwihtMSKTwnl1QIJLxv2rVBlZaWqqgqN1tL7JFLXolNpyNia1WWwDt61gJ9CrLgB35o7d+6UKVO6dOmCdyEAyAfstAAAFATiBgCgIBA3xGVqatrocFQAKBGIG+LKysrCxhcGgBwgbojL0NCQRoOTvoA8IG6IKy8vTywW410FAHIDcUNcxsbG0LoBZAJxQ1w5OTnQugFkAnFDXAYGBlQqfEGAPGBrJq78/HyJRIJ3FQDIDcQNAEBBIG6Iy9DQEC7zA2QCcUNceXl5cJkfIBOIGwCAgkDcEBeDwaBQKHhXAYDcQNwQl1AohN6IAJlA3BAXm82G1g0gE4gb4qqoqIDWDSATiBsAgIJA3BCXjo4O3MQAyAS2ZuIqLCyEmxgAmUDcAAAUBOKGuIyMjOAmBkAmEDfElZubCzcxADKBuAEAKAjEDXHBwC+AZCBuiAsGfgEkA3EDAFAQiBvi0tfXh5EYAJlA3BBXQUEBjMQAyATihrjgjnBAMhA3xAV3hAOSgbgBACgIxA1xwaC9gGQgbogLBu0FJANxQ1zGxsZwVTEgE4gb4srJyYGrigGZQNwQF3RAAUgG4oa4oAMKQDIQN8RlYmICrRtAJhS4kIxoBgwYQKFQaDRaaWkpk8mk0Wg0Gk1LSys4OBjv0gD4KfDPk3A0NTXT09Ox3wUCAUKIQqEMGDAA77oA+FmwM0U4vXr1qnGrlIWFxZgxY/CrCAD5gLghHE9PTysrK9lDqVTq6upqYWGBa1EAyAHEDeFYWFh0795d1sAxMzObOHEi3kUBIAcQN0Tk7e1tZmaGNW26d+8OTRtADhA3RGRqatq9e3epVGpqajp+/Hi8ywFAPkhyZqqYK6IgUp3RHz184vMnCd26dtNim5RwhXiXI08MBlVNE+50b4mU+7qb/K+C2PtFaW/LzdqolRSQ6m+SxNha9JKCaofOGt1H6OJdC1AoJY6bnDRBxMW8PmOMNfQYeNcCvk9VuTgzpSL9XdnoeaYU2KFvMZQ1bnLS+JGhBcNnmeNdCPhxGUkV7+OKf1lginchQEGU9T9L7P2ifhNN8K4C/BQLR7ahBet9bBnehQAFUcq44VdK8r5UsdThcKPSY7Jpuel8vKsACqKUcVOcX21ux8a7CiAHOkYq1QIJ3lUABVHKuJFKUVkRnIciA7EIlRdBnz4thVLGDQBAGUHcAAAUBOIGAKAgEDcAAAWBuAEAKAjEDQBAQSBuAAAKAnEDAFAQiBsAgIJA3AAAFATiBgCgIBA38lFeXv7h4/ufXMiIUX0OB+6tPb2iomLnrk2eXv2Hjeg1Z96UZ8+e/OSKfkZm1lePfq4PI+7hWANQUhA38jFj5vg7d64108KPHvsr/OGdQQOH+02dg6RS/7VLExPjm2ldADQfknSNLi9SqbTGCJZNVF1d3Qzl/GvypF/79xvSvr0TQqhXr37e44dGRUdiDwFQIi0obhIT40+fOZqUnIgQ6tjRZdrU2W1s7ff9te3xk4fLfvM/FLgnK+vrzh2HXDp1zsnNPnRo96vXL1RUVNvY2vv5zbW3c8SWcPbc8cS38Qghe7u2s2cvtmvjgBAaP2F4UVFh2LXQsGuhhoZGIX/fxNZ47fqli6HnuNx8IyOTfn0Hj/OepKqq2nCR5eVlm7esiY6O1NTQGj9+yqiRYxBCurp6urp62AwslhpCSCj8L93u3791PvhUdnamrq7esKGjJ06YRqU20mi9fefalashGRnp6uqc7t16Tfebq62tIxKJTgUF3rt/s6Sk2NKy1dQps3r26IPNX1xcdPDQruhnj1VUVJ2dXL9dVH2fFQC1tZS4iY2LWblqkY217exZiyUSyfPnT8Sif7tZqagoP3Hq0OJFK/j8qk7Objwed8FCP1NT8/nzllEolPv3by1aPCPw0NlWrWxyc7MF1YJJvjOoVOq1a6ErVi4MPn+DyWSu+3P773/Md+roMnbMRIaKCrbYoNNHQy+d8xo93tLS+uvX9AsXz2RmZaxasaHhOu/cvT5o4PAli1dFPLq3d9/WVlY2HTo4fzvDy9hnCCGXTl2wh/fu3dy6fV2/foOn+81NSko8eeowQmiS7/QGVhF0+sjpM8f69O4/9peJRcWFsbHP6QwGQmjnrk3hD+/4TvSzsrIJf3hnzdpl+/Yc69DBubq6etnvc7OyvnqP9TUyMrl2LVS2qAY+qx/9ogCpSZVQ9ueqi3u+lhZLm/4zevQvI0d6cvMFNaZv3LDVxcXlxfNE2ZQN67eM8/Yp5Aqxh4Vc4dChwzZv3lFaLC0pkshmexwZ6+Li8jD8Ofawf/8BGzdslT37+VN+ly5dblwPl005d/aSi4tL5teSBors3bv3qpV/Yr/n51b16d1n08Zt385QXCge5+3j6elVxBNh9QwaNHjq1OmyGdb4b3B3d8/NrqhvFakf8zp37rxixZoa09++SXNxcdm39zD2sKRIMmLEqBnTZ5UWS48fO+vi4hLxMObbOa9eudvwZ9XEnw//VF7+KxPvDQooSIto3RQW8jIy0mdMn6fyv6bHt5hMpoNDO9nDFy+i8wvyhg53l00RCoUF+XkIIQqF8jTq0cXQc1++pKmpqSGEigp5da7x1asXIpFoc4D/5gB/bAo24gW3IF+Do9FAqZqaWrKqTEzM8gvyvn027Frop9QPAZv20Gg0hFBmZgaXWzDOe5JsBje3brfvXMvMymhja193Ya9fiMXiUSPG1Jie8OY1QqhnTw/sIYVCcXPt+iD8NkLoadQja+vWri7/tqeotP+6iG7gswKgthYRN+XlZQghA33DOp/FjobIFBbxunVznzljwbcT2Wx1hNCZs8dPBQX+4uUzc8YCXiF3/YYVEmnd/ezyCrkIoYDNe2us1MTErOllU2k0sVgse5ifn3fi5MEePXp36/bvn3d5RTlCSEtLRzYPh6OBhVp9cVNYyEMI6df6KCoqyhFC2t8sSkNDs7KysqKiIj8/17a+pdX/WQFQW4uIGyaThf1tNGVmDkejpKTYwsKqxnSBQPB38KlhQz3nz1uK/fHXmOHbEbs4/2vC1F7OD/vrwHaJRLJg3nLZFCzLSkqKZVOKigq/XXtt6uoc7KMwMPh/iaOnZ4AQKi0t0dPTx6YUFvLodDqTydTS1MYWW1t9nxUAdWoR193o6xvo6xvcu39T9L/Dw1KpVCKpu2HSqVPnt28TUj4ky6ZUVVUhhPj8KoFA0KaNAzaxpLQYISRbCIvJ4vG4spc4O7tRKJSrYRdqLOSHRT4Oj45+PHnSr4aGRrKJurp6RobGL19Gy6Y8fhzOZDJbt7arbznYeaXbt8NkU7DPxMGhHYVCiXkRhU2srq6OeRHVtm0HGo1ma2ufkpL09euX2kur77MCoE60devW4V3DdysvFmWkVLZ2buggyLcoFIq2tu71G5dfvIgSCoUpH5L3H9ihqqJqY2P74kX0ly9p3x7+sLa2fRB++8GD22Kx+Gvml/PnTz5++rCvxyAmk/k0KiIpKVFPzyA5+e3efVsrKyuMDE06d+6OEPr4MeVpVASdTk//8plBZ1hatiorK7t//9aHj8kCgSDmRXTA1jXOzm6y89l1Cg4JsrW1d3Ptij28dTuMyWT27ze4vLx8lf9iobDawaFdYmJ8wpvXCW9ea2vraGpqcdQ1LoSeKyjIEwqFV66GhD+8M3GCn2wJtWlqavF4BTdvXU1PT62orIiLi9m67c8ePfqYGJvm5uZcDbuAEIXLLTh8eE9aeuryZWuNjU0trayvX78U8eieWCzOzs4MCTmdnZ3Zq1c/61at6/usmvxNovJiUX5GlUOXpn6VQKm1iJ0phFD/foOZTOaZM8cOB+7R1NRq08Zs/epgAAAgAElEQVTB1MyizjlNTcwO/HXy8JG95/8+SaFQbG3tR3uOw55aszpg2/Z1GzauNDOzmDNnSWrqh8uXg2fNXMhgMGbNXFhYyD177riWpvbcub9ZW7eeN/c3AwPDq1cvxMY+19XVc+/poa9n8GPFB50+gjWdzpw9LptoYWFlYWE1aNBwvoAfeun8/Qe39HT1Z/66YPy4yQ0vbcnilUZGJjdvXol+9lhfz8DNrRudRkcILV60gs1Wvxp2oaystJWVTcCmPZ2c3bAPZNvW/YGBe4NOHzHQN+zZ0yM2LqbRzwqA2pRyjPCcNH7UNe7gad9x2BUQU25aVeLTQi8YJrxlaCmtG4I4dvzA9RuXak/X4GiePye3W65iYqI2b/Gv86kDf52ytGwlrxUB8F0gbhTK23vS8OFetadTKfI8Zu/k5Hr0yN91PvXDO3QA/DyIG4XS1NDU1NBs7rUwmUxjI5PmXgsA36tFnAgHABABxA0AQEEgbgAACgJxAwBQEIgbAICCQNwAABQE4gYAoCAQNwAABYG4AQAoCMQNAEBBlDNuKIijw8C7CCAHFDpFQxe+ypZCKeNGz1jly7tyvKsAclCYLWCo/sg4gkAZKWXcMFSp5nbsskIh3oWAn1VVLjJrzcK7CqAgShk3CKFuw3QenMvGuwrwU5JflJQXVdt0hJEbWgql7M0PU1IgvPxXpvsvRhp6KmocWhNeAYiClyPI+lhZXlQ9aHLdo/EAUlLiuEEIVZSKX9zhpSdVcLQZhTkCvMuRM4lEQqFQKBSyHdrQ0FOhUJCDm0bH3s3e9Q8gFOWOGxkhX4rI9leJlixZMnHiRFdXV7wLkTM6gyLXzguB0iBJb34MJunCBiEJqqbSJXDiBpAG/JcBACgIxA1xGRgY0OkkaX4CAHFDaPn5+bJRhgEgAYgb4jI1NYXWDSATiBviysrKgtYNIBOIG+KC1g0gGYgb4oLWDSAZiBviYrPZ5LukGLRkEDfEVVFRQY5rvgHAQNwAABQE4oa4TExM4FAxIBOIG+LKzs6GQ8WATCBuAAAKAnFDXLq6ulQqfEGAPGBrJi4ejyeRSPCuAgC5gbgBACgIxA1xMZlM2JkCZAJbM3Hx+XzYmQJkAnFDXDQaDC8BSAXihrjEYjHeJQAgTxA3AAAFgbghLnV1dbgjHJAJxA1xlZeXwx3hgEwgbgAACgJxQ1wGBgZw3Q0gE9iaiSs/Px+uuwFkAnEDAFAQiBvigpEYAMlA3BAXjMQASAbiBgCgIBA3xKWqqgqX+QEygbghLoFAAJf5ATKBuCEuGIkBkAzEDXHBSAyAZCBuiAu6RgckA1szcUHX6IBkIG6IS0dHB1o3gExgayauwsJCaN0AMoG4IS5TU1PorhiQCcQNcWVlZUF3xYBMKHAhGdEMHTo0Pz9f9r1QKBSpVNqrV689e/bgXRoAPwVaN4Tj4uIikUgo/4MQMjY29vPzw7suAH4WxA3h+Pj4mJiYyB5KpdKOHTu2b98e16IAkAOIG8JxdHR0cnKS7UwZGRn5+PjgXRQAcgBxQ0QTJkwwMjLCfnd2dm7Xrh3eFQEgBxA3ROTo6Oji4oIQMjQ0HD9+PN7lACAfEDcE5ePjo6ur26FDB2jaANJQshPhnxMr3j4rqSoXF+ZW411Ls5Odn8K7kOalZ6oqqpZY2LN7jNTFuxbQvJQpbl4/Ks5NE7TqwNE3YdJVSf5H2HJQKJTiAkFZoSgyNOfXTdaqatDiJi2liZuoa9zKMkm3EQZ4FwKai1SK/t6S6re+lQoTEoeclON7zf0iKC8RQ9aQG4WCBviaPrnCxbsQ0FyUI24yP1Yy2dCNJvnpmzFT4krxrgI0F+WIm8oysYE5C+8qQLOjUFGrduq8bPKfB2iZlCNuKopFYhH0/NIilHCFEiU5ngi+l3LEDQCABCBuAAAKAnEDAFAQiBsAgIJA3AAAFATiBgCgIBA3AAAFgbgBACgIxA0AQEEgbgAACgJxAwBQEIgbAICCQNw0l6TktwKB4GeWcOny3x79XCsrK2s/FRMTNXPWxEFDuo/zGbZ339aS0pKfWdFP2hTgP3nqLzgWAJQFxE2zuHvvxrz5U/n8quZYeEFBvv/apQwVlVm/LuzTe8Ct22GbN69ujhUBIF/QZ1Wz+Ml2TcP09Q3+XLu1e7deNBoNIVRRUX7rdlh5ebm6unrzrRSAn0fauOHz+cdPHHwYcbe6WmBuZuntPamvx0CRSDRrji+dRj908DSNRhMKhbPnTlJVZe7fd4JGo+XkZh86tPvV6xcqKqptbO39/Oba2zliS0tMjD995mhSciJCqGNHl2lTZ7extV+waDqLydq+7QA2z4WLZwOP7Lt7O/pR5P29+7YihDy9+iOE/vj9z8GDRiCE/omPO3b8QGrqB21tHWcntxnT5+nq6jX6Rp4+jfg7JKigIK99O6dlS9fo6xsghNx7eshmYDJZCCGxWIQ9TEp+G3hkb0pKEpPJ6t6t15w5SzQ4Gg2vIi8v9/jJg7GxzysrK2xs2niP9fXoM6DhRUU8un/6zNG8vBwrS2uJ5P91RXTt+qWLoee43HwjI5N+fQeP856kqqr6nd8eICdy7kxJJJLV/kueP38yccK0JYtXtW5tt3HTqtt3rtHp9KW/+X/8lHLt+iWEUNDpI9nZmatWbqTRaDwed8FCv9Kykvnzls2auVAoFC5aPCMtLRUhFBsXs2TprLKy0tmzFs/8daFELBaLRA2svUvnHt5jfRFCWzbv/Wvv8S6deyCEXr1++fsf860srZctXeM9xvfNm9e/LZvN5/MbfS9nzh7zGj1+6pRZ75LebNm6tvYMsXHPbVvbaWpqIYTS0z8vXTZbKBT+vvzPKZN+jYp6tH79Hw0vn8fjzlswNS4uZvy4yUuXrLZu1ZrLzW94UeEP727ctEpXR2/B/OVubt1SP3+ULS3o9NGjx/7q6zFw+bK1fXr3v3DxzK49mxt9j6CFIGfr5snTiDeJ/wSfv6Gnp48Q6t9vcFVV5eUrwUOHjHJ0aDd69LhTQYcN9A1DLpxZtPAPM1NzhNDZc8e1tXR27ThMp9MRQgP6D/Wd7Hnz9tUF85YdOLjTyMhk/18nVVRUEEKeo8Y2vHZtbR0TEzOEkINDOywFEEL7D+wYMdxr4YLfsYeurl2nTBsTG/f823ZKnXbtDDQyMkYIiUSiY8cPlJQUy5aJEHoa9SgjI33Vyo3Yw3PnT1Cp1O3bDnDUOQghDkcjYOvahITXHTt2qm/5Z84eKy4uOnn8goWFFUJo0KDhDS/K3r7tgYM7O3Rw3rH9ILY3l5X19VPqB4QQl1tw/u+T/qs39+7VD1uIrq7+nr1bFs7/HXb0AGnjJiYmSiQSTfAdKZsiFovZ7H+3+OnT5kZHR675c1mXLj1Gjvj3lMqLF9H5BXlDh7vLXiIUCgvy83JyszMy0mdMn4dlzY/Jzc358iUtK+vrzVtXv52en5/X6Gs1NDSxX6xbtUYI5RfkyeKmqqrq4KFd9naO/fsNxqbEJ7xydnbDAgIh5ObWDSGU8iGpgbh58TK6k7MbljXfqm9RQpGwpKR4zC8TsKxBCFH/98urVy9EItHmAP/NAf7YFGxYoeLiQogbQNq4KSri6erq7d4Z+O1EGv3fN6umptbXY1BwyGmv0f8Nv11YxOvWzX3mjAXfvoTNVs/Pz0UIGegb/mQ9CKEpk2f2cu/77XQdncaP3chQqFQsN2VTTpw8lJ+ft37dDtlImxUV5Vqa2rIZOBwNrNHRYG2FLp261J5e36LU1TkIISMjk9ov4RVyEUIBm/fW+LiMjU2b/jYBiZEzbjgcjeLiIkND4zoPUmZlZ14Nu6Cmprb/wI6jgedZLBb2kpKS4tr/5CsqyrEwqr2cRofTlQ0ZiP2JCgT82sv/Ye9Tkq6GXfAcNdaujYNsop6eQek31+AUFRXK1l4fdXVOne+uvkVhGVRcXFT7JZz/HUiW49sEZELOQ8WdOnUWi8XXb1ySTamq+vcSGKlUunPnRl1d/YP7g3i8gv0Hdshe8vZtQsqH5BovMTe31Nc3uHf/puh/h4elUil2LkZLUxv7f47Jzc2W/c5isr5tVpiZWRgaGt25e11WhkgkEgqFP/wGRSLRrl2btLS0/abN/XZ627Yd4hNeyY5AP3nyECHUvr1TA4vq5Oz2+vXLnG+Kx95pfYuysWlDpVLDH96pvShnZzcKhXI17IJsiuz9AoAQoq1btw7vGhr3Kb5cQ09Fy6CpR0+srGxi42Lu3b9ZUlpcVFR4997N/Qe2Dx/mRafTr12/dO166No1Wxwd22tp6Zw5e8zSslUrKxtra9sH4bcfPLgtFou/Zn45f/7k46cP+3oMolAo2tq6129cfvEiSigUpnxI3n9gh6qKqo2NbXlF+e0719TU1BgqKjduXr5yNUQikfhOnE6n05kstWvXQ9O/fKYgSlJyor2do6Gh8e3b1549fyKVoqSkxL/2bxeKhI6O7Rt4F0nJibGxzydOmMZgMLAjsuEP7wwfNlpPT//CxbMPwu/Y27UtKi5MePM64c3rjK/pdm0crCytL18Jjk94xWCoxLyIOnHqUIf2zlMm/9pAQ8zK0vrO3Wv3H9wSiURZWV9DQk6/evWie/de9S1KXZ1TUJB39+6NL18+V1VVvngRfe/+DRZLbbTnOA0NzbKysvv3b334mCwQCGJeRAdsXePs7NaU8/0yH16VWndgszXI2e5u4cgZNzQarU/vAeXlpZGRD548jaioLB8yeFT79k75+Xlr1y3r02fAeO9JCKE2tvafUlPCrl7o6zHIxNi0R/feXzLSHjy4FRv3nM1WHzbU08rKGiFkbd26des2CQmvHoTf/vAh2dTUvGdPD319AxtrW4GAf/3GpTt3r+nrGbq6dElMjMfiRoOjoa9vGBn54Pnzp2VlpYMGDbe0aGVv5/jmzT/3H9xKfv/Wxtp2wIBhDf8d1hc3FArlz/W/i0Si3Nzs+Pg47CcnN8tz1FgNDc327Zxj457fuHk55UOyR5+By5etbfiyF01NrW5d3dPSPj0Iv/369Usane7RZ6C1desGFuXi0qWiojz62ePY2GcUCoXD0aiqqhrtOQ47oqymxn7+/GnEo3uZWRk9uvfu3q0XtrvaRBA3JEaRKsMQYndO5ZrZqVu1hbMb5HfzyNf+Ew30TeHKQBKC/yF4Ki8v95k4vM6nZs1cNHzYaHmtaOHiGWlpn2pP796998o/1strLQA0DOIGT2pqakeP/F3nUxocTTmuaK3/FqGojiPT2CFtABQD4gZPVCrVuK4LWOQOu7oaAHyR80Q4AICAIG4AAAoCcQMAUBCIGwCAgkDcAAAUBOIGAKAgEDcAAAWBuAEAKAjEDQBAQZQjbljqVBoD7yKAQnB04JsmLeWIG1U1WlHuj3dGBZTI15QKnSb3NAKUi3LEjaE5s5ovbsKMQLmVF4nM7dg0RiO9sgIlpRxxY92BXcqr/pJUgXchoHk9vpTbqa9WE2YESkk5utdCCEml6NqhbHMHdZuOHBod/vuRTVW5+FFITo8RumZtoE8M0lKauME8DeO+jS4xsGCKqxVdtlAopDPoFARJV1N1dTWNTqNRaT/2co4u42tKhbEVy7mvlpktZA2ZKVncYIryhIIqhR7KuXnzZnFxsa+vr8LWWFxcvHHjxq5du44d28ignbh79+7d5cuXV6xYgRD6gcH/KBSKlj5DVU059uvBz1DKuFGkiIiIvn37fv361dzcXJHrPXjw4KlTpywtLQ8ePGhkZKTIVf8YiURSUlIyadKk5cuX9+7dG+9yABHBv5SGTJ06tby8HCGk4KzhcrkPHz5ECGVkZISGhipy1T+MSqVqa2sfO3assLAQIRQXF8flcpvwOtCCQNzU7dOnTwihVatWjRw5sgmzy9nFixczMjKwIfQiIyNzc3MVX8OPMTY2Hj16NEKIyWT6+vrGx8fjXREgEIibmsrKyry8vLB9zDZt2ii+AC6XGx4eLnuoRA2cb7Vr1+7u3buGhoYIoXXr1n37jkCLBXHz/1RUVLx//37Pnj22trZ41XDx4sWvX7/KHmINnJycHLzq+RnGxsYIoQkTJoSHh4vFYti9auEgbv7F4/GGDBlCo9Hc3NwsLS1xrCQ8PLzG8XslbeDItGnTZuvWrTQaTSgUurq6RkVF4V0RwAfEzb+Sk5NPnz7NZDLxLgSVlpYaGhoaGhoyGAxNTU1DQ0MDA4Pnz5/jXZccGBsbx8XFqaurI4QuXbr09u1bvCsCCtXST4R//Phxzpw5xDyyMHfu3ClTpnTp0gXvQppFcnLytm3blixZ0rFjR7xrAQrS0ls3d+/evXTpEt5V1E1XV5dG+8FLdYnPwcEhKCioVatWCCEfH5+7d+/iXRFodi00bhISEs6dO4cQWrBggZYWQe8JLCgowLuEZqehoYEQ2rFjx5cvXxBCWVlZfD4f76JAc2mJcZOWlrZv3z4vLy+8C2mEqakpnd4iRlU2MzObNWsWdhquf//+jx49wrsi0CxaVtxkZmbm5uYyGIyTJ0+qqanhXU4jUlNTW0jcyJiZmUVFRenq6iKETp8+jV1sCUijBcVNcnLy/Pnz9fT0zMzM8K6lSfh8vqqqKt5V4KBDhw4IobZt2/r7+wsEAti9Io0WFDdlZWVhYWFK1F4wNDRksVpuhwyurq4hISF0Op3H482cORNaOiRA/rhJTU0dNGgQQqhz58541/J9EhISCHsYW2FoNJqpqemsWbOwiwPT09Pxrgj8OPLHzY0bN27fvo13Fd9NJBJVVVVhV8QBFxeXqVOnIoTevn3r6+vL4/Hwrgj8CDLHTVBQEEJo8eLFynj1SnFxcffu3fGugnCGDx/u7++fl5eHEIqNjcW7HPB9SBs3PXr06NWrF95V/LjMzMyysjK8qyAie3t7R0dHrOezCRMm4F0O+A5Kc9y06dLT062srB4+fEiEG6B+WGZmprKcQcPLH3/8kZKSgn3jBQUFbm5ueFcEGkG21s3OnTvz8/Ox7p3wruWnQNw0hZ2dHXYK78SJE8HBwXiXAxpBqrjJzc01NTVVujNQdfr06ZONjQ3eVSgHFosVGBjYrVu32r0FAUIhSdxIpdKYmBgtLS0fHx+8a5GPd+/etW3bFu8qlImVlRVCqGPHjgsWLIBTV8REhrgRCoVubm5OTk7KvgMlU1BQIJVKDQwM8C5E+djZ2YWFhamoqBQXFx87dgzvcsD/U++hYolEothKflBZWRmXy3358mVTaqZSlSNeU1JSevTogXcVRPEDmyKbzUYIMRiMbdu2LV++vHnqIjQKhUKhEG4IxnrjhsfjEb/nLaFQiBDicDhN7ARXWdoL0dHR2EFQwOfzS0tLf+y1Q4cOxbqar6ysVFVVVcbLr36YhoYGARv7yvHfvj6VlZUMBgPvKuQvKiqqZ8+eeFdBHqqqqnARExEocdxIpVJNTU28q5C/1NRUFotlYmKCdyHkQaPRsLvPRCKRQCDAu5yWSynjRiQSVVRUEHDXVC5iY2P79++PdxXkRKfThUIh9GiBF+WLG4lEIhAIsGOBpBQWFta3b1+8qyAtdXV1FRUVhBA0cxTv++Lm6dOnM2fO9PLyOnv2LELo3r17Pj4+2FW8CkOlUrGsuXv37tChQ7ERqUnj7du3TCazdevWeBdCZtgJSqlUWlZWNmfOnK1btzb6EpFINGPGjOPHjyukwDqQY2v/jnum0tPTt2/f3r9//549exoZGSGEVFRU1NTUFHl2uaSkRENDg6y7UVjTxtPTE+8qWgQmk4mdZ2jKiXYKhcLhcFpm54py9B1xEx8fT6PRFixYIMsXDw8PDw+PZqutpoqKCjabTeKsweLG398f7ypaCtmp8dLSUmxMiAbm3LNnj6LqIq2mxs3KlSsTEhKwDkd69OixevXq3bt3Y6PBXb9+vaCgYN68eYMGDcL608/JyZk7d+7w4cOnT5/ewDL5fH5ISMjjx495PJ6BgUG/fv28vb1pNFphYeGxY8fi4uLEYrGjo+P06dOx0YjYbHZqampgYODHjx+1tbVr3MGYkJAQFBSUlpampaXVsWPHKVOm6Ojo/NyHo2jBwcGTJk3CuwolUOd3/fjx423btvn7+2P9BGEP161b17lz57Fjx7Zp04bP53/+/FlDQ6Nfv34TJkyQdSNLpVJZLJZYLBaJRMHBwY8fP+ZyuTo6On379vX19aXRaLm5uX5+fgihcePGTZkyJTU1ddmyZevXrz916lRaWpqBgYGfn1/Xrl0brvl7t3bsHCXJtvam7gdNmjTJ3d2dTqevWbNmzJgxCKGRI0fKjmgaGxv7+vreuHHj8+fPEolk9+7dxsbGDf/liMXidevWXblypUePHosXL+7Zs2dmZiaNRuPz+StXroyPj/fz85s/fz6Px1u1alVpaWllZeXXr1//+OMPHo83depULy+vbzuvjY+PX7NmjaWl5aJFi0aPHp2YmLhy5UqlOwFx8ODBmTNn4l0F0dX3Xffu3btz585Hjx7l8/mFhYWHDh0aPHiw7H7dzMxMT0/PzZs3e3h4XLx4scb9DQwGg0ajUSiU169fd+nSZcaMGU5OThcuXLh27RpCSEtLa82aNd/2ci0QCLZs2eLp6bl161YDA4Pt27eXlJQ0UPN3be3l5eUIIVJu7U1t3Tg6OsbGxlIoFOy+W4RQ69atLSwsZDOMGjUqMjLywIED3bp1S0lJ2bdvH3b8vz5RUVFv3rxZtGgR1pGwzKNHj75+/RoQEODk5IR1x+/n5xcaGjp16tQTJ05QqdTdu3dj11BQqdSDBw9irwoMDBwyZMicOXOwh506dZo1a9br16+VqEO806dPe3t7t+S+0Juoge967ty5s2fPDgkJSU9PV1dX//XXX2Wvcnd3d3d3x7bk0tLSO3fuTJw4scYOlIqKSkBAAIvFwnaycnJyoqOjvby8mExmt27dauzFz549u3fv3gihqVOnLly48O3btw3cd/JdW/v169cnTJhAyq1dbt1r0Wi0hQsXLl68+P3799OmTZM1COvz6tUrVVXV2heYvHnzhs1mY58+1pWJubn5169fhULh69evhw0bJustXLbjnZeXl5GRkZ2dXWPgV+UahTIwMPDx48d4V0F0DX/XBgYGU6ZMOXLkCJVK3bFjR33Z7erqevfu3dTUVGdn5xpPCYXCoKCgf/75B2tiNHC9hewWAezOmIbvQf+urf3Dhw98Pp+UW7s8e/Nr3bq1ra1tWlrakCFDGp25qKhIR0en9m0slZWVNa4VVldXLywsLCoqEolEhoaGdS4KITRhwoQa/16UYm8WExISMm/evIbbg6Ap33X//v1PnjxpbW3t4OBQ30KwEKmqqqq98AULFrBYrEmTJrHZ7LCwsKysrEZLasrpraZv7RwOh8RbuzzjJjIyMiUlhclkHjp06Pfff294ZnV1deyDq0FXV/f9+/eyh1VVVYWFhYaGhti3UlxcXOeisN1pc3NzebwPRcvOzj5//vyNGzfwLkQJNPpdnzx5kkajpaSk3L17d/DgwXXOg7VE9PT0aky/fft2cXHx7t27DQwMJBLJ06dPmxI3TSy7KVs7lib6+vpk3drldslMcXHxkSNHPDw8lixZEhkZ+fDhw4bn79ixI5/Pj4yMlE0RiUQIIQcHh7KyMtl38Pnz59zc3LZt26qpqZmYmDx9+hS7C/xbpqamBgYGDx48kP2/EolEtWcjrLVr127cuBHvKpRDw991fHz8nTt3Zs6cOWzYsKNHj9bZrZ9UKr1//766ujr258pgMGR3b5aWlmpqamI7R1Qqtby8XF7dsDRxa09LS8vOzibx1k5bt25dnU9UVlbWmJKQkJCcnPxtd3nv3r2Lj4/38fGhUql79uzJycn5888/7ezssrOzr1692qtXLw6HU9+Kzc3NX758ee/evbKysqKiooiIiFOnTg0ZMsTKyurJkyePHz9msVipqalHjx6l0+lLlixhsVgcDufevXtxcXEikejTp09hYWGlpaVeXl5qamoGBgb37t178eIFNjhvYGCgSCSyt7evsVIC3vpw+fJliUQyduxYvAshqBo3VVIolPq+66qqqrVr11pZWc2ZM6dDhw6RkZExMTEDBgyg0WihoaHZ2dlSqfTz588nT5588+aNr69vx44dEUIfPnyIioqqqKjo0KGDSCS6f/++WCwWCoWhoaHR0dGVlZU9evTADqCEhIQ4ODg4OTkVFRXduXOnT58+2MlpkUh08eJFFxeX2tubTBO39oMHD8pra1dVVSXggLHyiZtnz56dP39+9uzZ7dq1Qwg5OTmFh4e/evWqf//+9V1zTKfT3d3dS0pKoqKiYmJiSktL3d3d7e3tGQxGly5d0tPTb926FRsba2Njs3LlSmwntlWrVhoaGgkJCc+fP+dyua1bt/78+bOXlxeLxTI3N7e1tX337t3Dhw8/fPjQqlWrvn371t6bJVrcVFdXb9u2Da4fa0Dte7jr+66PHz/+5s2b9evXa2lp0el0Ozu70NDQiooKV1fX0NBQIyOjjx8/Yu2LcePGYRdzYL3/5ebmPnv2bMSIETY2NhKJ5NatW9HR0SYmJosWLXr37p1UKm3bti2VSv2ZuGnK1h4XF9e6desVK1bIZWsnZtxQ6utDC+u/UuH1/D+FhYXa2tpyvIyYaN1r/frrr3PmzOnUqRPehRDXz3SvJTN27NhBgwbNmDHjx14ulUolEoly9c5FzO61mjf/li9fXuegzl27dl26dGnDr5VKpeS+ZeHIkSNubm6QNcRHoVD4fD528XEDswUFBd26dav2dA6Hc/LkyeYsUGk0b9ysWLGizoNYTcldCoVC4jviXr169erVq6NHj+JdCGgSNpstEomkUmkD//+8vLzqPBemLD1kKwBxd6aqq6spFIp8+wYlzs6Uq6srdpU23oUQnVx2plogYu5METd3BQKBsowG8b2WLl168uRJyBqlU1VVVfsUCmg64lhkh7MAACAASURBVMaNiooKKbs937Rpk7u7e4cOHfAuBHw3FoslkUhwP4WivOo9doP7cRMCNgV/3vnz59XU1KADraaj0+mE2hIIVUwDiHkerd64abi3IQW4deuWi4sL1m0gOTx//vz58+cHDhzAuxBlQqfTcd8Uazhz5syQIUP09fXxLkT5EHdn6uHDhykpKXhXITf5+fmXLl2CrCEBbW1tWV8Q4LsQ7rpDmdGjR5NmrCWBQODp6fns2TO8CwFyMGLECENDQ5FIRMDLdgmu3hPhQI569+5969Yt7F5eAFos4u5MJScn37lzB+8q5MDT0/PcuXOQNWSSlJS0adMmvKtQPsSNGxaLheOwPvKyfv36TZs2KWPXJKABjo6Ot2/fhoHxvhdx48bKymrUqFFYtyBKas6cOUOGDMHukgckc/bs2dr9AYKGwbGb5rJo0aKxY8f27NkT70IAIAritm4QQv/880+N/p+VxbJlyzw9PSFrSOyff/7ZuXMn3lUoGULHjaWl5a5du/Cu4rvNnTt3yZIlihxfFCiejo4OXNnwvYi+M5WammpsbKympoZ3IU21dOnSYcOGyQb8AwDIED1ulAs2qmGfPn3wLgQAIiL0zhRm9uzZSpGJmzdvHjt2LGRNyzFv3rw6x2YB9VGCuGnduvWFCxfwrqIRU6dOHTx4MBwbblEyMjKg+5vvogQ7UyKRqKSkRFdXF+9C6jV27Ni1a9e2b98e70KAQuXm5urp6cGdU02nBJ8UnU4vKiqi0Wiy8ZIJZdKkSdu2bbO2tsa7EKBourq6xOxWhrCUYGcKG95w+fLl2O+9e/desGAB3hUhbPh6d3f3nTt3Qta0TD4+Pl++fMG7CmWiBK0bhFCnTp0+f/7cvXv3poy/oRhcLnf48OERERFKdJIeyIWLi4vs9zFjxmCHIzw8POCqv0YpQdwMHTpUNiwElUqVSqW4t2A/ffq0b9++mJgYfMsAuLCzs0tJSZH1bE+hUAwNDf38/PCuSwkQfWfK09OTx+PVOJ6N79i7cXFxq1ev3r9/P441ABz5+PjU6Mm7U6dOjo6O+FWkNIgeN2FhYR4eHjV2WHDsOyY8PPzYsWPEPzEPms+IESMsLS1lD42MjCZOnIhrRUqD6HGDENq6deu4ceO+PS2F1ygRly9ffvDgwZEjR3BZOyAOHx8fFRUVbGjpDh062Nvb412RclCCuMEu31y0aBE2BiaFQsHl6OyJEydSUlK2bdum+FUDohk5ciTWZZqRkdHkyZPxLkdpKEfcYC3YXbt2mZubS6VSxR+72bVrl0AgWLVqlYLXCwhr0qRJdDrdyckJmjZN1/hVxe9jy7JTq0QiaUmBUFFV1UsqlWZlZZmZmSlypQIBn88XaGpqNsfCNfUZdDrFxIZl78ZpjuXL1+c3FRkfKoUCSXE+/hsD7rKysvT19bG9qpaMrcWgM5CRFatDz0ZGBGswbqQo7HCWnhlLjUPXMVKVSIh+u4MyolIphbmCyjIRL4s/ag6hB7p5GJxPpVPVtRl6Jkzi3/sCFIZKo5QUVFeWiT/Fl4xfZsFQodQ3Z0Nxc+NIjrm9uo2TEvzXJYGPr0uzP1UM/9UY70Lq9uQKFyGqcz8dvAsBxFXCFUYEZ0/2t6xvhnrjJu5BkURCdejaLHsQoE5Jz0todIlrf228C6npfWxZ7heBywA9vAsBRJf1qTIrpbzfBIM6n633UHFKXJmxDf73CrQoJq1ZKbFleFdRhw+vywyt4F4N0DjT1mrJsaWonl2muuNGLEIMVYqWfks/BqZgWvoqdFWKmHhD3YiEUl0TJt5VAOVg5cguyKp7BK564kYoKSbAeagWqIQrFAkleFdREzdbQIWOFkDTVFWIhdV1b8NKc90NAEDZQdwAABQE4gYAoCAQNwAABYG4AQAoCMQNAEBBIG4AAAoCcQMAUBCIGwCAgkDcAAAUBOIGAKAgEDcAAAWRZ9xIJJITJw+N8R480rNvTEyUSCTynTz6cODeH1ua/9qls2b71p4ulUpDLpzxmTBi0JDuk6f+EnLhjESC5z2NmwL8J0/9BccCgExubk5ObjbeVfxHLBYnJsY36yqSkt8KBHXffo3Lchomz7i5eetqcMjpcd6TVq3Y0K6dE4VC4XA0mEw5d1zwKPLBkaN/tWvXcdbMRa1t2hw5+lfIhTPyXQVQRlnZmRN8R6akJOFdyH927Nq4e29A8y3/7r0b8+ZP5fOrCLKcRslz0N6Xsc86ObuNHfPfEF+HD56W4/Ix7j09Nq7f2bNnH4SQ1+hxHz6+f/w4fILPVLmvCMiLVCqVDXHbfMsXi0Ry70E5KzvTxNj0h4uvbub2grzaIwpo12DkFjf9BnTGdmo8+rkumL+8Wzf3CRNHIoR8J/pN95sb8ej+xk2rNqzf4d7TAyGEPdyyeW/Xrj0bXmzQ6aM3bl4Wi8V9evefO+c3FRUVBoOBZQ2GxWQJRf91zZOU/DbwyN6UlCQmk9W9W685c5ZocBrpHT4vL/f4yYOxsc8rKytsbNp4j/X16DMAIXT//q3zwaeyszN1dfWGDR09ccI0KvXfxmDEo/unzxzNy8uxsrT+dleOz+cfP3HwYcTd6mqBuZmlt/ekvh4Df/QTVWKRj8PXb1ixcf3OC6Fn379/5zN+it+0OfV9OJcu/33w0G4vr/GPH4eXl5c5OrSfNWuRXRsHbFH1faHTpnu3srKxsrK5cjVEIOAf+OvUjJk+CKH1G1asR2jQoOErfl/XQIV37l4PC7v4Oe0Ti6XW2a3b/HnLtLS0EUJCofDkqcPhD+9UVVV26NDpw4fkSb4zRo0cgxDKyc0+dGj3q9cvVFRU29ja+/nNtbdzxPb6zc0s6XT6zVtXRUJh1649Fy1coa6uvnX7ukeRD7C/CITQ3+evGxs11PV9fe90waLpLCZr+7YD2GwXLp4NPLLv7u3oR5H39+7bihDy9OqPEPrj9z8HDxrhv3Zpelqqra193KsYCoXapUuPubOXaGvrfO9yYmKijh7fn52daWRkMnLEGK/R4+SyYcgtbjas23H0+H5VFdXJk3+1trbV1tLZuGHn+g0rsGf7egx8EH774KFdbq7dKirK9+7bOnzY6Eaz5sPH96pM5qxfF378lHLp8t86OnqTJ834dgYut+Bz2iev0eOxh+npn5cum21lZfP78j9LiotOBQXm5+fu2nm4gVXweNx5C6aKxeLx4yZra+m8SfyHy81HCN27d3Pr9nX9+g2e7jc3KSnx5KnDCKFJvtMRQuEP724O8Hd2cvUe65ubm/13cJCpqTl26Gq1/5Lc3OyJE6ZpaenEx8dt3LSKz68aOmTUT3+6Smnf/m0z/Ob5TZtjZmrR6IcjrK7euH5nATc/6PSR35bOOn4sxNjIpOEvNDb2OV/AD9i0p7Kq0tzccvWqTZsD/KdNne3s5Ir9gTUgKSnRwsJqwIChRUWFV66GVFRWbNm8FyEUeHTf9euXZkyfp6dncDhwj0DAHzJ4JLadLFjoZ2pqPn/eMgqFcv/+rUWLZwQeOtuqlQ1C6GLoub4eAwM27834krZz9yZdXf3Zsxb5TvAryM/LyclauWIDQkhXp6Genn9g0+3SuYf3WN+Loee2bN7LZqubmVlg0wu4+SNHjvH2nvThQ/KJk4fS01IPHzpDp9f7l157OZWVles2/GFlab30N/+0tE88XkHDH2bTyS1uevToHXLxDIvJ6tnj36ZHzx59vm2FLl64Ytr0sWfPHf+c9kmDozF3zm+NLtPExGzPriM0Gm3gwGEZGWmRjx/UiJvgC6epVKqnpzf28Nz5E1Qqdfu2Axx1DkKIw9EI2Lo2IeF1x46d6lvFmbPHiouLTh6/YGFhhRAaNGg41jg/fvJg+/ZO/qs2IYR6ufctKysNuXD6Fy8fGo124ODODh2cd2w/SKPREEJZWV8/pX5ACD15GvEm8Z/g8zf09PQRQv37Da6qqrx8JbjFxs1oz3HY54m1dxr+cGbPWqympuaAkF0bR9/JnlevXpg7Z0nDXyiNTl+zOoDF+rdH7Ta29gghCwur9u2dGq3ttyWrZBsnnU4/d/6kQCCg0+k3b14ZNtRznPckbDPYHOCf+DbepVPns+eOa2vp7NpxGPu7HdB/qO9kz5u3ry6YtwwhZGZmsWrlRgqF4mDf9klURGzc89mzFpmZWWhqahUW8ZpSzw9sutraOiYmZgghB4d2mpr/DWltZWntPdYXIeRg35bNVt8c4P/y5bPu3Xs1fTlZ2ZkCgcDdve+A/kMarfy7yPPYTcMMDY2m+807cHAnlUr9a+9x2VbSAHW2OvYnjRCysrJJSk789tmPn1KuXQv9xcvH1OTfUe7iE145O7thXxhCyM2tG0Io5UNSA9/Zi5fRnZzdsKyRyczM4HILsG1Otqjbd65lZmWUlpaUlBSP+WWCrDDq/37BTsZN8B0pe5VYLGaz1Zvw2ZBTp06dZb83/cMxNDSysLBKfv+20S/UwaFdU7aiOgmF/9fencc1ceZ/AH8mMyH3QcJNRAqCCl5IQfC2rfVCxaO2HpW1W+2vv55bt5e1/txqr13b2q7dbdVeFu2h9T7wQvEWkGKlIpRLROQI5A7kmOT3R7YpqxCSMdfE7/vFHzCQJ9+Eyef1zDPPzGPauev7o8cOtrQ0sVhsi8WiVCqYTKbRaLT1VRFCtm80GjVC6OLFsy2tzVOzxnRtobWl2fY9m8W2h1d4eGRZ2WVX66Gw6zojPX0kQqj8WpmDuLlTVGR0cvKQ3K1fsNmc6Vmz3bhun/fiBiE06eGszzd+3K9f/+TkIa4+Fsdxs/mPm4aTJPnBB2slEmnO4mX2jTqdViz6Y9UUgUBoO+By0KxC0Z46fMRtG7U6LUJILP6jQ/6fplpblCoFQiiiu4NwhaJNKg35cN1n/1V2z53YgMfl/LF4g0tvjkAgtH3IHf9DOWyKWWO1Wle88WJF5dWcxcuSkoacPp3//Q9bLFaLSCTm8/hXrpTaTneUl5chhOLjEhBC7Yq2zMwxy558rms73cYlk2BaLKSrJVHYdZ3B5/ExDNN36F16FIZh773zyeYvNnz2+frtO3Jff/Wtu0w9O69+GDZu+oQgiPLysgMHd0+bmn03Te3c9X1FZfnq/3ufy/1jnw4JCVOrVfYfFYp2hBCf72hZPj5f0K5ou21jWGg4QkilUt7WlOD3UWelUnFnUwKBUKlUhIdHslisu3hlgcmlN0fe2tInJpbaP9QZly+XXCopfGPF2ocenIwQutlQb9uO4/j8+X/atHnD2rffCAkJ27N3+5zZ8/v06WurX6VS3tYLdoaTJ8scvNJez4s5eAq5vNVqtdr2Z5fa4fP5L77w2rx5j7+5avnKN1/6accRt/RxvDeruOTnon37dz7zv8tnzpi74dN19fV1lJtqarr15Vf/Tk8fOW7sg123JycPKb18qbOz0/bjqVPHEUKOj5yHp6SVlBR2nRtmNpul0pCI8MjCwrP2jQUFx9hsdr9+/ePjExkMxrHjh7ppang6SZJ79+2wb+no8PhEBrpw/s0pLb10s7EhOWmIq/9QFouNEGpzokegUivtYz32H21nGLNnzku7P0OhaNdqNW+sWPvsM8vt9ZeVXa6oLO+1/q7YbE57e5sz01AdvFKxKLitXW7/y6Yu+6qtf+egE3Tw0B6EkO3NdKkd26nxqMjo2bMe0+q0ts7m3fNS76ajo2PdujWDBw+bOmWm4YHJl0oK16xd8a9Pv2EymRRaW//Je52dnVJJyJZvN9u2DBiQnJ6WuWjBE/n5h199/bnpWXNaWpq+2bIxZdj9w4amOmjq8UVPnjt/6tnnlsye9ZhEIi0uvsDhcP+6fOWfcp567++r/7FuTVpaZklJ4ZmzJ3MWL+NwOBwOZ8rkGQcO7jYaDOnpI9va5BcvngkOltqGD/ft3/nZ5x/fampMTBhQVVV55uyJr7/c4faJjnTU65vz0fp3UlNHNDY2/LTzO4lEOiv7UYSQS//QsLDwqMjoH3fksjkctVo1e9ZjPfWkkgYODgoK2rR5w7Rps2pqftv23VcIodqaqugo2Zq3VwiFoszMsQghDGHNzU3h4REIoZzFyy5cOPPyK8/Me2RRcLCksPAcaSHXvvWB41c9dMjwQ3l7P/zoncGDhgkEQgcDKA5eaVpa5umPTvy4PXfYsPvPnSs4cHC3/VHJg4biOL7hX+umTJphMBpmTJ+DEKqtq960eYNMFlNWdvngoT0jRowaNGioS+1MmTwjZ8mc8eMm3hcbv2fPdj6PLxS6ZzVdfPXqbqYnkGZraYFy8GjXVo89lLeXSTAnTpxq3/Jt7heDkocOT0n792cf/Vxa/N47H4vFwQRBDBw4aNt3X+t0WttQVrfyTxzR63TTs2bbfrxUUnjt2q+LFj5x5szJb3O/QAhVVVWUlhbbvvg8fsaI0UKhaPCglKLi8/v2/1RRWT5h/MMv/3WV4967SCTOzBhTW1t19NjBkpJCnCAmjH84Lq5fv36JwcGS/BNHDuXtVSraFyxYsmjhE7buaGrqCJ1Oe/ZcQVHROdvM6Y6OjlnZj+I4Pn7cRK1WffLk0VOn83V67ZTJMwcPHmafreOMsrOKYePEBNODk+IouHRcMTBDTBDOVlV3vaag4Nis7Hn2MyYO3pyr5VeKis7HxMTuP7DrSlnp0KGpK1e8HRoahhBy8A/ds3d7sFgybtxD9ifFMCwpaUhh0bn8E4dvNTWOHjWhp8MuHo8XGxuXd3hf3uF9ZrP5jRVr5fKWsrLSSZOyFIq2/Qd2Hs8/fOp0fv6JI7t2/xARHhUfnygUCEeNHHe9vvbo0QNFxed5PP60qdmxsXF37qjFxRd+q7pmm3caF9dPo1Edz8+7/EuJSCRO7TJ2fhsHrzQ+LsFg6Ny7b8ehvD2hIeH3p464cqV00cI/EwQhFAhDQ8NPnjx6/vxpjUY9aVJW/okjer3OYDAcPLT71q2bD0+c9pcXXrcdBznfzujRExoa6s+cPXH6TL5UGvraK6sjI6Od31uqStUx/bmC4G56Et2vEW7ssHz9Vt381+Kcfw7gFt+9X5PzZiyL41+Xzm5cUTP7hVgW2yNV2ab5Hdh3quswnK+QJGk/56jWqF97/XmCID5Zv9nXdTlr5arlrS3Nn3+W68Ma8r5qGDVDGhXXzUC+L8+baLXa+Quzuv3VU8teyJo2i0bPAvzNps0bug4V2QkFoq25e3p61Acfvl1dXZmZOVYsDq6/UVdT89s09+0hz7/4ZG1t1Z3bR44c9/qrf3PXs/gzX8YNl8vd+Pm2bn8lFLjnWNFrzwL8zbx5j2f9foDTFQNz1EdLTx/Z0tL0085tJpMpMjJ68eNLu14DeJdWrXy36wU3dpTP6NOOL+OGwWA4voqERs8CKJs7Z8HcOQvc3qxIKBK5PsA5ftxD47uMB7mXbUa1R/U6eu1b/jVGAAAIYBA3AAAvgbgBAHgJxA0AwEsgbgAAXgJxAwDwEogbAICXQNwAALwE4gYA4CXdx40VITYX93oxALG5OHLz4iVuwObi/nWJOvBjTBaOoe73l+7jhsVh6LVms9H/dvyAZjZa9Rozi+t3XU6CwPRqsxN/CABSthj44u6vjupxz5YlcFVyoyerArdTyY2yBN/fhOFOkfEcVVs31xYCcBuTwcLi4jxX42bYeHHRYbetLwOcUZjXmjLBtVuaeUfaxOCiPNgZQO+Kj8iTM4Q93VGu595NP07qg5Jj3/rRAu+B7ei3jemTpNHx/ninUb6YmL40av/GBrMJjq9Bjy7sbw0OCxoypscL8bu/m59dZYnm13Nqk9EaFcfVa11ezgL0iiPAb1XrmUHYoFGihBS/XpSqsbrjwqF2g94iS+B16GBnAP/B5uEtNzpwHIsZwEl90FH3vJe4QQhZSNRyo1PZajIZe7+hPHAVMwgXhxJhfdgMmpwJbG0wtDeZjAaIG7Rx48Y5c+ZIpVJfF+JjOIEJJUxpFIvD6+UsR++312LgKCKWHRHrj5184H2hMlaoDBbSQgihWx9djB3ySGws3BPSWX53zhUAEKggbgAAXgJxAwBFHM69cktzd4G4AYAigeBuVyu/10DcAEBRS0uLr0ugGYgbACiC3o2rIG4AoEij0fi6BJqBuAEAeAnEDQAUyWQyDIMbAbkA4gYAihoaGnq9Bgh0BXEDAEUSicTXJdAMxA0AFLW3t/u6BJqBuAEAeAnEDQAUxcXFwVCxSyBuAKCopqYGhopdAnEDAPASiBsAKIqPj4eDKZdA3ABAUXV1NRxMuQTiBgDgJRA3AFAUGxsLB1MugbgBgKK6ujo4mHIJxA0AwEsgbgCgKCoqCg6mXAJxAwBFjY2NcDDlEogbAICXQNwAQBGXy/V1CTQDcQMARXq93tcl0AzEDQAUhYWF+boEmoG4AYAiWGfKVRA3AAAvgbgBgCK4ItxVEDcAUARXhLsK4gYAikQika9LoBmIGwAoUqlUvi6BZiBuAABeAnEDAEWwrJ2rIG4AoAiWtXMVxA0AFIWEhPi6BJqBuAGAIq1W6+sSaAbiBgCKOjs7fV0CzWAwTwkAl6SkpGAYxmAwrFarxWKxfZ+UlLRlyxZfl+bvoHcDgGsGDhzIYDAQQhiG4TjOYDCEQuHTTz/t67poAOIGANdMnz6dIIiuWxITEzMzM31XEW1A3ADgmuzs7KioKPuPIpEoJyfHpxXRBsQNAK7hcDizZs2ydXCsVmtCQgJ0bZwEcQOAy+bOnRsdHQ1dG1dB3ADgMlsHB8fxAQMGQNfGeXAiHNwTmmo7VW0mvYbUqc1moxv2eZIk8/Ly0tLS3HLHYg4fxwmMK8D5YkKWyGUEaDcA4gYEsrqr+opL2utXtVIZz2S04kycQeDI/z7NDAZmNpgsJpIIYjRVqaITuInD+UkjhL6uy80gbkBgulHZcWqXnM1nMVhBwlAeHuR3EeOAprXDoOmQX1ePmhkyeFTg3MQL4gYEoCO5rc03DCH3STgilq9roc5itrbWtFlNpsmLw8VhTF+X4wYQNyCgdOrI3HfrIweE8aRsX9fiHmYDeb2kcdzs0H7DeL6u5W5B3IDAYTZav1hVG58hI1i4r2txs8Zfm8fOlMgS6J2hEDcgQHTqyG/WXO8/rq+vC/GUxrLmYWPpPX5Mp/EzABzIfbc+PkPm6yo8KGpQeOERZWuDwdeFUAe9GxAIjm1rNSAuL5jexxrOaLraNO8v0f53Kt8p9KwagC5uVnU01hnuhaxBCBE8zqldcl9XQRHEDaC9gp3y0Lh7ZVGEkL6iiiJ1h5b0dSFUQNwAerterg/isf1zfs3W7ave/3ie25uN6B9SfFzp9ma9AOIG0FtliRZnBcIUOOfxJJzyi7RcwBPiBtBb7a9aQSjt57+5hAhisHjMpjr63ZidcOJvAPBTzdcN4nAu4ZnrodoVjXsPra+sLmQSrOio/lMe+p8+0UkIoa+2vhwa0hfHiYvFu82kaWDiqNnTX+Gw+bZHlV45euTEZoXyVnhonNVq8URhCCFBmOBGZWdELM1Gx6F3A2hM1WY0ueNuEndSq+UbNi3V69Uzp740bdKzJGn6dPNTt5qrbb8tOLu1XdH4xKIPsqe+9EvZ8eMnv7JtL7l8OPfHlUK+NHvq8v4JGY1Nv3miNoQQTjCa66F3A4AX6dUkHuSRffhowZd8nuSpJRtwnEAIpQ6d8t76OReL92RPewkhFCqNWTD3bxiGxciSf7l6oqLqQhZ6zmQy7Dn4YVzflKU5/8RxHCEkb7vhocQhgnBdi9kTLXsUxA2gMY3STAR55PKoa5XnlKrmFWvG27eQpEmpbrZ9z2SyMQyzfS8RR9bV/4IQqr1+WadXjhn5mC1rEEIMhqcu3SJYOB3PhUPcABr7/SPvfhptW1L/0dMefqbrRjaLf+df4jjTYiERQgpVky19PFUT/UHcABrjiwjyuskTLXM5Qp1eFRYa60IxvGCEkFbvjRkxZgPJFdDvqncYKgY0xhMSpNEjQxgJcWl19Zdv3Cy3bzEYOxw/JCoiAcMYJZfzPFHPbcxGkiukX1+BfhUDYCcKYTKDPHJANXHCk+WVZzd98/zYUQsEPMm1385bLOSShf9w8JBgcUT68OkXL+0xmw39EzLVGnl55VkBX+qJ8iykJaKvP06kdgziBtBYWAxL2dQR3Jd0+4BxiFT27NJN+w5/kl/wNcIwWeSAURmP9Pqo7GnLCSLo518OV1RdvC9maFREokbb5t7CbDTNGtnEUE+07FFwAwpAb0e3tWh0QRKZwNeFeI/ZSNZevLn0nft8XYjLoHcD6C0xhV94XOvgD9Rq+d//+eid261WK0JWDOtm+DJr0nMZ92e7q8LyirNbd6zq9lchEpm8veHO7ZMeWDYms5uabXRtnUmZtFyeAXo3gPa+X3dDGC3lirsfyyBJUvX7fJmuLBaL1Wq1z5HpissRsdluuw7LaOzU6tp7+CWGUDcfQA5HaL8q4k4VBddz3oxl8+h3ngfiBtBeY3XHsR/aYlLuiQkvbfWqsHDLmOwQXxdCBf0CEoDbRMVzZPEsvbKXE9WBwaztoGnWQNyAAPHAo6FN5XJTJ/3m9bukrujmg/Ppd0LKDuIGBIiFr8VUX+hm2DVg3LzSnDElOCQqyNeFUAdjNyBwmE1o88qa+AwZk02/Cf6O3SxrnjBbGhVPv6l9XUHvBgQOgomWrI698XOjrj1wxnFMnWT1uRuZk0V0zxro3YDAlP9D680agzRW0tPZcVogTRZ5bTsDmSc9Hi6UBMIUOYgbEJhu1XSe2iUn2EFYUJAwjOuh2+J4iLpFb9B0tjdoRs8MSc6k8Sq9t4G4AYGs/pq+4pK27qouOIJjMllxJoEzcQz3uzEEDMPMBhNpNBNBjKYqdcwAXkIKf2B6oF2ZAXED7gnN9QaV3KhXkzq12WTwu32eyydwJuIKcb6I2ac/x9fleArEDQDAS/yuVwkACFQQNwAAL4G4AQB4CcQNAMBLIG4AAF4CFc0+rgAAAAtJREFUcQMA8JL/B/2rTmDrA3PZAAAAAElFTkSuQmCC",
      "text/plain": [
       "<ai_data_science_team.ml_agents.h2o_ml_agent.H2OMLAgent object at 0x7feef87e4f40>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ml_agent = H2OMLAgent(\n",
    "    model=llm, \n",
    "    log=True, \n",
    "    log_path=LOG_PATH,\n",
    "    model_directory=MODEL_PATH, \n",
    "    enable_mlflow=True, # Use this if you wish to log to MLflow \n",
    ")\n",
    "ml_agent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run the Agent\n",
    "\n",
    "Run the code to run the AI Machine Learning Agent and create the models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---H2O ML AGENT----\n",
      "    * RECOMMEND MACHINE LEARNING STEPS\n",
      "    * CREATE H2O AUTOML CODE\n",
      "      File saved to: logs/h2o_automl.py\n",
      "    * EXECUTING AGENT CODE\n",
      "Checking whether there is an H2O instance running at http://localhost:54321. connected.\n",
      "Warning: Your H2O cluster version is (10 months and 4 days) old.  There may be a newer version available.\n",
      "Please download and install the latest version from: https://h2o-release.s3.amazonaws.com/h2o/latest_stable.html\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "\n",
       "#h2o-table-1.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-1 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-1 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-1 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-1 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-1 .h2o-table th,\n",
       "#h2o-table-1 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-1 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-1\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption></caption>\n",
       "    <thead></thead>\n",
       "    <tbody><tr><td>H2O_cluster_uptime:</td>\n",
       "<td>7 hours 14 mins</td></tr>\n",
       "<tr><td>H2O_cluster_timezone:</td>\n",
       "<td>America/New_York</td></tr>\n",
       "<tr><td>H2O_data_parsing_timezone:</td>\n",
       "<td>UTC</td></tr>\n",
       "<tr><td>H2O_cluster_version:</td>\n",
       "<td>3.46.0.1</td></tr>\n",
       "<tr><td>H2O_cluster_version_age:</td>\n",
       "<td>10 months and 4 days</td></tr>\n",
       "<tr><td>H2O_cluster_name:</td>\n",
       "<td>H2O_from_python_mdancho_ag7ena</td></tr>\n",
       "<tr><td>H2O_cluster_total_nodes:</td>\n",
       "<td>1</td></tr>\n",
       "<tr><td>H2O_cluster_free_memory:</td>\n",
       "<td>11.99 Gb</td></tr>\n",
       "<tr><td>H2O_cluster_total_cores:</td>\n",
       "<td>14</td></tr>\n",
       "<tr><td>H2O_cluster_allowed_cores:</td>\n",
       "<td>14</td></tr>\n",
       "<tr><td>H2O_cluster_status:</td>\n",
       "<td>locked, healthy</td></tr>\n",
       "<tr><td>H2O_connection_url:</td>\n",
       "<td>http://localhost:54321</td></tr>\n",
       "<tr><td>H2O_connection_proxy:</td>\n",
       "<td>{\"http\": null, \"https\": null}</td></tr>\n",
       "<tr><td>H2O_internal_security:</td>\n",
       "<td>False</td></tr>\n",
       "<tr><td>Python_version:</td>\n",
       "<td>3.10.13 final</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n"
      ],
      "text/plain": [
       "--------------------------  ------------------------------\n",
       "H2O_cluster_uptime:         7 hours 14 mins\n",
       "H2O_cluster_timezone:       America/New_York\n",
       "H2O_data_parsing_timezone:  UTC\n",
       "H2O_cluster_version:        3.46.0.1\n",
       "H2O_cluster_version_age:    10 months and 4 days\n",
       "H2O_cluster_name:           H2O_from_python_mdancho_ag7ena\n",
       "H2O_cluster_total_nodes:    1\n",
       "H2O_cluster_free_memory:    11.99 Gb\n",
       "H2O_cluster_total_cores:    14\n",
       "H2O_cluster_allowed_cores:  14\n",
       "H2O_cluster_status:         locked, healthy\n",
       "H2O_connection_url:         http://localhost:54321\n",
       "H2O_connection_proxy:       {\"http\": null, \"https\": null}\n",
       "H2O_internal_security:      False\n",
       "Python_version:             3.10.13 final\n",
       "--------------------------  ------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parse progress: |████████████████████████████████████████████████████████████████| (done) 100%\n",
      "AutoML progress: |\n",
      "18:27:21.352: Stopping tolerance set by the user is < 70% of the recommended default of 0.011915743770127944, so models may take a long time to converge or may not converge at all.\n",
      "\n",
      "███████████████████████████████████████████████████████████████| (done) 100%\n",
      "    * REPORT AGENT OUTPUTS\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/mdancho/opt/anaconda3/envs/ds4b_301p_dev/lib/python3.10/site-packages/h2o/frame.py:1983: H2ODependencyWarning: Converting H2O frame to pandas dataframe using single-thread.  For faster conversion using multi-thread, install datatable (for Python 3.9 or lower), or polars and pyarrow (for Python 3.10 or above) and activate it using:\n",
      "\n",
      "with h2o.utils.threading.local_context(polars_enabled=True, datatable_enabled=True):\n",
      "    pandas_df = h2o_df.as_data_frame()\n",
      "\n",
      "  warnings.warn(\"Converting H2O frame to pandas dataframe using single-thread.  For faster conversion using\"\n"
     ]
    }
   ],
   "source": [
    "\n",
    "ml_agent.invoke_agent(\n",
    "    data_raw=df.drop(columns=[\"customerID\"]),\n",
    "    user_instructions=\"Please do classification on 'Churn'. Use a max runtime of 30 seconds.\",\n",
    "    target_variable=\"Churn\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Leaderboard\n",
    "\n",
    "The leaderboard shows the ML models created by the AI Machine Learning Agent ranked by various ML metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model_id</th>\n",
       "      <th>auc</th>\n",
       "      <th>logloss</th>\n",
       "      <th>aucpr</th>\n",
       "      <th>mean_per_class_error</th>\n",
       "      <th>rmse</th>\n",
       "      <th>mse</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>StackedEnsemble_BestOfFamily_2_AutoML_2_202501...</td>\n",
       "      <td>0.850006</td>\n",
       "      <td>0.410950</td>\n",
       "      <td>0.666045</td>\n",
       "      <td>0.228422</td>\n",
       "      <td>0.365294</td>\n",
       "      <td>0.133439</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>StackedEnsemble_BestOfFamily_1_AutoML_2_202501...</td>\n",
       "      <td>0.849827</td>\n",
       "      <td>0.411221</td>\n",
       "      <td>0.665194</td>\n",
       "      <td>0.226759</td>\n",
       "      <td>0.365353</td>\n",
       "      <td>0.133483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>StackedEnsemble_AllModels_2_AutoML_2_20250117_...</td>\n",
       "      <td>0.849722</td>\n",
       "      <td>0.411135</td>\n",
       "      <td>0.666036</td>\n",
       "      <td>0.224464</td>\n",
       "      <td>0.365316</td>\n",
       "      <td>0.133456</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>StackedEnsemble_BestOfFamily_4_AutoML_2_202501...</td>\n",
       "      <td>0.849691</td>\n",
       "      <td>0.410955</td>\n",
       "      <td>0.667338</td>\n",
       "      <td>0.230516</td>\n",
       "      <td>0.365232</td>\n",
       "      <td>0.133394</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>StackedEnsemble_AllModels_1_AutoML_2_20250117_...</td>\n",
       "      <td>0.849660</td>\n",
       "      <td>0.411465</td>\n",
       "      <td>0.664531</td>\n",
       "      <td>0.230069</td>\n",
       "      <td>0.365506</td>\n",
       "      <td>0.133594</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>StackedEnsemble_BestOfFamily_3_AutoML_2_202501...</td>\n",
       "      <td>0.849502</td>\n",
       "      <td>0.411334</td>\n",
       "      <td>0.667723</td>\n",
       "      <td>0.226358</td>\n",
       "      <td>0.365353</td>\n",
       "      <td>0.133483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>StackedEnsemble_AllModels_3_AutoML_2_20250117_...</td>\n",
       "      <td>0.848822</td>\n",
       "      <td>0.411923</td>\n",
       "      <td>0.664691</td>\n",
       "      <td>0.227170</td>\n",
       "      <td>0.365783</td>\n",
       "      <td>0.133797</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>GBM_1_AutoML_2_20250117_182721</td>\n",
       "      <td>0.847581</td>\n",
       "      <td>0.414998</td>\n",
       "      <td>0.662772</td>\n",
       "      <td>0.229477</td>\n",
       "      <td>0.367105</td>\n",
       "      <td>0.134766</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>XGBoost_grid_1_AutoML_2_20250117_182721_model_1</td>\n",
       "      <td>0.845952</td>\n",
       "      <td>0.415312</td>\n",
       "      <td>0.660968</td>\n",
       "      <td>0.236311</td>\n",
       "      <td>0.367351</td>\n",
       "      <td>0.134947</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>XGBoost_grid_1_AutoML_2_20250117_182721_model_7</td>\n",
       "      <td>0.845670</td>\n",
       "      <td>0.419965</td>\n",
       "      <td>0.658424</td>\n",
       "      <td>0.238885</td>\n",
       "      <td>0.368094</td>\n",
       "      <td>0.135493</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>GLM_1_AutoML_2_20250117_182721</td>\n",
       "      <td>0.845134</td>\n",
       "      <td>0.417470</td>\n",
       "      <td>0.652031</td>\n",
       "      <td>0.234092</td>\n",
       "      <td>0.368076</td>\n",
       "      <td>0.135480</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>XGBoost_grid_1_AutoML_2_20250117_182721_model_5</td>\n",
       "      <td>0.844876</td>\n",
       "      <td>0.417112</td>\n",
       "      <td>0.655436</td>\n",
       "      <td>0.239689</td>\n",
       "      <td>0.368512</td>\n",
       "      <td>0.135801</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>XGBoost_grid_1_AutoML_2_20250117_182721_model_6</td>\n",
       "      <td>0.844580</td>\n",
       "      <td>0.416563</td>\n",
       "      <td>0.661210</td>\n",
       "      <td>0.232717</td>\n",
       "      <td>0.368033</td>\n",
       "      <td>0.135448</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>GBM_grid_1_AutoML_2_20250117_182721_model_1</td>\n",
       "      <td>0.843988</td>\n",
       "      <td>0.419304</td>\n",
       "      <td>0.655764</td>\n",
       "      <td>0.232006</td>\n",
       "      <td>0.368750</td>\n",
       "      <td>0.135977</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>GBM_5_AutoML_2_20250117_182721</td>\n",
       "      <td>0.842354</td>\n",
       "      <td>0.421817</td>\n",
       "      <td>0.655105</td>\n",
       "      <td>0.233498</td>\n",
       "      <td>0.370081</td>\n",
       "      <td>0.136960</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>GBM_2_AutoML_2_20250117_182721</td>\n",
       "      <td>0.841591</td>\n",
       "      <td>0.422390</td>\n",
       "      <td>0.656352</td>\n",
       "      <td>0.237992</td>\n",
       "      <td>0.370223</td>\n",
       "      <td>0.137065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>XGBoost_3_AutoML_2_20250117_182721</td>\n",
       "      <td>0.838633</td>\n",
       "      <td>0.481533</td>\n",
       "      <td>0.649947</td>\n",
       "      <td>0.235944</td>\n",
       "      <td>0.401041</td>\n",
       "      <td>0.160834</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>XGBoost_grid_1_AutoML_2_20250117_182721_model_3</td>\n",
       "      <td>0.837571</td>\n",
       "      <td>0.427998</td>\n",
       "      <td>0.648548</td>\n",
       "      <td>0.237621</td>\n",
       "      <td>0.372482</td>\n",
       "      <td>0.138743</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>XRT_1_AutoML_2_20250117_182721</td>\n",
       "      <td>0.837332</td>\n",
       "      <td>0.430208</td>\n",
       "      <td>0.648442</td>\n",
       "      <td>0.241145</td>\n",
       "      <td>0.374037</td>\n",
       "      <td>0.139904</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>GBM_3_AutoML_2_20250117_182721</td>\n",
       "      <td>0.835770</td>\n",
       "      <td>0.430271</td>\n",
       "      <td>0.642863</td>\n",
       "      <td>0.244853</td>\n",
       "      <td>0.373325</td>\n",
       "      <td>0.139372</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>XGBoost_grid_1_AutoML_2_20250117_182721_model_2</td>\n",
       "      <td>0.835729</td>\n",
       "      <td>0.430062</td>\n",
       "      <td>0.647138</td>\n",
       "      <td>0.249140</td>\n",
       "      <td>0.373906</td>\n",
       "      <td>0.139806</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>GBM_grid_1_AutoML_2_20250117_182721_model_4</td>\n",
       "      <td>0.834568</td>\n",
       "      <td>0.433872</td>\n",
       "      <td>0.644549</td>\n",
       "      <td>0.240181</td>\n",
       "      <td>0.375531</td>\n",
       "      <td>0.141023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>GBM_4_AutoML_2_20250117_182721</td>\n",
       "      <td>0.831271</td>\n",
       "      <td>0.438734</td>\n",
       "      <td>0.637577</td>\n",
       "      <td>0.245945</td>\n",
       "      <td>0.376506</td>\n",
       "      <td>0.141757</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>GBM_grid_1_AutoML_2_20250117_182721_model_2</td>\n",
       "      <td>0.830516</td>\n",
       "      <td>0.442387</td>\n",
       "      <td>0.631798</td>\n",
       "      <td>0.248531</td>\n",
       "      <td>0.377900</td>\n",
       "      <td>0.142809</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>XGBoost_grid_1_AutoML_2_20250117_182721_model_4</td>\n",
       "      <td>0.826230</td>\n",
       "      <td>0.457860</td>\n",
       "      <td>0.630066</td>\n",
       "      <td>0.249123</td>\n",
       "      <td>0.382691</td>\n",
       "      <td>0.146453</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>GBM_grid_1_AutoML_2_20250117_182721_model_3</td>\n",
       "      <td>0.826118</td>\n",
       "      <td>0.512924</td>\n",
       "      <td>0.630421</td>\n",
       "      <td>0.247201</td>\n",
       "      <td>0.392623</td>\n",
       "      <td>0.154153</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>XGBoost_1_AutoML_2_20250117_182721</td>\n",
       "      <td>0.823323</td>\n",
       "      <td>0.501767</td>\n",
       "      <td>0.624059</td>\n",
       "      <td>0.253169</td>\n",
       "      <td>0.407095</td>\n",
       "      <td>0.165726</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>DRF_1_AutoML_2_20250117_182721</td>\n",
       "      <td>0.821350</td>\n",
       "      <td>0.496097</td>\n",
       "      <td>0.615213</td>\n",
       "      <td>0.267490</td>\n",
       "      <td>0.386310</td>\n",
       "      <td>0.149236</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>XGBoost_2_AutoML_2_20250117_182721</td>\n",
       "      <td>0.815923</td>\n",
       "      <td>0.515674</td>\n",
       "      <td>0.602748</td>\n",
       "      <td>0.261969</td>\n",
       "      <td>0.409996</td>\n",
       "      <td>0.168096</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>GBM_grid_1_AutoML_2_20250117_182721_model_5</td>\n",
       "      <td>0.789088</td>\n",
       "      <td>0.552519</td>\n",
       "      <td>0.551880</td>\n",
       "      <td>0.271343</td>\n",
       "      <td>0.429957</td>\n",
       "      <td>0.184863</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             model_id       auc   logloss  \\\n",
       "0   StackedEnsemble_BestOfFamily_2_AutoML_2_202501...  0.850006  0.410950   \n",
       "1   StackedEnsemble_BestOfFamily_1_AutoML_2_202501...  0.849827  0.411221   \n",
       "2   StackedEnsemble_AllModels_2_AutoML_2_20250117_...  0.849722  0.411135   \n",
       "3   StackedEnsemble_BestOfFamily_4_AutoML_2_202501...  0.849691  0.410955   \n",
       "4   StackedEnsemble_AllModels_1_AutoML_2_20250117_...  0.849660  0.411465   \n",
       "5   StackedEnsemble_BestOfFamily_3_AutoML_2_202501...  0.849502  0.411334   \n",
       "6   StackedEnsemble_AllModels_3_AutoML_2_20250117_...  0.848822  0.411923   \n",
       "7                      GBM_1_AutoML_2_20250117_182721  0.847581  0.414998   \n",
       "8     XGBoost_grid_1_AutoML_2_20250117_182721_model_1  0.845952  0.415312   \n",
       "9     XGBoost_grid_1_AutoML_2_20250117_182721_model_7  0.845670  0.419965   \n",
       "10                     GLM_1_AutoML_2_20250117_182721  0.845134  0.417470   \n",
       "11    XGBoost_grid_1_AutoML_2_20250117_182721_model_5  0.844876  0.417112   \n",
       "12    XGBoost_grid_1_AutoML_2_20250117_182721_model_6  0.844580  0.416563   \n",
       "13        GBM_grid_1_AutoML_2_20250117_182721_model_1  0.843988  0.419304   \n",
       "14                     GBM_5_AutoML_2_20250117_182721  0.842354  0.421817   \n",
       "15                     GBM_2_AutoML_2_20250117_182721  0.841591  0.422390   \n",
       "16                 XGBoost_3_AutoML_2_20250117_182721  0.838633  0.481533   \n",
       "17    XGBoost_grid_1_AutoML_2_20250117_182721_model_3  0.837571  0.427998   \n",
       "18                     XRT_1_AutoML_2_20250117_182721  0.837332  0.430208   \n",
       "19                     GBM_3_AutoML_2_20250117_182721  0.835770  0.430271   \n",
       "20    XGBoost_grid_1_AutoML_2_20250117_182721_model_2  0.835729  0.430062   \n",
       "21        GBM_grid_1_AutoML_2_20250117_182721_model_4  0.834568  0.433872   \n",
       "22                     GBM_4_AutoML_2_20250117_182721  0.831271  0.438734   \n",
       "23        GBM_grid_1_AutoML_2_20250117_182721_model_2  0.830516  0.442387   \n",
       "24    XGBoost_grid_1_AutoML_2_20250117_182721_model_4  0.826230  0.457860   \n",
       "25        GBM_grid_1_AutoML_2_20250117_182721_model_3  0.826118  0.512924   \n",
       "26                 XGBoost_1_AutoML_2_20250117_182721  0.823323  0.501767   \n",
       "27                     DRF_1_AutoML_2_20250117_182721  0.821350  0.496097   \n",
       "28                 XGBoost_2_AutoML_2_20250117_182721  0.815923  0.515674   \n",
       "29        GBM_grid_1_AutoML_2_20250117_182721_model_5  0.789088  0.552519   \n",
       "\n",
       "       aucpr  mean_per_class_error      rmse       mse  \n",
       "0   0.666045              0.228422  0.365294  0.133439  \n",
       "1   0.665194              0.226759  0.365353  0.133483  \n",
       "2   0.666036              0.224464  0.365316  0.133456  \n",
       "3   0.667338              0.230516  0.365232  0.133394  \n",
       "4   0.664531              0.230069  0.365506  0.133594  \n",
       "5   0.667723              0.226358  0.365353  0.133483  \n",
       "6   0.664691              0.227170  0.365783  0.133797  \n",
       "7   0.662772              0.229477  0.367105  0.134766  \n",
       "8   0.660968              0.236311  0.367351  0.134947  \n",
       "9   0.658424              0.238885  0.368094  0.135493  \n",
       "10  0.652031              0.234092  0.368076  0.135480  \n",
       "11  0.655436              0.239689  0.368512  0.135801  \n",
       "12  0.661210              0.232717  0.368033  0.135448  \n",
       "13  0.655764              0.232006  0.368750  0.135977  \n",
       "14  0.655105              0.233498  0.370081  0.136960  \n",
       "15  0.656352              0.237992  0.370223  0.137065  \n",
       "16  0.649947              0.235944  0.401041  0.160834  \n",
       "17  0.648548              0.237621  0.372482  0.138743  \n",
       "18  0.648442              0.241145  0.374037  0.139904  \n",
       "19  0.642863              0.244853  0.373325  0.139372  \n",
       "20  0.647138              0.249140  0.373906  0.139806  \n",
       "21  0.644549              0.240181  0.375531  0.141023  \n",
       "22  0.637577              0.245945  0.376506  0.141757  \n",
       "23  0.631798              0.248531  0.377900  0.142809  \n",
       "24  0.630066              0.249123  0.382691  0.146453  \n",
       "25  0.630421              0.247201  0.392623  0.154153  \n",
       "26  0.624059              0.253169  0.407095  0.165726  \n",
       "27  0.615213              0.267490  0.386310  0.149236  \n",
       "28  0.602748              0.261969  0.409996  0.168096  \n",
       "29  0.551880              0.271343  0.429957  0.184863  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Retrieve and display the leaderboard of models\n",
    "ml_agent.get_leaderboard()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get The AI Generated Python Code\n",
    "\n",
    "The AI Machine Learning Agent will generate Python code that was used to create the models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "```python\n",
       "# Disclaimer: This function was generated by AI. Please review before using.\n",
       "# Agent Name: h2o_ml_agent\n",
       "# Time Created: 2025-01-17 18:27:20\n",
       "\n",
       "def h2o_automl(data_raw, model_directory=None, log_path=None):\n",
       "    import h2o\n",
       "    from h2o.automl import H2OAutoML\n",
       "    import pandas as pd\n",
       "\n",
       "\n",
       "\n",
       "    # Initialize or connect to H2O if not already started\n",
       "    h2o.init()\n",
       "\n",
       "    # Convert the pandas DataFrame to an H2OFrame\n",
       "    data_h2o = h2o.H2OFrame(data_raw)\n",
       "\n",
       "    # Identify the target variable and predictors\n",
       "    target = \"Churn\"\n",
       "    x = [col for col in data_h2o.columns if col != target]\n",
       "\n",
       "    # Convert target variable to factor (categorical)\n",
       "    data_h2o[target] = data_h2o[target].asfactor()\n",
       "\n",
       "    # Set AutoML parameters based on user instructions\n",
       "    aml = H2OAutoML(\n",
       "        max_runtime_secs=30,\n",
       "        exclude_algos=[\"DeepLearning\"],\n",
       "        stopping_metric=\"AUC\",\n",
       "        stopping_rounds=3,\n",
       "        stopping_tolerance=0.001,\n",
       "        balance_classes=True,\n",
       "        seed=42\n",
       "    )\n",
       "\n",
       "    # Train AutoML model\n",
       "    aml.train(x=x, y=target, training_frame=data_h2o)\n",
       "\n",
       "    # Determine model saving logic\n",
       "    if model_directory is None and log_path is None:\n",
       "        model_path = None\n",
       "    else:\n",
       "        the_directory = model_directory if model_directory is not None else log_path\n",
       "        model_path = h2o.save_model(model=aml.leader, path=the_directory, force=True)\n",
       "\n",
       "    return {\n",
       "        \"leaderboard\": aml.leaderboard.as_data_frame().to_dict(),\n",
       "        \"best_model_id\": aml.leader.model_id,\n",
       "        \"model_path\": model_path,\n",
       "    }\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the H2O training function in markdown format\n",
    "ml_agent.get_h2o_train_function(markdown=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Recommended ML Steps\n",
    "\n",
    "The AI Machine Learning Agent will provide you with a recommended set of ML steps, which were used to help the Coding Agent create the H2O AutoML models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "# Recommended ML Steps:\n",
       "Here is a recommended list of steps and considerations for performing H2O AutoML on the provided dataset, focusing on maximizing model accuracy while adhering to the user instructions:\n",
       "\n",
       "1. **Initialize H2O Cluster**:\n",
       "   - Start the H2O cluster locally.\n",
       "   ```python\n",
       "   import h2o\n",
       "   h2o.init()\n",
       "   ```\n",
       "\n",
       "2. **Import Dataset**:\n",
       "   - Load the dataset into the H2O environment.\n",
       "   ```python\n",
       "   data = h2o.import_file(\"path_to_your_dataset.csv\")\n",
       "   ```\n",
       "\n",
       "3. **Identify Predictors and Response**:\n",
       "   - Define the response column (`y`) and the predictor columns (`x`).\n",
       "   ```python\n",
       "   y = \"Churn\"  # response column\n",
       "   x = data.columns\n",
       "   x.remove(y)  # predictors\n",
       "   ```\n",
       "\n",
       "4. **Convert Response Column to Factor**:\n",
       "   - Convert the response variable to a categorical type, as it is a classification problem.\n",
       "   ```python\n",
       "   data[y] = data[y].asfactor()\n",
       "   ```\n",
       "\n",
       "5. **Set AutoML Parameters**:\n",
       "   - Specify `max_runtime_secs` to 30 seconds as per user instructions.\n",
       "   - Exclude Deep Learning algorithms to focus on potentially higher-performing models.\n",
       "   ```python\n",
       "   aml = H2OAutoML(max_runtime_secs=30, exclude_algos=[\"DeepLearning\"], seed=42)\n",
       "   ```\n",
       "\n",
       "6. **Set Stopping Criteria**:\n",
       "   - Use `stopping_metric` to specify AUC as the metric for early stopping, which is relevant for binary classification.\n",
       "   - Set `stopping_rounds` to 3 to allow some patience for model improvement.\n",
       "   - Set `stopping_tolerance` to 0.001 for relative improvement.\n",
       "   ```python\n",
       "   aml = H2OAutoML(max_runtime_secs=30, exclude_algos=[\"DeepLearning\"], \n",
       "                   stopping_metric=\"AUC\", stopping_rounds=3, stopping_tolerance=0.001, seed=42)\n",
       "   ```\n",
       "\n",
       "7. **Balance Classes** (if applicable):\n",
       "   - Consider setting `balance_classes=True` to help handle any class imbalance if present in the dataset.\n",
       "   ```python\n",
       "   aml = H2OAutoML(max_runtime_secs=30, exclude_algos=[\"DeepLearning\"], \n",
       "                   stopping_metric=\"AUC\", stopping_rounds=3, stopping_tolerance=0.001, \n",
       "                   balance_classes=True, seed=42)\n",
       "   ```\n",
       "\n",
       "8. **Train AutoML**:\n",
       "   - Train the AutoML model with the defined parameters.\n",
       "   ```python\n",
       "   aml.train(x=x, y=y, training_frame=data)\n",
       "   ```\n",
       "\n",
       "9. **Review Leaderboard**:\n",
       "   - After training, extract and review the leaderboard to evaluate model performances.\n",
       "   ```python\n",
       "   lb = aml.leaderboard\n",
       "   ```\n",
       "\n",
       "10. **Select Best Model**:\n",
       "   - Identify and retrieve the best model based on the default metric (AUC for binary classification).\n",
       "   ```python\n",
       "   best_model = aml.leader\n",
       "   ```\n",
       "\n",
       "By following these steps, you can effectively set up and execute H2O AutoML on the provided dataset while maximizing predictive accuracy and adhering to user specifications."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the recommended machine learning steps in markdown format\n",
    "ml_agent.get_recommended_ml_steps(markdown=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get A Complete Workflow Summary\n",
    "\n",
    "The AI Machine Learning Agent will provide you with a complete workflow summary that includes the Best ML model created, the recommended ML steps, and the Python code used to create the models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "# H2O Machine Learning Agent Outputs\n",
       "\n",
       "## ---RECOMMENDED STEPS----\n",
       "# Recommended ML Steps:\n",
       "Here is a recommended list of steps and considerations for performing H2O AutoML on the provided dataset, focusing on maximizing model accuracy while adhering to the user instructions:\n",
       "\n",
       "1. **Initialize H2O Cluster**:\n",
       "   - Start the H2O cluster locally.\n",
       "   ```python\n",
       "   import h2o\n",
       "   h2o.init()\n",
       "   ```\n",
       "\n",
       "2. **Import Dataset**:\n",
       "   - Load the dataset into the H2O environment.\n",
       "   ```python\n",
       "   data = h2o.import_file(\"path_to_your_dataset.csv\")\n",
       "   ```\n",
       "\n",
       "3. **Identify Predictors and Response**:\n",
       "   - Define the response column (`y`) and the predictor columns (`x`).\n",
       "   ```python\n",
       "   y = \"Churn\"  # response column\n",
       "   x = data.columns\n",
       "   x.remove(y)  # predictors\n",
       "   ```\n",
       "\n",
       "4. **Convert Response Column to Factor**:\n",
       "   - Convert the response variable to a categorical type, as it is a classification problem.\n",
       "   ```python\n",
       "   data[y] = data[y].asfactor()\n",
       "   ```\n",
       "\n",
       "5. **Set AutoML Parameters**:\n",
       "   - Specify `max_runtime_secs` to 30 seconds as per user instructions.\n",
       "   - Exclude Deep Learning algorithms to focus on potentially higher-performing models.\n",
       "   ```python\n",
       "   aml = H2OAutoML(max_runtime_secs=30, exclude_algos=[\"DeepLearning\"], seed=42)\n",
       "   ```\n",
       "\n",
       "6. **Set Stopping Criteria**:\n",
       "   - Use `stopping_metric` to specify AUC as the metric for early stopping, which is relevant for binary classification.\n",
       "   - Set `stopping_rounds` to 3 to allow some patience for model improvement.\n",
       "   - Set `stopping_tolerance` to 0.001 for relative improvement.\n",
       "   ```python\n",
       "   aml = H2OAutoML(max_runtime_secs=30, exclude_algos=[\"DeepLearning\"], \n",
       "                   stopping_metric=\"AUC\", stopping_rounds=3, stopping_tolerance=0.001, seed=42)\n",
       "   ```\n",
       "\n",
       "7. **Balance Classes** (if applicable):\n",
       "   - Consider setting `balance_classes=True` to help handle any class imbalance if present in the dataset.\n",
       "   ```python\n",
       "   aml = H2OAutoML(max_runtime_secs=30, exclude_algos=[\"DeepLearning\"], \n",
       "                   stopping_metric=\"AUC\", stopping_rounds=3, stopping_tolerance=0.001, \n",
       "                   balance_classes=True, seed=42)\n",
       "   ```\n",
       "\n",
       "8. **Train AutoML**:\n",
       "   - Train the AutoML model with the defined parameters.\n",
       "   ```python\n",
       "   aml.train(x=x, y=y, training_frame=data)\n",
       "   ```\n",
       "\n",
       "9. **Review Leaderboard**:\n",
       "   - After training, extract and review the leaderboard to evaluate model performances.\n",
       "   ```python\n",
       "   lb = aml.leaderboard\n",
       "   ```\n",
       "\n",
       "10. **Select Best Model**:\n",
       "   - Identify and retrieve the best model based on the default metric (AUC for binary classification).\n",
       "   ```python\n",
       "   best_model = aml.leader\n",
       "   ```\n",
       "\n",
       "By following these steps, you can effectively set up and execute H2O AutoML on the provided dataset while maximizing predictive accuracy and adhering to user specifications.\n",
       "\n",
       "## ---H2O TRAIN FUNCTION----\n",
       "```python\n",
       "# Disclaimer: This function was generated by AI. Please review before using.\n",
       "# Agent Name: h2o_ml_agent\n",
       "# Time Created: 2025-01-17 18:27:20\n",
       "\n",
       "def h2o_automl(data_raw, model_directory=None, log_path=None):\n",
       "    import h2o\n",
       "    from h2o.automl import H2OAutoML\n",
       "    import pandas as pd\n",
       "\n",
       "\n",
       "\n",
       "    # Initialize or connect to H2O if not already started\n",
       "    h2o.init()\n",
       "\n",
       "    # Convert the pandas DataFrame to an H2OFrame\n",
       "    data_h2o = h2o.H2OFrame(data_raw)\n",
       "\n",
       "    # Identify the target variable and predictors\n",
       "    target = \"Churn\"\n",
       "    x = [col for col in data_h2o.columns if col != target]\n",
       "\n",
       "    # Convert target variable to factor (categorical)\n",
       "    data_h2o[target] = data_h2o[target].asfactor()\n",
       "\n",
       "    # Set AutoML parameters based on user instructions\n",
       "    aml = H2OAutoML(\n",
       "        max_runtime_secs=30,\n",
       "        exclude_algos=[\"DeepLearning\"],\n",
       "        stopping_metric=\"AUC\",\n",
       "        stopping_rounds=3,\n",
       "        stopping_tolerance=0.001,\n",
       "        balance_classes=True,\n",
       "        seed=42\n",
       "    )\n",
       "\n",
       "    # Train AutoML model\n",
       "    aml.train(x=x, y=target, training_frame=data_h2o)\n",
       "\n",
       "    # Determine model saving logic\n",
       "    if model_directory is None and log_path is None:\n",
       "        model_path = None\n",
       "    else:\n",
       "        the_directory = model_directory if model_directory is not None else log_path\n",
       "        model_path = h2o.save_model(model=aml.leader, path=the_directory, force=True)\n",
       "\n",
       "    return {\n",
       "        \"leaderboard\": aml.leaderboard.as_data_frame().to_dict(),\n",
       "        \"best_model_id\": aml.leader.model_id,\n",
       "        \"model_path\": model_path,\n",
       "    }\n",
       "```\n",
       "\n",
       "## ---H2O TRAIN FUNCTION PATH----\n",
       "```python\n",
       "logs/h2o_automl.py\n",
       "```\n",
       "\n",
       "## ---H2O TRAIN FUNCTION NAME----\n",
       "```python\n",
       "h2o_automl\n",
       "```\n",
       "\n",
       "## ---H2O TRAIN ERROR----\n",
       "None\n",
       "\n",
       "## ---MODEL PATH----\n",
       "None\n",
       "\n",
       "## ---BEST MODEL ID----\n",
       "StackedEnsemble_BestOfFamily_2_AutoML_2_20250117_182721"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get a summary of the workflow in markdown format\n",
    "ml_agent.get_workflow_summary(markdown=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get The Logging Summary \n",
    "\n",
    "The AI Machine Learning Agent will provide you with a logging summary that includes the ML models created and Python code location."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "## H2O Machine Learning Agent Log Summary:\n",
       "\n",
       "Function Path: logs/h2o_automl.py\n",
       "\n",
       "Function Name: h2o_automl\n",
       "\n",
       "Best Model ID: StackedEnsemble_BestOfFamily_2_AutoML_2_20250117_182721\n",
       "\n",
       "Model Path: None\n",
       "                "
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get a summary of the logs in markdown format\n",
    "ml_agent.get_log_summary(markdown=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get the Best Model ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'StackedEnsemble_BestOfFamily_2_AutoML_2_20250117_182721'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ml_agent.get_best_model_id()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load The Model\n",
    "\n",
    "Run the code to load the model and view the model summary, get performance metrics, make predictions, and explain the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking whether there is an H2O instance running at http://localhost:54321. connected.\n",
      "Warning: Your H2O cluster version is (10 months and 4 days) old.  There may be a newer version available.\n",
      "Please download and install the latest version from: https://h2o-release.s3.amazonaws.com/h2o/latest_stable.html\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "\n",
       "#h2o-table-12.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-12 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-12 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-12 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-12 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-12 .h2o-table th,\n",
       "#h2o-table-12 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-12 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-12\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption></caption>\n",
       "    <thead></thead>\n",
       "    <tbody><tr><td>H2O_cluster_uptime:</td>\n",
       "<td>7 hours 22 mins</td></tr>\n",
       "<tr><td>H2O_cluster_timezone:</td>\n",
       "<td>America/New_York</td></tr>\n",
       "<tr><td>H2O_data_parsing_timezone:</td>\n",
       "<td>UTC</td></tr>\n",
       "<tr><td>H2O_cluster_version:</td>\n",
       "<td>3.46.0.1</td></tr>\n",
       "<tr><td>H2O_cluster_version_age:</td>\n",
       "<td>10 months and 4 days</td></tr>\n",
       "<tr><td>H2O_cluster_name:</td>\n",
       "<td>H2O_from_python_mdancho_ag7ena</td></tr>\n",
       "<tr><td>H2O_cluster_total_nodes:</td>\n",
       "<td>1</td></tr>\n",
       "<tr><td>H2O_cluster_free_memory:</td>\n",
       "<td>11.94 Gb</td></tr>\n",
       "<tr><td>H2O_cluster_total_cores:</td>\n",
       "<td>14</td></tr>\n",
       "<tr><td>H2O_cluster_allowed_cores:</td>\n",
       "<td>14</td></tr>\n",
       "<tr><td>H2O_cluster_status:</td>\n",
       "<td>locked, healthy</td></tr>\n",
       "<tr><td>H2O_connection_url:</td>\n",
       "<td>http://localhost:54321</td></tr>\n",
       "<tr><td>H2O_connection_proxy:</td>\n",
       "<td>{\"http\": null, \"https\": null}</td></tr>\n",
       "<tr><td>H2O_internal_security:</td>\n",
       "<td>False</td></tr>\n",
       "<tr><td>Python_version:</td>\n",
       "<td>3.10.13 final</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n"
      ],
      "text/plain": [
       "--------------------------  ------------------------------\n",
       "H2O_cluster_uptime:         7 hours 22 mins\n",
       "H2O_cluster_timezone:       America/New_York\n",
       "H2O_data_parsing_timezone:  UTC\n",
       "H2O_cluster_version:        3.46.0.1\n",
       "H2O_cluster_version_age:    10 months and 4 days\n",
       "H2O_cluster_name:           H2O_from_python_mdancho_ag7ena\n",
       "H2O_cluster_total_nodes:    1\n",
       "H2O_cluster_free_memory:    11.94 Gb\n",
       "H2O_cluster_total_cores:    14\n",
       "H2O_cluster_allowed_cores:  14\n",
       "H2O_cluster_status:         locked, healthy\n",
       "H2O_connection_url:         http://localhost:54321\n",
       "H2O_connection_proxy:       {\"http\": null, \"https\": null}\n",
       "H2O_internal_security:      False\n",
       "Python_version:             3.10.13 final\n",
       "--------------------------  ------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "h2o.init()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load The Model and Display Performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style='margin: 1em 0 1em 0;'>Model Details\n",
       "=============\n",
       "H2OStackedEnsembleEstimator : Stacked Ensemble\n",
       "Model Key: StackedEnsemble_BestOfFamily_2_AutoML_2_20250117_182721\n",
       "</pre>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-13.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-13 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-13 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-13 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-13 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-13 .h2o-table th,\n",
       "#h2o-table-13 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-13 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-13\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>Model Summary for Stacked Ensemble: </caption>\n",
       "    <thead><tr><th>key</th>\n",
       "<th>value</th></tr></thead>\n",
       "    <tbody><tr><td>Stacking strategy</td>\n",
       "<td>cross_validation</td></tr>\n",
       "<tr><td>Number of base models (used / total)</td>\n",
       "<td>2/4</td></tr>\n",
       "<tr><td># GBM base models (used / total)</td>\n",
       "<td>1/1</td></tr>\n",
       "<tr><td># XGBoost base models (used / total)</td>\n",
       "<td>0/1</td></tr>\n",
       "<tr><td># GLM base models (used / total)</td>\n",
       "<td>1/1</td></tr>\n",
       "<tr><td># DRF base models (used / total)</td>\n",
       "<td>0/1</td></tr>\n",
       "<tr><td>Metalearner algorithm</td>\n",
       "<td>GLM</td></tr>\n",
       "<tr><td>Metalearner fold assignment scheme</td>\n",
       "<td>Random</td></tr>\n",
       "<tr><td>Metalearner nfolds</td>\n",
       "<td>5</td></tr>\n",
       "<tr><td>Metalearner fold_column</td>\n",
       "<td>None</td></tr>\n",
       "<tr><td>Custom metalearner hyperparameters</td>\n",
       "<td>None</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "</div>\n",
       "<div style='margin: 1em 0 1em 0;'><pre style='margin: 1em 0 1em 0;'>ModelMetricsBinomialGLM: stackedensemble\n",
       "** Reported on train data. **\n",
       "\n",
       "MSE: 0.12280258481192449\n",
       "RMSE: 0.3504319974145119\n",
       "LogLoss: 0.38118316635829386\n",
       "AUC: 0.8755385872855243\n",
       "AUCPR: 0.7225887060960569\n",
       "Gini: 0.7510771745710485\n",
       "Null degrees of freedom: 7042\n",
       "Residual degrees of freedom: 7040\n",
       "Null deviance: 8150.145899019416\n",
       "Residual deviance: 5369.346081322927\n",
       "AIC: 5375.346081322927</pre>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-14.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-14 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-14 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-14 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-14 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-14 .h2o-table th,\n",
       "#h2o-table-14 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-14 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-14\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.28863204785206925</caption>\n",
       "    <thead><tr><th></th>\n",
       "<th>No</th>\n",
       "<th>Yes</th>\n",
       "<th>Error</th>\n",
       "<th>Rate</th></tr></thead>\n",
       "    <tbody><tr><td>No</td>\n",
       "<td>4002.0</td>\n",
       "<td>1172.0</td>\n",
       "<td>0.2265</td>\n",
       "<td> (1172.0/5174.0)</td></tr>\n",
       "<tr><td>Yes</td>\n",
       "<td>341.0</td>\n",
       "<td>1528.0</td>\n",
       "<td>0.1825</td>\n",
       "<td> (341.0/1869.0)</td></tr>\n",
       "<tr><td>Total</td>\n",
       "<td>4343.0</td>\n",
       "<td>2700.0</td>\n",
       "<td>0.2148</td>\n",
       "<td> (1513.0/7043.0)</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "</div>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-15.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-15 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-15 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-15 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-15 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-15 .h2o-table th,\n",
       "#h2o-table-15 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-15 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-15\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>Maximum Metrics: Maximum metrics at their respective thresholds</caption>\n",
       "    <thead><tr><th>metric</th>\n",
       "<th>threshold</th>\n",
       "<th>value</th>\n",
       "<th>idx</th></tr></thead>\n",
       "    <tbody><tr><td>max f1</td>\n",
       "<td>0.2886320</td>\n",
       "<td>0.6688553</td>\n",
       "<td>229.0</td></tr>\n",
       "<tr><td>max f2</td>\n",
       "<td>0.1639926</td>\n",
       "<td>0.7709251</td>\n",
       "<td>289.0</td></tr>\n",
       "<tr><td>max f0point5</td>\n",
       "<td>0.5533829</td>\n",
       "<td>0.6779901</td>\n",
       "<td>117.0</td></tr>\n",
       "<tr><td>max accuracy</td>\n",
       "<td>0.5092172</td>\n",
       "<td>0.8233707</td>\n",
       "<td>137.0</td></tr>\n",
       "<tr><td>max precision</td>\n",
       "<td>0.8685693</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max recall</td>\n",
       "<td>0.0125838</td>\n",
       "<td>1.0</td>\n",
       "<td>391.0</td></tr>\n",
       "<tr><td>max specificity</td>\n",
       "<td>0.8685693</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max absolute_mcc</td>\n",
       "<td>0.2886320</td>\n",
       "<td>0.5367256</td>\n",
       "<td>229.0</td></tr>\n",
       "<tr><td>max min_per_class_accuracy</td>\n",
       "<td>0.3096726</td>\n",
       "<td>0.7881220</td>\n",
       "<td>220.0</td></tr>\n",
       "<tr><td>max mean_per_class_accuracy</td>\n",
       "<td>0.2886320</td>\n",
       "<td>0.7955161</td>\n",
       "<td>229.0</td></tr>\n",
       "<tr><td>max tns</td>\n",
       "<td>0.8685693</td>\n",
       "<td>5174.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max fns</td>\n",
       "<td>0.8685693</td>\n",
       "<td>1867.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max fps</td>\n",
       "<td>0.0050385</td>\n",
       "<td>5174.0</td>\n",
       "<td>399.0</td></tr>\n",
       "<tr><td>max tps</td>\n",
       "<td>0.0125838</td>\n",
       "<td>1869.0</td>\n",
       "<td>391.0</td></tr>\n",
       "<tr><td>max tnr</td>\n",
       "<td>0.8685693</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max fnr</td>\n",
       "<td>0.8685693</td>\n",
       "<td>0.9989299</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max fpr</td>\n",
       "<td>0.0050385</td>\n",
       "<td>1.0</td>\n",
       "<td>399.0</td></tr>\n",
       "<tr><td>max tpr</td>\n",
       "<td>0.0125838</td>\n",
       "<td>1.0</td>\n",
       "<td>391.0</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "</div>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-16.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-16 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-16 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-16 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-16 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-16 .h2o-table th,\n",
       "#h2o-table-16 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-16 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-16\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>Gains/Lift Table: Avg response rate: 26.54 %, avg score: 26.55 %</caption>\n",
       "    <thead><tr><th>group</th>\n",
       "<th>cumulative_data_fraction</th>\n",
       "<th>lower_threshold</th>\n",
       "<th>lift</th>\n",
       "<th>cumulative_lift</th>\n",
       "<th>response_rate</th>\n",
       "<th>score</th>\n",
       "<th>cumulative_response_rate</th>\n",
       "<th>cumulative_score</th>\n",
       "<th>capture_rate</th>\n",
       "<th>cumulative_capture_rate</th>\n",
       "<th>gain</th>\n",
       "<th>cumulative_gain</th>\n",
       "<th>kolmogorov_smirnov</th></tr></thead>\n",
       "    <tbody><tr><td>1</td>\n",
       "<td>0.0100809</td>\n",
       "<td>0.8249117</td>\n",
       "<td>3.6621753</td>\n",
       "<td>3.6621753</td>\n",
       "<td>0.9718310</td>\n",
       "<td>0.8408366</td>\n",
       "<td>0.9718310</td>\n",
       "<td>0.8408366</td>\n",
       "<td>0.0369181</td>\n",
       "<td>0.0369181</td>\n",
       "<td>266.2175299</td>\n",
       "<td>266.2175299</td>\n",
       "<td>0.0365316</td></tr>\n",
       "<tr><td>2</td>\n",
       "<td>0.0200199</td>\n",
       "<td>0.8016166</td>\n",
       "<td>3.3376596</td>\n",
       "<td>3.5010682</td>\n",
       "<td>0.8857143</td>\n",
       "<td>0.8130285</td>\n",
       "<td>0.9290780</td>\n",
       "<td>0.8270312</td>\n",
       "<td>0.0331728</td>\n",
       "<td>0.0700910</td>\n",
       "<td>233.7659558</td>\n",
       "<td>250.1068194</td>\n",
       "<td>0.0681582</td></tr>\n",
       "<tr><td>3</td>\n",
       "<td>0.0301008</td>\n",
       "<td>0.7848294</td>\n",
       "<td>3.3968003</td>\n",
       "<td>3.4661483</td>\n",
       "<td>0.9014085</td>\n",
       "<td>0.7920673</td>\n",
       "<td>0.9198113</td>\n",
       "<td>0.8153216</td>\n",
       "<td>0.0342429</td>\n",
       "<td>0.1043339</td>\n",
       "<td>239.6800277</td>\n",
       "<td>246.6148278</td>\n",
       "<td>0.1010482</td></tr>\n",
       "<tr><td>4</td>\n",
       "<td>0.0400398</td>\n",
       "<td>0.7693139</td>\n",
       "<td>3.4453260</td>\n",
       "<td>3.4609796</td>\n",
       "<td>0.9142857</td>\n",
       "<td>0.7772038</td>\n",
       "<td>0.9184397</td>\n",
       "<td>0.8058597</td>\n",
       "<td>0.0342429</td>\n",
       "<td>0.1385768</td>\n",
       "<td>244.5325996</td>\n",
       "<td>246.0979627</td>\n",
       "<td>0.1341315</td></tr>\n",
       "<tr><td>5</td>\n",
       "<td>0.0501207</td>\n",
       "<td>0.7543438</td>\n",
       "<td>3.1845003</td>\n",
       "<td>3.4053705</td>\n",
       "<td>0.8450704</td>\n",
       "<td>0.7618947</td>\n",
       "<td>0.9036827</td>\n",
       "<td>0.7970169</td>\n",
       "<td>0.0321027</td>\n",
       "<td>0.1706795</td>\n",
       "<td>218.4500260</td>\n",
       "<td>240.5370462</td>\n",
       "<td>0.1641082</td></tr>\n",
       "<tr><td>6</td>\n",
       "<td>0.1000994</td>\n",
       "<td>0.6652166</td>\n",
       "<td>2.8262440</td>\n",
       "<td>3.1162179</td>\n",
       "<td>0.75</td>\n",
       "<td>0.7081436</td>\n",
       "<td>0.8269504</td>\n",
       "<td>0.7526433</td>\n",
       "<td>0.1412520</td>\n",
       "<td>0.3119315</td>\n",
       "<td>182.6243981</td>\n",
       "<td>211.6217949</td>\n",
       "<td>0.2883521</td></tr>\n",
       "<tr><td>7</td>\n",
       "<td>0.1500781</td>\n",
       "<td>0.5947576</td>\n",
       "<td>2.5050799</td>\n",
       "<td>2.9126980</td>\n",
       "<td>0.6647727</td>\n",
       "<td>0.6294255</td>\n",
       "<td>0.7729423</td>\n",
       "<td>0.7116095</td>\n",
       "<td>0.1252006</td>\n",
       "<td>0.4371322</td>\n",
       "<td>150.5079892</td>\n",
       "<td>191.2697991</td>\n",
       "<td>0.3907464</td></tr>\n",
       "<tr><td>8</td>\n",
       "<td>0.2000568</td>\n",
       "<td>0.5306794</td>\n",
       "<td>2.0982720</td>\n",
       "<td>2.7092360</td>\n",
       "<td>0.5568182</td>\n",
       "<td>0.5631694</td>\n",
       "<td>0.7189496</td>\n",
       "<td>0.6745258</td>\n",
       "<td>0.1048689</td>\n",
       "<td>0.5420011</td>\n",
       "<td>109.8272046</td>\n",
       "<td>170.9236009</td>\n",
       "<td>0.4654645</td></tr>\n",
       "<tr><td>9</td>\n",
       "<td>0.3000142</td>\n",
       "<td>0.3818328</td>\n",
       "<td>1.6539951</td>\n",
       "<td>2.3576555</td>\n",
       "<td>0.4389205</td>\n",
       "<td>0.4565728</td>\n",
       "<td>0.6256507</td>\n",
       "<td>0.6019092</td>\n",
       "<td>0.1653291</td>\n",
       "<td>0.7073301</td>\n",
       "<td>65.3995057</td>\n",
       "<td>135.7655493</td>\n",
       "<td>0.5544503</td></tr>\n",
       "<tr><td>10</td>\n",
       "<td>0.3999716</td>\n",
       "<td>0.2728038</td>\n",
       "<td>1.2364817</td>\n",
       "<td>2.0774616</td>\n",
       "<td>0.328125</td>\n",
       "<td>0.3257965</td>\n",
       "<td>0.5512957</td>\n",
       "<td>0.5329056</td>\n",
       "<td>0.1235955</td>\n",
       "<td>0.8309256</td>\n",
       "<td>23.6481742</td>\n",
       "<td>107.7461556</td>\n",
       "<td>0.5866272</td></tr>\n",
       "<tr><td>11</td>\n",
       "<td>0.5000710</td>\n",
       "<td>0.1778435</td>\n",
       "<td>0.7536651</td>\n",
       "<td>1.8124767</td>\n",
       "<td>0.2</td>\n",
       "<td>0.2246125</td>\n",
       "<td>0.4809767</td>\n",
       "<td>0.4711944</td>\n",
       "<td>0.0754414</td>\n",
       "<td>0.9063670</td>\n",
       "<td>-24.6334938</td>\n",
       "<td>81.2476738</td>\n",
       "<td>0.5530621</td></tr>\n",
       "<tr><td>12</td>\n",
       "<td>0.6000284</td>\n",
       "<td>0.1109879</td>\n",
       "<td>0.4870989</td>\n",
       "<td>1.5916850</td>\n",
       "<td>0.1292614</td>\n",
       "<td>0.1420329</td>\n",
       "<td>0.4223852</td>\n",
       "<td>0.4163601</td>\n",
       "<td>0.0486891</td>\n",
       "<td>0.9550562</td>\n",
       "<td>-51.2901132</td>\n",
       "<td>59.1684968</td>\n",
       "<td>0.4832742</td></tr>\n",
       "<tr><td>13</td>\n",
       "<td>0.6999858</td>\n",
       "<td>0.0574955</td>\n",
       "<td>0.2783422</td>\n",
       "<td>1.4041407</td>\n",
       "<td>0.0738636</td>\n",
       "<td>0.0820331</td>\n",
       "<td>0.3726166</td>\n",
       "<td>0.3686185</td>\n",
       "<td>0.0278224</td>\n",
       "<td>0.9828785</td>\n",
       "<td>-72.1657790</td>\n",
       "<td>40.4140688</td>\n",
       "<td>0.3850819</td></tr>\n",
       "<tr><td>14</td>\n",
       "<td>0.7999432</td>\n",
       "<td>0.0293164</td>\n",
       "<td>0.1017020</td>\n",
       "<td>1.2413936</td>\n",
       "<td>0.0269886</td>\n",
       "<td>0.0422363</td>\n",
       "<td>0.3294285</td>\n",
       "<td>0.3278352</td>\n",
       "<td>0.0101659</td>\n",
       "<td>0.9930444</td>\n",
       "<td>-89.8298039</td>\n",
       "<td>24.1393641</td>\n",
       "<td>0.2628550</td></tr>\n",
       "<tr><td>15</td>\n",
       "<td>0.8999006</td>\n",
       "<td>0.0157667</td>\n",
       "<td>0.0535273</td>\n",
       "<td>1.1094501</td>\n",
       "<td>0.0142045</td>\n",
       "<td>0.0216630</td>\n",
       "<td>0.2944146</td>\n",
       "<td>0.2938268</td>\n",
       "<td>0.0053505</td>\n",
       "<td>0.9983949</td>\n",
       "<td>-94.6472652</td>\n",
       "<td>10.9450146</td>\n",
       "<td>0.1340733</td></tr>\n",
       "<tr><td>16</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0044344</td>\n",
       "<td>0.0160354</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0042553</td>\n",
       "<td>0.0106067</td>\n",
       "<td>0.2653699</td>\n",
       "<td>0.2654766</td>\n",
       "<td>0.0016051</td>\n",
       "<td>1.0</td>\n",
       "<td>-98.3964573</td>\n",
       "<td>0.0</td>\n",
       "<td>0.0</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "</div></div>\n",
       "<div style='margin: 1em 0 1em 0;'><pre style='margin: 1em 0 1em 0;'>ModelMetricsBinomialGLM: stackedensemble\n",
       "** Reported on cross-validation data. **\n",
       "\n",
       "MSE: 0.13343936715812627\n",
       "RMSE: 0.36529353560955097\n",
       "LogLoss: 0.41095045439479133\n",
       "AUC: 0.8500058323473151\n",
       "AUCPR: 0.6660452986059002\n",
       "Gini: 0.7000116646946302\n",
       "Null degrees of freedom: 7042\n",
       "Residual degrees of freedom: 7039\n",
       "Null deviance: 8151.868367609408\n",
       "Residual deviance: 5788.648100605031\n",
       "AIC: 5796.648100605031</pre>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-17.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-17 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-17 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-17 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-17 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-17 .h2o-table th,\n",
       "#h2o-table-17 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-17 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-17\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.288417727959037</caption>\n",
       "    <thead><tr><th></th>\n",
       "<th>No</th>\n",
       "<th>Yes</th>\n",
       "<th>Error</th>\n",
       "<th>Rate</th></tr></thead>\n",
       "    <tbody><tr><td>No</td>\n",
       "<td>3937.0</td>\n",
       "<td>1237.0</td>\n",
       "<td>0.2391</td>\n",
       "<td> (1237.0/5174.0)</td></tr>\n",
       "<tr><td>Yes</td>\n",
       "<td>407.0</td>\n",
       "<td>1462.0</td>\n",
       "<td>0.2178</td>\n",
       "<td> (407.0/1869.0)</td></tr>\n",
       "<tr><td>Total</td>\n",
       "<td>4344.0</td>\n",
       "<td>2699.0</td>\n",
       "<td>0.2334</td>\n",
       "<td> (1644.0/7043.0)</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "</div>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-18.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-18 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-18 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-18 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-18 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-18 .h2o-table th,\n",
       "#h2o-table-18 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-18 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-18\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>Maximum Metrics: Maximum metrics at their respective thresholds</caption>\n",
       "    <thead><tr><th>metric</th>\n",
       "<th>threshold</th>\n",
       "<th>value</th>\n",
       "<th>idx</th></tr></thead>\n",
       "    <tbody><tr><td>max f1</td>\n",
       "<td>0.2884177</td>\n",
       "<td>0.6401051</td>\n",
       "<td>234.0</td></tr>\n",
       "<tr><td>max f2</td>\n",
       "<td>0.1585833</td>\n",
       "<td>0.7515611</td>\n",
       "<td>296.0</td></tr>\n",
       "<tr><td>max f0point5</td>\n",
       "<td>0.5163654</td>\n",
       "<td>0.6403497</td>\n",
       "<td>135.0</td></tr>\n",
       "<tr><td>max accuracy</td>\n",
       "<td>0.5163654</td>\n",
       "<td>0.8077524</td>\n",
       "<td>135.0</td></tr>\n",
       "<tr><td>max precision</td>\n",
       "<td>0.8862381</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max recall</td>\n",
       "<td>0.0071124</td>\n",
       "<td>1.0</td>\n",
       "<td>397.0</td></tr>\n",
       "<tr><td>max specificity</td>\n",
       "<td>0.8862381</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max absolute_mcc</td>\n",
       "<td>0.2650552</td>\n",
       "<td>0.4937726</td>\n",
       "<td>245.0</td></tr>\n",
       "<tr><td>max min_per_class_accuracy</td>\n",
       "<td>0.2963764</td>\n",
       "<td>0.7683253</td>\n",
       "<td>230.0</td></tr>\n",
       "<tr><td>max mean_per_class_accuracy</td>\n",
       "<td>0.2650552</td>\n",
       "<td>0.7753190</td>\n",
       "<td>245.0</td></tr>\n",
       "<tr><td>max tns</td>\n",
       "<td>0.8862381</td>\n",
       "<td>5174.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max fns</td>\n",
       "<td>0.8862381</td>\n",
       "<td>1867.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max fps</td>\n",
       "<td>0.0049395</td>\n",
       "<td>5174.0</td>\n",
       "<td>399.0</td></tr>\n",
       "<tr><td>max tps</td>\n",
       "<td>0.0071124</td>\n",
       "<td>1869.0</td>\n",
       "<td>397.0</td></tr>\n",
       "<tr><td>max tnr</td>\n",
       "<td>0.8862381</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max fnr</td>\n",
       "<td>0.8862381</td>\n",
       "<td>0.9989299</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>max fpr</td>\n",
       "<td>0.0049395</td>\n",
       "<td>1.0</td>\n",
       "<td>399.0</td></tr>\n",
       "<tr><td>max tpr</td>\n",
       "<td>0.0071124</td>\n",
       "<td>1.0</td>\n",
       "<td>397.0</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "</div>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-19.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-19 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-19 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-19 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-19 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-19 .h2o-table th,\n",
       "#h2o-table-19 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-19 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-19\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>Gains/Lift Table: Avg response rate: 26.54 %, avg score: 26.54 %</caption>\n",
       "    <thead><tr><th>group</th>\n",
       "<th>cumulative_data_fraction</th>\n",
       "<th>lower_threshold</th>\n",
       "<th>lift</th>\n",
       "<th>cumulative_lift</th>\n",
       "<th>response_rate</th>\n",
       "<th>score</th>\n",
       "<th>cumulative_response_rate</th>\n",
       "<th>cumulative_score</th>\n",
       "<th>capture_rate</th>\n",
       "<th>cumulative_capture_rate</th>\n",
       "<th>gain</th>\n",
       "<th>cumulative_gain</th>\n",
       "<th>kolmogorov_smirnov</th></tr></thead>\n",
       "    <tbody><tr><td>1</td>\n",
       "<td>0.0100809</td>\n",
       "<td>0.8213740</td>\n",
       "<td>3.4498753</td>\n",
       "<td>3.4498753</td>\n",
       "<td>0.9154930</td>\n",
       "<td>0.8382947</td>\n",
       "<td>0.9154930</td>\n",
       "<td>0.8382947</td>\n",
       "<td>0.0347780</td>\n",
       "<td>0.0347780</td>\n",
       "<td>244.9875282</td>\n",
       "<td>244.9875282</td>\n",
       "<td>0.0336183</td></tr>\n",
       "<tr><td>2</td>\n",
       "<td>0.0200199</td>\n",
       "<td>0.7955516</td>\n",
       "<td>3.1223267</td>\n",
       "<td>3.2872625</td>\n",
       "<td>0.8285714</td>\n",
       "<td>0.8065485</td>\n",
       "<td>0.8723404</td>\n",
       "<td>0.8225342</td>\n",
       "<td>0.0310326</td>\n",
       "<td>0.0658106</td>\n",
       "<td>212.2326683</td>\n",
       "<td>228.7262502</td>\n",
       "<td>0.0623317</td></tr>\n",
       "<tr><td>3</td>\n",
       "<td>0.0301008</td>\n",
       "<td>0.7787034</td>\n",
       "<td>3.0783503</td>\n",
       "<td>3.2172966</td>\n",
       "<td>0.8169014</td>\n",
       "<td>0.7870960</td>\n",
       "<td>0.8537736</td>\n",
       "<td>0.8106657</td>\n",
       "<td>0.0310326</td>\n",
       "<td>0.0968432</td>\n",
       "<td>207.8350251</td>\n",
       "<td>221.7296607</td>\n",
       "<td>0.0908517</td></tr>\n",
       "<tr><td>4</td>\n",
       "<td>0.0400398</td>\n",
       "<td>0.7650570</td>\n",
       "<td>3.1223267</td>\n",
       "<td>3.1937225</td>\n",
       "<td>0.8285714</td>\n",
       "<td>0.7725948</td>\n",
       "<td>0.8475177</td>\n",
       "<td>0.8012155</td>\n",
       "<td>0.0310326</td>\n",
       "<td>0.1278759</td>\n",
       "<td>212.2326683</td>\n",
       "<td>219.3722513</td>\n",
       "<td>0.1195651</td></tr>\n",
       "<tr><td>5</td>\n",
       "<td>0.0501207</td>\n",
       "<td>0.7455306</td>\n",
       "<td>3.0783503</td>\n",
       "<td>3.1705173</td>\n",
       "<td>0.8169014</td>\n",
       "<td>0.7544141</td>\n",
       "<td>0.8413598</td>\n",
       "<td>0.7918022</td>\n",
       "<td>0.0310326</td>\n",
       "<td>0.1589085</td>\n",
       "<td>207.8350251</td>\n",
       "<td>217.0517327</td>\n",
       "<td>0.1480852</td></tr>\n",
       "<tr><td>6</td>\n",
       "<td>0.1000994</td>\n",
       "<td>0.6592997</td>\n",
       "<td>2.6228401</td>\n",
       "<td>2.8970671</td>\n",
       "<td>0.6960227</td>\n",
       "<td>0.7008956</td>\n",
       "<td>0.7687943</td>\n",
       "<td>0.7464134</td>\n",
       "<td>0.1310861</td>\n",
       "<td>0.2899946</td>\n",
       "<td>162.2840058</td>\n",
       "<td>189.7067116</td>\n",
       "<td>0.2584910</td></tr>\n",
       "<tr><td>7</td>\n",
       "<td>0.1500781</td>\n",
       "<td>0.5946767</td>\n",
       "<td>2.3337924</td>\n",
       "<td>2.7094865</td>\n",
       "<td>0.6193182</td>\n",
       "<td>0.6260252</td>\n",
       "<td>0.7190161</td>\n",
       "<td>0.7063219</td>\n",
       "<td>0.1166399</td>\n",
       "<td>0.4066346</td>\n",
       "<td>133.3792378</td>\n",
       "<td>170.9486503</td>\n",
       "<td>0.3492322</td></tr>\n",
       "<tr><td>8</td>\n",
       "<td>0.2000568</td>\n",
       "<td>0.5283296</td>\n",
       "<td>2.0554502</td>\n",
       "<td>2.5460935</td>\n",
       "<td>0.5454545</td>\n",
       "<td>0.5608333</td>\n",
       "<td>0.6756565</td>\n",
       "<td>0.6699756</td>\n",
       "<td>0.1027287</td>\n",
       "<td>0.5093633</td>\n",
       "<td>105.5450168</td>\n",
       "<td>154.6093465</td>\n",
       "<td>0.4210370</td></tr>\n",
       "<tr><td>9</td>\n",
       "<td>0.3000142</td>\n",
       "<td>0.3857774</td>\n",
       "<td>1.6165259</td>\n",
       "<td>2.2363843</td>\n",
       "<td>0.4289773</td>\n",
       "<td>0.4583619</td>\n",
       "<td>0.5934690</td>\n",
       "<td>0.5994711</td>\n",
       "<td>0.1615837</td>\n",
       "<td>0.6709470</td>\n",
       "<td>61.6525913</td>\n",
       "<td>123.6384257</td>\n",
       "<td>0.5049246</td></tr>\n",
       "<tr><td>10</td>\n",
       "<td>0.3999716</td>\n",
       "<td>0.2752011</td>\n",
       "<td>1.2632454</td>\n",
       "<td>1.9931859</td>\n",
       "<td>0.3352273</td>\n",
       "<td>0.3234361</td>\n",
       "<td>0.5289315</td>\n",
       "<td>0.5304868</td>\n",
       "<td>0.1262707</td>\n",
       "<td>0.7972178</td>\n",
       "<td>26.3245416</td>\n",
       "<td>99.3185910</td>\n",
       "<td>0.5407431</td></tr>\n",
       "<tr><td>11</td>\n",
       "<td>0.5000710</td>\n",
       "<td>0.1822800</td>\n",
       "<td>0.8391873</td>\n",
       "<td>1.7621896</td>\n",
       "<td>0.2226950</td>\n",
       "<td>0.2273001</td>\n",
       "<td>0.4676320</td>\n",
       "<td>0.4697978</td>\n",
       "<td>0.0840021</td>\n",
       "<td>0.8812199</td>\n",
       "<td>-16.0812662</td>\n",
       "<td>76.2189603</td>\n",
       "<td>0.5188310</td></tr>\n",
       "<tr><td>12</td>\n",
       "<td>0.6000284</td>\n",
       "<td>0.1103539</td>\n",
       "<td>0.5620372</td>\n",
       "<td>1.5622589</td>\n",
       "<td>0.1491477</td>\n",
       "<td>0.1441392</td>\n",
       "<td>0.4145764</td>\n",
       "<td>0.4155471</td>\n",
       "<td>0.0561798</td>\n",
       "<td>0.9373997</td>\n",
       "<td>-43.7962845</td>\n",
       "<td>56.2258859</td>\n",
       "<td>0.4592396</td></tr>\n",
       "<tr><td>13</td>\n",
       "<td>0.6999858</td>\n",
       "<td>0.0596438</td>\n",
       "<td>0.3372223</td>\n",
       "<td>1.3873246</td>\n",
       "<td>0.0894886</td>\n",
       "<td>0.0830578</td>\n",
       "<td>0.3681542</td>\n",
       "<td>0.3680679</td>\n",
       "<td>0.0337079</td>\n",
       "<td>0.9711075</td>\n",
       "<td>-66.2777707</td>\n",
       "<td>38.7324632</td>\n",
       "<td>0.3690588</td></tr>\n",
       "<tr><td>14</td>\n",
       "<td>0.7999432</td>\n",
       "<td>0.0309274</td>\n",
       "<td>0.1712875</td>\n",
       "<td>1.2353740</td>\n",
       "<td>0.0454545</td>\n",
       "<td>0.0445700</td>\n",
       "<td>0.3278310</td>\n",
       "<td>0.3276450</td>\n",
       "<td>0.0171215</td>\n",
       "<td>0.9882290</td>\n",
       "<td>-82.8712486</td>\n",
       "<td>23.5373952</td>\n",
       "<td>0.2563001</td></tr>\n",
       "<tr><td>15</td>\n",
       "<td>0.8999006</td>\n",
       "<td>0.0159662</td>\n",
       "<td>0.0695856</td>\n",
       "<td>1.1058828</td>\n",
       "<td>0.0184659</td>\n",
       "<td>0.0226224</td>\n",
       "<td>0.2934680</td>\n",
       "<td>0.2937643</td>\n",
       "<td>0.0069556</td>\n",
       "<td>0.9951846</td>\n",
       "<td>-93.0414447</td>\n",
       "<td>10.5882782</td>\n",
       "<td>0.1297033</td></tr>\n",
       "<tr><td>16</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0041373</td>\n",
       "<td>0.0481063</td>\n",
       "<td>1.0</td>\n",
       "<td>0.0127660</td>\n",
       "<td>0.0107384</td>\n",
       "<td>0.2653699</td>\n",
       "<td>0.2654336</td>\n",
       "<td>0.0048154</td>\n",
       "<td>1.0</td>\n",
       "<td>-95.1893719</td>\n",
       "<td>0.0</td>\n",
       "<td>0.0</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "</div></div>\n",
       "<div style='margin: 1em 0 1em 0;'>\n",
       "<style>\n",
       "\n",
       "#h2o-table-20.h2o-container {\n",
       "  overflow-x: auto;\n",
       "}\n",
       "#h2o-table-20 .h2o-table {\n",
       "  /* width: 100%; */\n",
       "  margin-top: 1em;\n",
       "  margin-bottom: 1em;\n",
       "}\n",
       "#h2o-table-20 .h2o-table caption {\n",
       "  white-space: nowrap;\n",
       "  caption-side: top;\n",
       "  text-align: left;\n",
       "  /* margin-left: 1em; */\n",
       "  margin: 0;\n",
       "  font-size: larger;\n",
       "}\n",
       "#h2o-table-20 .h2o-table thead {\n",
       "  white-space: nowrap; \n",
       "  position: sticky;\n",
       "  top: 0;\n",
       "  box-shadow: 0 -1px inset;\n",
       "}\n",
       "#h2o-table-20 .h2o-table tbody {\n",
       "  overflow: auto;\n",
       "}\n",
       "#h2o-table-20 .h2o-table th,\n",
       "#h2o-table-20 .h2o-table td {\n",
       "  text-align: right;\n",
       "  /* border: 1px solid; */\n",
       "}\n",
       "#h2o-table-20 .h2o-table tr:nth-child(even) {\n",
       "  /* background: #F5F5F5 */\n",
       "}\n",
       "\n",
       "</style>      \n",
       "<div id=\"h2o-table-20\" class=\"h2o-container\">\n",
       "  <table class=\"h2o-table\">\n",
       "    <caption>Cross-Validation Metrics Summary: </caption>\n",
       "    <thead><tr><th></th>\n",
       "<th>mean</th>\n",
       "<th>sd</th>\n",
       "<th>cv_1_valid</th>\n",
       "<th>cv_2_valid</th>\n",
       "<th>cv_3_valid</th>\n",
       "<th>cv_4_valid</th>\n",
       "<th>cv_5_valid</th></tr></thead>\n",
       "    <tbody><tr><td>accuracy</td>\n",
       "<td>0.7810399</td>\n",
       "<td>0.0178729</td>\n",
       "<td>0.7913521</td>\n",
       "<td>0.7698687</td>\n",
       "<td>0.8004246</td>\n",
       "<td>0.7559775</td>\n",
       "<td>0.7875767</td></tr>\n",
       "<tr><td>aic</td>\n",
       "<td>1164.9021</td>\n",
       "<td>61.866287</td>\n",
       "<td>1207.9692</td>\n",
       "<td>1216.1278</td>\n",
       "<td>1133.7238</td>\n",
       "<td>1196.0629</td>\n",
       "<td>1070.6268</td></tr>\n",
       "<tr><td>auc</td>\n",
       "<td>0.8502371</td>\n",
       "<td>0.0063955</td>\n",
       "<td>0.8421056</td>\n",
       "<td>0.8503820</td>\n",
       "<td>0.8568639</td>\n",
       "<td>0.8457873</td>\n",
       "<td>0.8560467</td></tr>\n",
       "<tr><td>err</td>\n",
       "<td>0.2189601</td>\n",
       "<td>0.0178729</td>\n",
       "<td>0.2086479</td>\n",
       "<td>0.2301313</td>\n",
       "<td>0.1995754</td>\n",
       "<td>0.2440225</td>\n",
       "<td>0.2124233</td></tr>\n",
       "<tr><td>err_count</td>\n",
       "<td>308.6</td>\n",
       "<td>30.810713</td>\n",
       "<td>304.0</td>\n",
       "<td>333.0</td>\n",
       "<td>282.0</td>\n",
       "<td>347.0</td>\n",
       "<td>277.0</td></tr>\n",
       "<tr><td>f0point5</td>\n",
       "<td>0.5967632</td>\n",
       "<td>0.0185078</td>\n",
       "<td>0.5931818</td>\n",
       "<td>0.593415</td>\n",
       "<td>0.6143264</td>\n",
       "<td>0.5692979</td>\n",
       "<td>0.6135949</td></tr>\n",
       "<tr><td>f1</td>\n",
       "<td>0.6467064</td>\n",
       "<td>0.0102975</td>\n",
       "<td>0.6319613</td>\n",
       "<td>0.6505771</td>\n",
       "<td>0.6475</td>\n",
       "<td>0.6433711</td>\n",
       "<td>0.6601227</td></tr>\n",
       "<tr><td>f2</td>\n",
       "<td>0.7068882</td>\n",
       "<td>0.0261811</td>\n",
       "<td>0.6761658</td>\n",
       "<td>0.7199257</td>\n",
       "<td>0.6844609</td>\n",
       "<td>0.7396030</td>\n",
       "<td>0.7142857</td></tr>\n",
       "<tr><td>lift_top_group</td>\n",
       "<td>3.4178927</td>\n",
       "<td>0.2412670</td>\n",
       "<td>3.6952899</td>\n",
       "<td>3.1351666</td>\n",
       "<td>3.623077</td>\n",
       "<td>3.2346456</td>\n",
       "<td>3.4012842</td></tr>\n",
       "<tr><td>loglikelihood</td>\n",
       "<td>0.0</td>\n",
       "<td>0.0</td>\n",
       "<td>0.0</td>\n",
       "<td>0.0</td>\n",
       "<td>0.0</td>\n",
       "<td>0.0</td>\n",
       "<td>0.0</td></tr>\n",
       "<tr><td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td>\n",
       "<td>---</td></tr>\n",
       "<tr><td>mean_per_class_error</td>\n",
       "<td>0.2277239</td>\n",
       "<td>0.0053745</td>\n",
       "<td>0.2358304</td>\n",
       "<td>0.2285458</td>\n",
       "<td>0.2285968</td>\n",
       "<td>0.2232446</td>\n",
       "<td>0.2224020</td></tr>\n",
       "<tr><td>mse</td>\n",
       "<td>0.1334272</td>\n",
       "<td>0.0035193</td>\n",
       "<td>0.1321111</td>\n",
       "<td>0.1364434</td>\n",
       "<td>0.1282075</td>\n",
       "<td>0.1368108</td>\n",
       "<td>0.1335631</td></tr>\n",
       "<tr><td>null_deviance</td>\n",
       "<td>1630.3737</td>\n",
       "<td>65.76145</td>\n",
       "<td>1648.7882</td>\n",
       "<td>1707.6018</td>\n",
       "<td>1613.0315</td>\n",
       "<td>1652.984</td>\n",
       "<td>1529.4628</td></tr>\n",
       "<tr><td>pr_auc</td>\n",
       "<td>0.6674669</td>\n",
       "<td>0.0153246</td>\n",
       "<td>0.6578724</td>\n",
       "<td>0.6730958</td>\n",
       "<td>0.675541</td>\n",
       "<td>0.6461514</td>\n",
       "<td>0.6846741</td></tr>\n",
       "<tr><td>precision</td>\n",
       "<td>0.5678514</td>\n",
       "<td>0.0255227</td>\n",
       "<td>0.569869</td>\n",
       "<td>0.5605786</td>\n",
       "<td>0.5940367</td>\n",
       "<td>0.5287162</td>\n",
       "<td>0.5860566</td></tr>\n",
       "<tr><td>r2</td>\n",
       "<td>0.3154401</td>\n",
       "<td>0.0136175</td>\n",
       "<td>0.3001858</td>\n",
       "<td>0.3178450</td>\n",
       "<td>0.3296201</td>\n",
       "<td>0.3025007</td>\n",
       "<td>0.3270491</td></tr>\n",
       "<tr><td>recall</td>\n",
       "<td>0.7545836</td>\n",
       "<td>0.0469247</td>\n",
       "<td>0.7092391</td>\n",
       "<td>0.775</td>\n",
       "<td>0.7115384</td>\n",
       "<td>0.8215223</td>\n",
       "<td>0.755618</td></tr>\n",
       "<tr><td>residual_deviance</td>\n",
       "<td>1157.7021</td>\n",
       "<td>61.52713</td>\n",
       "<td>1199.9692</td>\n",
       "<td>1210.1278</td>\n",
       "<td>1125.7238</td>\n",
       "<td>1188.0629</td>\n",
       "<td>1064.6268</td></tr>\n",
       "<tr><td>rmse</td>\n",
       "<td>0.3652513</td>\n",
       "<td>0.0048323</td>\n",
       "<td>0.3634709</td>\n",
       "<td>0.3693825</td>\n",
       "<td>0.3580608</td>\n",
       "<td>0.3698794</td>\n",
       "<td>0.3654629</td></tr>\n",
       "<tr><td>specificity</td>\n",
       "<td>0.7899685</td>\n",
       "<td>0.0402957</td>\n",
       "<td>0.8191001</td>\n",
       "<td>0.7679083</td>\n",
       "<td>0.8312679</td>\n",
       "<td>0.7319885</td>\n",
       "<td>0.7995781</td></tr></tbody>\n",
       "  </table>\n",
       "</div>\n",
       "<pre style='font-size: smaller; margin-bottom: 1em;'>[24 rows x 8 columns]</pre></div><pre style=\"font-size: smaller; margin: 1em 0 0 0;\">\n",
       "\n",
       "[tips]\n",
       "Use `model.explain()` to inspect the model.\n",
       "--\n",
       "Use `h2o.display.toggle_user_tips()` to switch on/off this section.</pre>"
      ],
      "text/plain": [
       "Model Details\n",
       "=============\n",
       "H2OStackedEnsembleEstimator : Stacked Ensemble\n",
       "Model Key: StackedEnsemble_BestOfFamily_2_AutoML_2_20250117_182721\n",
       "\n",
       "\n",
       "Model Summary for Stacked Ensemble: \n",
       "key                                   value\n",
       "------------------------------------  ----------------\n",
       "Stacking strategy                     cross_validation\n",
       "Number of base models (used / total)  2/4\n",
       "# GBM base models (used / total)      1/1\n",
       "# XGBoost base models (used / total)  0/1\n",
       "# GLM base models (used / total)      1/1\n",
       "# DRF base models (used / total)      0/1\n",
       "Metalearner algorithm                 GLM\n",
       "Metalearner fold assignment scheme    Random\n",
       "Metalearner nfolds                    5\n",
       "Metalearner fold_column\n",
       "Custom metalearner hyperparameters    None\n",
       "\n",
       "ModelMetricsBinomialGLM: stackedensemble\n",
       "** Reported on train data. **\n",
       "\n",
       "MSE: 0.12280258481192449\n",
       "RMSE: 0.3504319974145119\n",
       "LogLoss: 0.38118316635829386\n",
       "AUC: 0.8755385872855243\n",
       "AUCPR: 0.7225887060960569\n",
       "Gini: 0.7510771745710485\n",
       "Null degrees of freedom: 7042\n",
       "Residual degrees of freedom: 7040\n",
       "Null deviance: 8150.145899019416\n",
       "Residual deviance: 5369.346081322927\n",
       "AIC: 5375.346081322927\n",
       "\n",
       "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.28863204785206925\n",
       "       No    Yes    Error    Rate\n",
       "-----  ----  -----  -------  ---------------\n",
       "No     4002  1172   0.2265   (1172.0/5174.0)\n",
       "Yes    341   1528   0.1825   (341.0/1869.0)\n",
       "Total  4343  2700   0.2148   (1513.0/7043.0)\n",
       "\n",
       "Maximum Metrics: Maximum metrics at their respective thresholds\n",
       "metric                       threshold    value     idx\n",
       "---------------------------  -----------  --------  -----\n",
       "max f1                       0.288632     0.668855  229\n",
       "max f2                       0.163993     0.770925  289\n",
       "max f0point5                 0.553383     0.67799   117\n",
       "max accuracy                 0.509217     0.823371  137\n",
       "max precision                0.868569     1         0\n",
       "max recall                   0.0125838    1         391\n",
       "max specificity              0.868569     1         0\n",
       "max absolute_mcc             0.288632     0.536726  229\n",
       "max min_per_class_accuracy   0.309673     0.788122  220\n",
       "max mean_per_class_accuracy  0.288632     0.795516  229\n",
       "max tns                      0.868569     5174      0\n",
       "max fns                      0.868569     1867      0\n",
       "max fps                      0.00503849   5174      399\n",
       "max tps                      0.0125838    1869      391\n",
       "max tnr                      0.868569     1         0\n",
       "max fnr                      0.868569     0.99893   0\n",
       "max fpr                      0.00503849   1         399\n",
       "max tpr                      0.0125838    1         391\n",
       "\n",
       "Gains/Lift Table: Avg response rate: 26.54 %, avg score: 26.55 %\n",
       "group    cumulative_data_fraction    lower_threshold    lift       cumulative_lift    response_rate    score      cumulative_response_rate    cumulative_score    capture_rate    cumulative_capture_rate    gain      cumulative_gain    kolmogorov_smirnov\n",
       "-------  --------------------------  -----------------  ---------  -----------------  ---------------  ---------  --------------------------  ------------------  --------------  -------------------------  --------  -----------------  --------------------\n",
       "1        0.0100809                   0.824912           3.66218    3.66218            0.971831         0.840837   0.971831                    0.840837            0.0369181       0.0369181                  266.218   266.218            0.0365316\n",
       "2        0.0200199                   0.801617           3.33766    3.50107            0.885714         0.813029   0.929078                    0.827031            0.0331728       0.070091                   233.766   250.107            0.0681582\n",
       "3        0.0301008                   0.784829           3.3968     3.46615            0.901408         0.792067   0.919811                    0.815322            0.0342429       0.104334                   239.68    246.615            0.101048\n",
       "4        0.0400398                   0.769314           3.44533    3.46098            0.914286         0.777204   0.91844                     0.80586             0.0342429       0.138577                   244.533   246.098            0.134131\n",
       "5        0.0501207                   0.754344           3.1845     3.40537            0.84507          0.761895   0.903683                    0.797017            0.0321027       0.17068                    218.45    240.537            0.164108\n",
       "6        0.100099                    0.665217           2.82624    3.11622            0.75             0.708144   0.82695                     0.752643            0.141252        0.311932                   182.624   211.622            0.288352\n",
       "7        0.150078                    0.594758           2.50508    2.9127             0.664773         0.629425   0.772942                    0.71161             0.125201        0.437132                   150.508   191.27             0.390746\n",
       "8        0.200057                    0.530679           2.09827    2.70924            0.556818         0.563169   0.71895                     0.674526            0.104869        0.542001                   109.827   170.924            0.465465\n",
       "9        0.300014                    0.381833           1.654      2.35766            0.43892          0.456573   0.625651                    0.601909            0.165329        0.70733                    65.3995   135.766            0.55445\n",
       "10       0.399972                    0.272804           1.23648    2.07746            0.328125         0.325797   0.551296                    0.532906            0.123596        0.830926                   23.6482   107.746            0.586627\n",
       "11       0.500071                    0.177844           0.753665   1.81248            0.2              0.224613   0.480977                    0.471194            0.0754414       0.906367                   -24.6335  81.2477            0.553062\n",
       "12       0.600028                    0.110988           0.487099   1.59168            0.129261         0.142033   0.422385                    0.41636             0.0486891       0.955056                   -51.2901  59.1685            0.483274\n",
       "13       0.699986                    0.0574955          0.278342   1.40414            0.0738636        0.0820331  0.372617                    0.368618            0.0278224       0.982879                   -72.1658  40.4141            0.385082\n",
       "14       0.799943                    0.0293164          0.101702   1.24139            0.0269886        0.0422363  0.329428                    0.327835            0.0101659       0.993044                   -89.8298  24.1394            0.262855\n",
       "15       0.899901                    0.0157667          0.0535273  1.10945            0.0142045        0.021663   0.294415                    0.293827            0.00535045      0.998395                   -94.6473  10.945             0.134073\n",
       "16       1                           0.00443439         0.0160354  1                  0.00425532       0.0106067  0.26537                     0.265477            0.00160514      1                          -98.3965  0                  0\n",
       "\n",
       "ModelMetricsBinomialGLM: stackedensemble\n",
       "** Reported on cross-validation data. **\n",
       "\n",
       "MSE: 0.13343936715812627\n",
       "RMSE: 0.36529353560955097\n",
       "LogLoss: 0.41095045439479133\n",
       "AUC: 0.8500058323473151\n",
       "AUCPR: 0.6660452986059002\n",
       "Gini: 0.7000116646946302\n",
       "Null degrees of freedom: 7042\n",
       "Residual degrees of freedom: 7039\n",
       "Null deviance: 8151.868367609408\n",
       "Residual deviance: 5788.648100605031\n",
       "AIC: 5796.648100605031\n",
       "\n",
       "Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.288417727959037\n",
       "       No    Yes    Error    Rate\n",
       "-----  ----  -----  -------  ---------------\n",
       "No     3937  1237   0.2391   (1237.0/5174.0)\n",
       "Yes    407   1462   0.2178   (407.0/1869.0)\n",
       "Total  4344  2699   0.2334   (1644.0/7043.0)\n",
       "\n",
       "Maximum Metrics: Maximum metrics at their respective thresholds\n",
       "metric                       threshold    value     idx\n",
       "---------------------------  -----------  --------  -----\n",
       "max f1                       0.288418     0.640105  234\n",
       "max f2                       0.158583     0.751561  296\n",
       "max f0point5                 0.516365     0.64035   135\n",
       "max accuracy                 0.516365     0.807752  135\n",
       "max precision                0.886238     1         0\n",
       "max recall                   0.00711245   1         397\n",
       "max specificity              0.886238     1         0\n",
       "max absolute_mcc             0.265055     0.493773  245\n",
       "max min_per_class_accuracy   0.296376     0.768325  230\n",
       "max mean_per_class_accuracy  0.265055     0.775319  245\n",
       "max tns                      0.886238     5174      0\n",
       "max fns                      0.886238     1867      0\n",
       "max fps                      0.00493952   5174      399\n",
       "max tps                      0.00711245   1869      397\n",
       "max tnr                      0.886238     1         0\n",
       "max fnr                      0.886238     0.99893   0\n",
       "max fpr                      0.00493952   1         399\n",
       "max tpr                      0.00711245   1         397\n",
       "\n",
       "Gains/Lift Table: Avg response rate: 26.54 %, avg score: 26.54 %\n",
       "group    cumulative_data_fraction    lower_threshold    lift       cumulative_lift    response_rate    score      cumulative_response_rate    cumulative_score    capture_rate    cumulative_capture_rate    gain      cumulative_gain    kolmogorov_smirnov\n",
       "-------  --------------------------  -----------------  ---------  -----------------  ---------------  ---------  --------------------------  ------------------  --------------  -------------------------  --------  -----------------  --------------------\n",
       "1        0.0100809                   0.821374           3.44988    3.44988            0.915493         0.838295   0.915493                    0.838295            0.034778        0.034778                   244.988   244.988            0.0336183\n",
       "2        0.0200199                   0.795552           3.12233    3.28726            0.828571         0.806549   0.87234                     0.822534            0.0310326       0.0658106                  212.233   228.726            0.0623317\n",
       "3        0.0301008                   0.778703           3.07835    3.2173             0.816901         0.787096   0.853774                    0.810666            0.0310326       0.0968432                  207.835   221.73             0.0908517\n",
       "4        0.0400398                   0.765057           3.12233    3.19372            0.828571         0.772595   0.847518                    0.801215            0.0310326       0.127876                   212.233   219.372            0.119565\n",
       "5        0.0501207                   0.745531           3.07835    3.17052            0.816901         0.754414   0.84136                     0.791802            0.0310326       0.158909                   207.835   217.052            0.148085\n",
       "6        0.100099                    0.6593             2.62284    2.89707            0.696023         0.700896   0.768794                    0.746413            0.131086        0.289995                   162.284   189.707            0.258491\n",
       "7        0.150078                    0.594677           2.33379    2.70949            0.619318         0.626025   0.719016                    0.706322            0.11664         0.406635                   133.379   170.949            0.349232\n",
       "8        0.200057                    0.52833            2.05545    2.54609            0.545455         0.560833   0.675656                    0.669976            0.102729        0.509363                   105.545   154.609            0.421037\n",
       "9        0.300014                    0.385777           1.61653    2.23638            0.428977         0.458362   0.593469                    0.599471            0.161584        0.670947                   61.6526   123.638            0.504925\n",
       "10       0.399972                    0.275201           1.26325    1.99319            0.335227         0.323436   0.528931                    0.530487            0.126271        0.797218                   26.3245   99.3186            0.540743\n",
       "11       0.500071                    0.18228            0.839187   1.76219            0.222695         0.2273     0.467632                    0.469798            0.0840021       0.88122                    -16.0813  76.219             0.518831\n",
       "12       0.600028                    0.110354           0.562037   1.56226            0.149148         0.144139   0.414576                    0.415547            0.0561798       0.9374                     -43.7963  56.2259            0.45924\n",
       "13       0.699986                    0.0596438          0.337222   1.38732            0.0894886        0.0830578  0.368154                    0.368068            0.0337079       0.971108                   -66.2778  38.7325            0.369059\n",
       "14       0.799943                    0.0309274          0.171288   1.23537            0.0454545        0.04457    0.327831                    0.327645            0.0171215       0.988229                   -82.8712  23.5374            0.2563\n",
       "15       0.899901                    0.0159662          0.0695856  1.10588            0.0184659        0.0226224  0.293468                    0.293764            0.00695559      0.995185                   -93.0414  10.5883            0.129703\n",
       "16       1                           0.00413732         0.0481063  1                  0.012766         0.0107384  0.26537                     0.265434            0.00481541      1                          -95.1894  0                  0\n",
       "\n",
       "Cross-Validation Metrics Summary: \n",
       "                      mean        sd            cv_1_valid    cv_2_valid    cv_3_valid    cv_4_valid    cv_5_valid\n",
       "--------------------  ----------  ------------  ------------  ------------  ------------  ------------  ------------\n",
       "accuracy              0.7810399   0.017872874   0.7913521     0.7698687     0.80042464    0.7559775     0.7875767\n",
       "aic                   1164.9021   61.866287     1207.9692     1216.1278     1133.7238     1196.0629     1070.6268\n",
       "auc                   0.85023713  0.006395509   0.84210557    0.85038203    0.8568639     0.8457873     0.85604674\n",
       "err                   0.21896008  0.017872874   0.2086479     0.23013131    0.19957536    0.2440225     0.21242331\n",
       "err_count             308.6       30.810713     304.0         333.0         282.0         347.0         277.0\n",
       "f0point5              0.5967632   0.018507762   0.5931818     0.593415      0.61432636    0.5692979     0.6135949\n",
       "f1                    0.6467064   0.010297492   0.6319613     0.6505771     0.6475        0.64337105    0.6601227\n",
       "f2                    0.7068882   0.026181143   0.6761658     0.7199257     0.6844609     0.73960304    0.71428573\n",
       "lift_top_group        3.4178927   0.24126701    3.6952899     3.1351666     3.623077      3.2346456     3.4012842\n",
       "loglikelihood         0.0         0.0           0.0           0.0           0.0           0.0           0.0\n",
       "---                   ---         ---           ---           ---           ---           ---           ---\n",
       "mean_per_class_error  0.22772393  0.005374468   0.23583038    0.22854584    0.22859684    0.2232446     0.22240198\n",
       "mse                   0.1334272   0.0035192964  0.13211112    0.13644344    0.12820753    0.13681076    0.13356312\n",
       "null_deviance         1630.3737   65.76145      1648.7882     1707.6018     1613.0315     1652.984      1529.4628\n",
       "pr_auc                0.66746694  0.015324555   0.65787244    0.6730958     0.675541      0.64615136    0.6846741\n",
       "precision             0.5678514   0.02552267    0.569869      0.56057864    0.5940367     0.5287162     0.58605665\n",
       "r2                    0.31544012  0.013617506   0.3001858     0.31784502    0.3296201     0.3025007     0.32704905\n",
       "recall                0.7545836   0.04692466    0.7092391     0.775         0.71153843    0.8215223     0.755618\n",
       "residual_deviance     1157.7021   61.52713      1199.9692     1210.1278     1125.7238     1188.0629     1064.6268\n",
       "rmse                  0.3652513   0.004832292   0.3634709     0.36938253    0.35806078    0.3698794     0.3654629\n",
       "specificity           0.78996855  0.04029568    0.8191001     0.76790833    0.8312679     0.7319885     0.7995781\n",
       "[24 rows x 8 columns]\n",
       "\n",
       "\n",
       "[tips]\n",
       "Use `model.explain()` to inspect the model.\n",
       "--\n",
       "Use `h2o.display.toggle_user_tips()` to switch on/off this section."
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load the model\n",
    "model = h2o.get_model(ml_agent.get_best_model_id())\n",
    "\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Make Predictions\n",
    "\n",
    "Run the code to make predictions using the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parse progress: |████████████████████████████████████████████████████████████████| (done) 100%\n",
      "stackedensemble prediction progress: |███████████████████████████████████████████| (done) 100%\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class='dataframe'>\n",
       "<thead>\n",
       "<tr><th>predict  </th><th style=\"text-align: right;\">      No</th><th style=\"text-align: right;\">      Yes</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td>Yes      </td><td style=\"text-align: right;\">0.335686</td><td style=\"text-align: right;\">0.664314 </td></tr>\n",
       "<tr><td>No       </td><td style=\"text-align: right;\">0.96382 </td><td style=\"text-align: right;\">0.0361802</td></tr>\n",
       "<tr><td>Yes      </td><td style=\"text-align: right;\">0.651524</td><td style=\"text-align: right;\">0.348476 </td></tr>\n",
       "<tr><td>No       </td><td style=\"text-align: right;\">0.971191</td><td style=\"text-align: right;\">0.028809 </td></tr>\n",
       "<tr><td>Yes      </td><td style=\"text-align: right;\">0.357133</td><td style=\"text-align: right;\">0.642867 </td></tr>\n",
       "<tr><td>Yes      </td><td style=\"text-align: right;\">0.168803</td><td style=\"text-align: right;\">0.831197 </td></tr>\n",
       "<tr><td>Yes      </td><td style=\"text-align: right;\">0.582106</td><td style=\"text-align: right;\">0.417894 </td></tr>\n",
       "<tr><td>No       </td><td style=\"text-align: right;\">0.753557</td><td style=\"text-align: right;\">0.246443 </td></tr>\n",
       "<tr><td>Yes      </td><td style=\"text-align: right;\">0.419517</td><td style=\"text-align: right;\">0.580483 </td></tr>\n",
       "<tr><td>No       </td><td style=\"text-align: right;\">0.982409</td><td style=\"text-align: right;\">0.0175906</td></tr>\n",
       "</tbody>\n",
       "</table><pre style='font-size: smaller; margin-bottom: 1em;'>[7043 rows x 3 columns]</pre>"
      ],
      "text/plain": [
       "predict          No        Yes\n",
       "---------  --------  ---------\n",
       "Yes        0.335686  0.664314\n",
       "No         0.96382   0.0361802\n",
       "Yes        0.651524  0.348476\n",
       "No         0.971191  0.028809\n",
       "Yes        0.357133  0.642867\n",
       "Yes        0.168803  0.831197\n",
       "Yes        0.582106  0.417894\n",
       "No         0.753557  0.246443\n",
       "Yes        0.419517  0.580483\n",
       "No         0.982409  0.0175906\n",
       "[7043 rows x 3 columns]\n"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make predictions using the loaded model\n",
    "model.predict(h2o.H2OFrame(df))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Want To Become A Full-Stack Generative AI Data Scientist?\n",
    "\n",
    "![Generative AI Data Scientist](../../img/become_a_generative_ai_data_scientist.jpg)\n",
    "\n",
    "I teach Generative AI Data Science to help you build AI-powered data science apps. [**Register for my next Generative AI for Data Scientists workshop here.**](https://learn.business-science.io/ai-register)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ds4b_301p_dev",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}