{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU",
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/rmcpantoja/piper/blob/master/notebooks/piper_multilingual_training_notebook.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# <font color=\"pink\"> **[Piper](https://github.com/rhasspy/piper) training notebook.**\n",
        "## ![Piper logo](https://contribute.rhasspy.org/img/logo.png)\n",
        "\n",
        "---\n",
        "\n",
        "- Notebook made by [rmcpantoja](http://github.com/rmcpantoja)\n",
        "- Collaborator: [Xx_Nessu_xX](https://fakeyou.com/profile/Xx_Nessu_xX)"
      ],
      "metadata": {
        "id": "eK3nmYDB6C1a"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# <font color=\"pink\">🔧 ***First steps.*** 🔧"
      ],
      "metadata": {
        "id": "AICh6p5OJybj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown ## <font color=\"pink\"> **Google Colab Anti-Disconnect.** 🔌\n",
        "#@markdown ---\n",
        "#@markdown #### Avoid automatic disconnection. Still, it will disconnect after <font color=\"orange\">**6 to 12 hours**</font>.\n",
        "\n",
        "import IPython\n",
        "js_code = '''\n",
        "function ClickConnect(){\n",
        "console.log(\"Working\");\n",
        "document.querySelector(\"colab-toolbar-button#connect\").click()\n",
        "}\n",
        "setInterval(ClickConnect,60000)\n",
        "'''\n",
        "display(IPython.display.Javascript(js_code))"
      ],
      "metadata": {
        "cellView": "form",
        "id": "qyxSMuzjfQrz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown ## <font color=\"pink\"> **Check GPU type.** 👁️\n",
        "#@markdown ---\n",
        "#@markdown #### A higher capable GPU can lead to faster training speeds. By default, you will have a <font color=\"orange\">**Tesla T4**</font>.\n",
        "!nvidia-smi"
      ],
      "metadata": {
        "cellView": "form",
        "id": "ygxzp-xHTC7T"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "sUNjId07JfAK"
      },
      "outputs": [],
      "source": [
        "#@markdown # <font color=\"pink\"> **Mount Google Drive.** 📂\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive', force_remount=True)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown # <font color=\"pink\"> **Install software.** 📦\n",
        "#@markdown ####In this cell the synthesizer and its necessary dependencies to execute the training will be installed. (this may take a while)\n",
        "\n",
        "#@markdown <font color=\"orange\">**Note: Please restart the runtime environment when the cell execution is finished. Then you can continue with the training section.**\n",
        "\n",
        "# clone:\n",
        "!git clone https://github.com/rmcpantoja/piper\n",
        "%cd piper/src/python\n",
        "!pip install --upgrade pip\n",
        "!pip install --upgrade wheel setuptools\n",
        "!pip install -r requirements.txt\n",
        "!pip install torchtext==0.12.0\n",
        "!pip install torchvision==0.12.0\n",
        "!bash build_monotonic_align.sh\n",
        "!apt-get install espeak-ng\n",
        "%cd /content"
      ],
      "metadata": {
        "cellView": "form",
        "id": "_XwmTVlcUgCh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# <font color=\"pink\"> 🤖 ***Training.*** 🤖"
      ],
      "metadata": {
        "id": "A3bMzEE0V5Ma"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown # <font color=\"pink\"> **1. Extract dataset.** 📥\n",
        "#@markdown ####Important: the audios must be in <font color=\"orange\">**wav format, (16000 or 22050hz, 16-bits, mono), and, for convenience, numbered. Example:**\n",
        "\n",
        "#@markdown * <font color=\"orange\">**1.wav**</font>\n",
        "#@markdown * <font color=\"orange\">**2.wav**</font>\n",
        "#@markdown * <font color=\"orange\">**3.wav**</font>\n",
        "#@markdown * <font color=\"orange\">**.....**</font>\n",
        "\n",
        "#@markdown ---\n",
        "\n",
        "%cd /content\n",
        "!mkdir /content/dataset\n",
        "%cd /content/dataset\n",
        "!mkdir /content/dataset/wavs\n",
        "#@markdown ### Audio dataset path to unzip\n",
        "zip_path = \"/content/drive/MyDrive/wavs.zip\" #@param {type:\"string\"}\n",
        "!unzip \"{zip_path}\" -d /content/dataset/wavs\n",
        "#@markdown ---"
      ],
      "metadata": {
        "cellView": "form",
        "id": "SvEGjf0aV8eg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown # <font color=\"pink\"> **2. Upload the transcript file.** 📝\n",
        "#@markdown ---\n",
        "#@markdown ####Important: the transcription means writing what the character says in each of the audios, and it must have the following structure:\n",
        "\n",
        "#@markdown * wavs/1.wav|This is what my character says in audio 1.\n",
        "#@markdown * wavs/2.wav|This, the text that the character says in audio 2.\n",
        "#@markdown * ...\n",
        "\n",
        "#@markdown And so on. In addition, the transcript must be in a .csv format. (UTF-8 without BOM)\n",
        "\n",
        "%cd /content/dataset\n",
        "from google.colab import files\n",
        "!rm /content/dataset/metadata.csv\n",
        "listfn, length = files.upload().popitem()\n",
        "if listfn != \"metadata.csv\":\n",
        "  !mv \"$listfn\" metadata.csv\n",
        "%cd .."
      ],
      "metadata": {
        "cellView": "form",
        "id": "E0W0OCvXXvue"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown # <font color=\"pink\"> **3. Preprocess dataset.** 🔄\n",
        "\n",
        "import os\n",
        "#@markdown ### First of all, select the language of your dataset.\n",
        "language = \"English (U.S.)\" #@param [\"Català\", \"Dansk\", \"Deutsch\", \"Ελληνικά\", \"English (British)\", \"English (U.S.)\", \"Español\", \"Suomi\", \"Français\", \"ქართული\", \"Icelandic\", \"Italiano\", \"қазақша\", \"नेपाली\", \"Nederlands\", \"Norsk\", \"Polski\", \"Português (Brasil)\", \"Русский\", \"Svenska\", \"украї́нська\", \"Tiếng Việt\", \"简体中文\"]\n",
        "#@markdown ---\n",
        "# language definition:\n",
        "languages = {\n",
        "    \"Català\": \"ca\",\n",
        "    \"Dansk\": \"da\",\n",
        "    \"Deutsch\": \"de\",\n",
        "    \"Ελληνικά\": \"grc\",\n",
        "    \"English (British)\": \"en\",\n",
        "    \"English (U.S.)\": \"en-us\",\n",
        "    \"Español\": \"es\",\n",
        "    \"Suomi\": \"fi\",\n",
        "    \"Français\": \"fr\",\n",
        "    \"Icelandic\": \"is\",\n",
        "    \"Italiano\": \"it\",\n",
        "    \"ქართული\": \"ka\",\n",
        "    \"қазақша\": \"kk\",\n",
        "    \"नेपाली\": \"ne\",\n",
        "    \"Nederlands\": \"nl\",\n",
        "    \"Norsk\": \"nb\",\n",
        "    \"Polski\": \"pl\",\n",
        "    \"Português (Brasil)\": \"pt-br\",\n",
        "    \"Русский\": \"ru\",\n",
        "    \"Svenska\": \"sv\",\n",
        "    \"украї́нська\": \"uk\",\n",
        "    \"Tiếng Việt\": \"vi-vn-x-central\",\n",
        "    \"简体中文\": \"yue\"\n",
        "}\n",
        "\n",
        "def _get_language(code):\n",
        "    return languages[code]\n",
        "\n",
        "final_language = _get_language(language)\n",
        "#@markdown ### Choose a name for your model:\n",
        "model_name = \"Test\" #@param {type:\"string\"}\n",
        "#@markdown ---\n",
        "# output:\n",
        "#@markdown ### Choose the working folder: (recommended to save to Drive)\n",
        "\n",
        "#@markdown The working folder will be used in preprocessing, but also in training the model.\n",
        "output_path = \"/content/drive/MyDrive/colab/piper\" #@param {type:\"string\"}\n",
        "output_dir = output_path+\"/\"+model_name\n",
        "if not os.path.exists(output_dir):\n",
        "  os.makedirs(output_dir)\n",
        "#@markdown ---\n",
        "#@markdown ### Choose dataset format:\n",
        "dataset_format = \"ljspeech\" #@param [\"ljspeech\", \"mycroft\"]\n",
        "#@markdown ---\n",
        "#@markdown ### Is this a single speaker dataset? Otherwise, uncheck:\n",
        "single_speaker = True #@param {type:\"boolean\"}\n",
        "if single_speaker:\n",
        "  force_sp = \" --single-speaker\"\n",
        "else:\n",
        "  force_sp = \"\"\n",
        "#@markdown ---\n",
        "#@markdown ### Select the sample rate of the dataset:\n",
        "sample_rate = \"16000\" #@param [\"16000\", \"22050\"]\n",
        "#@markdown ---\n",
        "%cd /content/piper/src/python\n",
        "!python -m piper_train.preprocess \\\n",
        "  --language {final_language} \\\n",
        "  --input-dir /content/dataset \\\n",
        "  --output-dir {output_dir} \\\n",
        "  --dataset-format {dataset_format} \\\n",
        "  --sample-rate {sample_rate} \\\n",
        "  {force_sp}"
      ],
      "metadata": {
        "id": "dOyx9Y6JYvRF",
        "cellView": "form"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown # <font color=\"pink\"> **4. Settings.** 🧰\n",
        "import json\n",
        "import ipywidgets as widgets\n",
        "from IPython.display import display\n",
        "\n",
        "#@markdown ### Fine-tune this dataset?\n",
        "\n",
        "#@markdown note: Currently, some models are not working due to piper update, but it is expected to update soon\n",
        "\n",
        "#@markdown * ar-qasr low and high\n",
        "#@markdown * da-nst_talesyntese-medium\n",
        "#@markdown * kk-iseke-low\n",
        "#@markdown * kk-raya-low\n",
        "#@markdown * ne-Google-low\n",
        "#@markdown * no-talesyntese-medium\n",
        "finetune = True #@param {type:\"boolean\"}\n",
        "#@markdown ---\n",
        "if finetune:\n",
        "    ft_command = '--resume_from_checkpoint \"/content/pretrained.ckpt\" '\n",
        "    try:\n",
        "        with open('/content/piper/notebooks/pretrained_models.json') as f:\n",
        "            pretrained_models = json.load(f)\n",
        "        if final_language in pretrained_models:\n",
        "            models = pretrained_models[final_language]\n",
        "            model_options = [(model_name, model_name) for model_name, model_url in models.items()]\n",
        "            model_dropdown = widgets.Dropdown(description = \"Choose pretrained model\", options=model_options)\n",
        "            download_button = widgets.Button(description=\"Download\")\n",
        "            def download_model(btn):\n",
        "                model_name = model_dropdown.value\n",
        "                model_url = pretrained_models[final_language][model_name]\n",
        "                if model_url.startswith(\"1\"):\n",
        "                    !gdown \"{model_url}\" -O \"/content/pretrained.ckpt\"\n",
        "                elif model_url.startswith(\"https://drive.google.com/file/d/\"):\n",
        "                    !gdown \"{model_url}\" -O \"/content/pretrained.ckpt\" --fuzzy\n",
        "                else:\n",
        "                    !wget \"{model_url}\" -O \"/content/pretrained.ckpt\"\n",
        "            download_button.on_click(download_model)\n",
        "            display(model_dropdown, download_button)\n",
        "        else:\n",
        "            raise Exception(f\"There are no pretrained models available for the language {final_language}\")\n",
        "    except FileNotFoundError:\n",
        "        raise Exception(\"The pretrained_models.json file was not found.\")\n",
        "else:\n",
        "    ft_command = \"\"\n",
        "#@markdown ### Choose batch size based on this dataset:\n",
        "batch_size = 16 #@param {type:\"integer\"}\n",
        "#@markdown ---\n",
        "#@markdown ### Validation split:\n",
        "validation_split = 0.01 #@param {type:\"number\"}\n",
        "#@markdown ---\n",
        "#@markdown ### Choose the quality for this model:\n",
        "\n",
        "#@markdown * x-low - 16Khz audio, 5-7M params\n",
        "#@markdown * low - 16Khz audio, 15-20M params\n",
        "#@markdown * medium - 22.05Khz audio, 15-20 params\n",
        "#@markdown * high - 22.05Khz audio, 28-32M params\n",
        "quality = \"x-low\" #@param [\"high\", \"x-low\", \"medium\"]\n",
        "#@markdown ---\n",
        "#@markdown ### For how many epochs to save training checkpoints?\n",
        "#@markdown The larger your dataset, you should set this saving interval to a smaller value, as epochs can progress longer time.\n",
        "checkpoint_epochs = 25 #@param {type:\"integer\"}\n",
        "#@markdown ---\n",
        "#@markdown ### Step interval to generate model samples:\n",
        "log_every_n_steps = 250 #@param {type:\"integer\"}\n",
        "#@markdown ---\n",
        "#@markdown ### Training epochs:\n",
        "max_epochs = 10000 #@param {type:\"integer\"}\n",
        "#@markdown ---"
      ],
      "metadata": {
        "id": "ickQlOCRjkBL",
        "cellView": "form"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown # <font color=\"pink\"> **5. Run the TensorBoard extension.** 📈\n",
        "\n",
        "#@markdown The TensorBoard is used to visualize the results of the model while it is being trained.\n",
        "%load_ext tensorboard\n",
        "%tensorboard --logdir {output_dir}"
      ],
      "metadata": {
        "cellView": "form",
        "id": "MpKDfhAHjHJ3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#@markdown # <font color=\"pink\"> **6. Train.** 🏋️‍♂️\n",
        "\n",
        "#@markdown <font color=\"orange\">**Note: Remember to empty the trash of your Drive from time to time to avoid a lot of space consumption when saving the models.**\n",
        "!python -m piper_train \\\n",
        "    --dataset-dir \"{output_dir}\" \\\n",
        "    --accelerator 'gpu' \\\n",
        "    --devices 1 \\\n",
        "    --batch-size {batch_size} \\\n",
        "    --validation-split {validation_split} \\\n",
        "    --num-test-examples 2 \\\n",
        "    --quality {quality} \\\n",
        "    --checkpoint-epochs {checkpoint_epochs} \\\n",
        "    --check_val_every_n_epoch {checkpoint_epochs} \\\n",
        "    --log_every_n_steps {log_every_n_steps} \\\n",
        "    --max_epochs {max_epochs} \\\n",
        "    {ft_command}\\\n",
        "    --precision 32"
      ],
      "metadata": {
        "id": "X4zbSjXg2J3N",
        "cellView": "form"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Have you finished training and want to test the model?\n",
        "\n",
        "Export your model using the [model exporter notebook](https://colab.research.google.com/github/rmcpantoja/piper/blob/master/notebooks/piper_model_exporter.ipynb)!"
      ],
      "metadata": {
        "id": "6ISG085SYn85"
      }
    }
  ]
}