{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"ex7 - vae.ipynb","provenance":[],"collapsed_sections":[],"toc_visible":true,"authorship_tag":"ABX9TyNS2VkU+59d9c7FppBT4a6T"},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"}},"cells":[{"cell_type":"markdown","source":["## Variational autoencoder (VAE)\n","\n","Variational autoencoders are a slightly more modern and interesting take on autoencoding.\n","\n","
\n"," \n","
\n","\n","What is a variational autoencoder, you ask? It's a type of autoencoder with added constraints on the encoded representations being learned. More precisely, it is an autoencoder that learns a [latent variable model](https://en.wikipedia.org/wiki/Latent_variable_model) for its input data. So instead of letting your neural network learn an arbitrary function, you are learning the parameters of a probability distribution modeling your data. If you sample points from this distribution, you can generate new input data samples: a VAE is a \"generative model\".\n","\n","How does a variational autoencoder work?\n","\n","First, an encoder network turns the input samples x into two parameters in a latent space, which we will note z_mean and z_log_sigma. Then, we randomly sample similar points z from the latent normal distribution that is assumed to generate the data, via z = z_mean + exp(z_log_sigma) * epsilon, where epsilon is a random normal tensor. Finally, a decoder network maps these latent space points back to the original input data.\n","\n","The parameters of the model are trained via two loss functions: a reconstruction loss forcing the decoded samples to match the initial inputs (just like in our previous autoencoders), and the KL divergence between the learned latent distribution and the prior distribution, acting as a regularization term. You could actually get rid of this latter term entirely, although it does help in learning well-formed latent spaces and reducing overfitting to the training data.\n","\n","Because a VAE is a more complex example, we have made the code available on [Github](https://github.com/fchollet/keras/blob/master/examples/variational_autoencoder.py) as a standalone script. Here we will review step by step how the model is created.\n","\n","### Loss\n","\n","The loss function of the VAE is defined by two terms, the reconstruction loss and the regularizer which is essentially a KL divergence between the encoder’s distribution and the latent space.\n","\n","$$\\mathcal{L}(\\theta, \\phi) = -\\mathbb{E}_{X \\sim q_{\\theta}}[P_{\\theta}(x|z)] + \\mathcal{D}_\\text{KL}(q_{\\phi}(z|x) || p_{\\theta}(z)) $$\n","\n","where the KL divergence is defined below\n","\n","$$\\mathcal{D}_\\text{KL}(P||Q) = \\sum_x P(x) \\log(\\frac{P(x)}{Q(x)})$$\n","\n","\n","### KL divergence\n","\n","What does KL divergence mean?\n","\n","
\n"," \n","
\n","\n","Solution for $\\mathcal{D}(P||Q)$ is below\n","\n","
\n"," \n","
\n","\n","and for $\\mathcal{D}(Q||P)$ is below\n","\n","
\n","
\n","
