https://jwhogg.github.io/tags/guides/Recent content in Guides on jwhoggHugoen-usWed, 12 Jun 2024 00:00:00 +0000https://jwhogg.github.io/articles/word2vec_intro/Wed, 12 Jun 2024 00:00:00 +0000https://jwhogg.github.io/articles/word2vec_intro/<p>In this article we will introduce the context surrounding word2vec, including the motivation for distributed word embeddings, how the Continious Bag-of-Words and Skip-gram algorithms work, and the advancements since the original paper was released. We will also go into the training of the neural network, so it is assumed you have some knowledge on this.</p> <span class="text-gray-500 text-sm"> These 2 papers introduced word2vec to the world back in 2013: </span> <table> <thead> <tr> <th><figure> <img src="https://jwhogg.github.io/images/word2vec_paper_1.webp#smaller" alt="paper1" loading="lazy" /> <figcaption>paper1</figcaption> </figure> </th> <th><figure> <img src="https://jwhogg.github.io/images/word2vec_paper_2.webp#smaller" alt="paper2" loading="lazy" /> <figcaption>paper2</figcaption> </figure> </th> </tr> </thead> <tbody> <tr> <td><span class="text-gray-500 text-sm"> [Word2Vec Paper 1](https://arxiv.org/pdf/1301.3781)- introducing CBOW and Skip-Gram </span> </td> <td><span class="text-gray-500 text-sm"> [Word2Vec Paper 2](https://arxiv.org/pdf/1310.4546)- Performance Improvements </span> </td> </tr> </tbody> </table> <h3 id="motivation"> Motivation </h3> <p>For many NLP tasks, we need to learn on data which can&rsquo;t be easily represented numerically. For example, let&rsquo;s look at the popular <a href="https://huggingface.co/datasets/stanfordnlp/imdb/viewer/plain_text/train">IMDB dataset</a>, which gives reviews in one column, and a binary sentiment label in the next:</p>