Edge AI: ML inference in the browser

Recently, I stumbled across a guy who ported OpenAI’s Whisper model into c++, in various sizes, allowing the model to be run on-device, at impressive speed. I went down a rabbit-hole, and found a whole family of popular models that had been ported to work on-device, from the browser: Stable difusion running in browser Text emotion prediction in browser Llama c++ Web LLM YOLO in the browser Edge Computing This movement is part of the larger Edge computing trend, which focuses on bringing computing as close to the source as possible, to reduce latency.

Word2Vec Overview

In this article we will introduce the context surrounding word2vec, including the motivation for distributed word embeddings, how the Continious Bag-of-Words and Skip-gram algorithms work, and the advancements since the original paper was released. We will also go into the training of the neural network, so it is assumed you have some knowledge on this. These 2 papers introduced word2vec to the world back in 2013: Word2Vec Paper 1- introducing CBOW and Skip-Gram Word2Vec Paper 2- Performance Improvements Motivation For many NLP tasks, we need to learn on data which can’t be easily represented numerically.

Implementing Word2Vec in python

We will be implementing the Neural Network for the Continuous Bag of Words (CBOW) from the word2vec paper. This article assumes you have a good understanding of the high-level of word2vec. This will be covered in coming articles also. Our goal is to train with sample pairs $(y,X)$, where $y$ is the target word, and $X$ is one of the context words from within the window. The error function we want to minimise:

Fine-tuning a pre-trained model from HuggingFace

We will be using a 🤗 HuggingFace model (GPT-2 Medium) 📙Jupyter Notebook Link Create train/test split for custom dataset (can use sklearn for this) Get the model tokeniser from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("your-model-here") #eg "bert-base-cased" Create encodings for train/test using tokeniser def tokenize_function(examples): return tokenizer(examples["text"], padding="max_length", truncation=True) tokenized_datasets = raw_datasets.map(tokenize_function, batched=True) # where raw_datasets is a dict with train/dev/test we need padding as the inputs must fit the models input even if they are too short Create small datasets for development small_train_dataset = tokenized_datasets["train"].

Notes on: Getting an ML job

Summary of this article / video Intro can split getting an ML job into 2 steps: Getting ML skills building projects contributing to OS reading technical info Marketing ML skills communication interviewing portal creation passing application screening Strategies: Make ML jobs come to you by Learning in Public (great article!) Summary: write blogs / tutorials / cheatsheets answer things on stackoverflow / reddit make videos draw cartoons Pull Request libraries you use OS: github repo -> issues -> ‘good first issue’ (easy ones to solve) make your own libraries intended only for you goto conferences ==> make the thing you wish you had found Info