Large Language Models (LLMs) have captured the attention of the tech world with their remarkable common-sense reasoning and generalizability. However, their large size and server transfer requirements can make them resource-intensive and slow, which is problematic for use in mobile or wearable devices like smart glasses and smart watches. Moreover, on-device computing could offer a solution to privacy concerns by keeping sensitive data, such as text messages or photos, on the device itself. To tackle these challenges, we’ve developed a more compact language model, ranging from 0.5B to 1.4B parameters. This model is designed to run on-device, providing a competitive performance for conversational grounded tasks, while also managing latency and memory usage effectively. 

In this presentation, I’ll delve into our work on creating a versatile, on-device LLM, which is a distilled version of the LLAMA model, specifically tailored for conversational reasoning tasks. I’ll discuss our pretraining framework and our innovative approach to finetuning, which involves using LLM synthesized, task-specific data in a fresh dialogue format. Finally, I’ll share our strategy for scaling our text-based conversational model to a multimodal model, enhancing generative experiences such as composing text replies, document summarization, image captioning, and visual question-answering on wearables or mobile devices.