Learnings from running https://t.co/5clMuIOv4a.

Elmo is your AI Chrome extension designed to create summaries, insights, and extend knowledge for any website. We have been running it for a few weeks, and so far, feedback from friends and family has been universally positive. Here are a few things we've learned from it (mostly regarding LLMs):

(1) Open-source models work great!

We've been using Mixtral 8x7b mostly, with prompt engineering. It achieves good performance at a very affordable cost, allowing us to offer Elmo completely for free. We are currently testing @databricks's DBLX and Baichuan-2 for Chinese. The early results are very positive.

(2) Prompt engineering is too fragile, but oh well.

In fact, today, if we change the prompt asking the model to produce "tl;dr" instead of "summary", this one-word change makes the model behave completely differently. Also, prompts do not stay completely stable across different models (like from Mixtral to DBLX), making transfer a bit difficult. We don't have a good solution yet. Just saying. Can future research solve it? We surely hope so!

(3) Speed matters.

People have told us that they like Elmo being "blazing fast". We did not do a dedicated deployment for Elmo; it's mingled in our serverless API. We made a conscious choice to balance speed and cost while still achieving top performance in existing models, as well as new models like DBLX (as shown by @ArtificialAnlys). In general, we find that people care about speed more than we expected.

(4) Multilingual support is hard.

Models usually behave well within one language, but when asked to produce, e.g., Chinese output for English input, they hallucinate like crazy. Our favorite example was when Mixtral hallucinated a Japanese general called "杨万岛" (Yang Wandao) in Chinese. We ended up realizing it was (Isoroku) Yamamoto, just because the two sound similar. There is a huge opportunity, and also societal value, for multilingual LLMs, especially for Asian and Pacific languages.

(5) Minimalism is key.

People have told us that they like Elmo because it does not do too much. Click, summarize, and done. We don't bundle all kinds of image generation and chat, etc. This gives people peace of mind in using it as a simple tool. Granted, as a platform company, we don't make money out of Elmo, so our pressure is less than that of companies focusing on making browser tools; but the user signal is clear: minimal, simple, clean functionality.

(6) Awareness of AI is still in its early age.

Our friend circle in AI reacts to Elmo with "oh, cool, yeah, yet another tool". Interestingly, our non-tech circle reacts with "oh wow, can this be done? Amazing". Admittedly, for AI people, all this functionality is just normal. But we realize that a much larger part of society is still yet to know the recent advancements in AI, especially in a way they can experience firsthand. This means the potential market for AI is still huge!

Trivia: Elmo started as a hobby project during the Lunar New Year, with just a bit more than a weekend's time. We hope Elmo provides you with an intuitive experience of @LeptonAI, our AI infrastructure company. Maybe one day, you will build AI capabilities into your own product, and we can make it easy for you! And before that, enjoy Elmo as your personal AI assistant, just one click away.