Ai2 has announced the release of its OLMoE iOS app, a fully open-source application enabling users to run state-of-the-art language models directly on their devices. This marks a significant milestone in the development of on-device AI, offering users private and secure access to advanced language model capabilities without requiring an internet connection. The app is available for download on the Apple App Store or can be built from source via Ai2's open repository.
The app is designed to operate on newer Apple devices, including iPhone 15 Pro and later models, as well as M-series iPads. These hardware requirements stem from the need for 8 GB of memory to support the computational demands of the OLMoE model, which has been optimized for efficiency using Q4_K_M quantization. Despite this compression, the model maintains high performance, with minimal degradation in evaluation scores.
OLMoE app
OLMoE represents a continuation of Ai2's commitment to openness in AI development. The model leverages advancements from previous iterations, such as the Dolmino mid-training mix and Tülu 3 post-training recipe, achieving a 35% improvement in evaluation metrics compared to earlier versions. The app integrates these innovations into an accessible platform for researchers and developers to test real-world applications, improve local AI models, and experiment with open-source code.
Allen AI
The application was developed in collaboration with GenUI and utilizes Swift bindings for Llama.cpp to optimize performance. On an iPhone 16 Pro, it achieves an average processing speed of 41 tokens per second. Additionally, quantized versions of the OLMoE model are available in GGUF format on HuggingFace for broader experimentation.
Ai2 envisions this release as a foundational step toward widespread adoption of on-device AI. By eliminating reliance on cloud infrastructure, the OLMoE app ensures data privacy and reliability while demonstrating the growing potential of mobile devices as platforms for advanced AI capabilities. Researchers and developers are encouraged to explore the app's open-source framework to contribute to future advancements in efficient local AI solutions.
