A foundation model is a type of artificial intelligence neural network trained on vast amounts of raw data, typically through unsupervised learning, and designed to be adaptable for a wide range of tasks.
In a new paper Apple Intelligence Foundation Language Models, an Apple research team introduces the foundation language models developed to power Apple Intelligence features. These models include a ∼3 billion parameter model optimized for efficient on-device performance and a larger server-based model designed for Private Cloud Compute.
At the 2024 Worldwide Developers Conference, Apple unveiled Apple Intelligence, a personal intelligence system seamlessly integrated into iOS 18, iPadOS 18, and macOS Sequoia. Apple Intelligence consists of highly capable generative models that are fast, efficient, and tailored to users’ everyday needs, adapting in real-time to their current activities. interactions across different apps.
These foundation models that built in Apple Intelligence have been fine-tuned for various user experiences, such as writing and refining text, prioritizing and summarizing notifications, creating playful images for conversations, and automating in-app actions to streamline
The report details how two key models—AFM-on-device, a ∼3 billion parameter language model, and AFM-server, a larger server-based language model—have been designed and optimized to perform specialized tasks with efficiency, accuracy, and a focus on user privacy.
The AFM base models are dense decoder-only models based on the Transformer architecture, incorporating several key design choices:
- A shared input/output embedding matrix to reduce memory usage.
- Pre-Normalization using RMSNorm for improved training stability.
- Query/key normalization to enhance training stability.
- Grouped-query attention (GQA) with eight key-value heads to minimize the KV-cache memory footprint.
- The SwiGLU activation function for increased efficiency.
- RoPE positional embeddings with a base frequency set to 500k to support long-context processing.
This report provides an overview of the model architecture, the training data, the training process, the optimization techniques for inference, and the evaluation results. The team also emphasizes their commitment to Responsible AI, detailing how ethical principles were integrated throughout the development of these models.
The paper Apple Intelligence Foundation Language Models is on arXiv.
Author: Hecate He | Editor: Chain Zhang
Pingback: Apple Intelligence: Unveiling Foundation Models Powering the Future of iOS, iPadOS, and macOS - Apple News
I recently attended a family reunion that was catered by this fantastic kosher burger place, and it was the highlight of the day. The food was incredible—freshly smashed and grilled burgers that were juicy, flavorful, and absolutely delicious. What made it even better was that everything was kosher, so everyone in the family could enjoy it without any worries. The catering team was amazing, handling everything with professionalism and ensuring that everyone was served quickly. It was so much more than just a meal; it was an experience that brought the whole family together. The burgers were a hit with everyone, from the kids to the grandparents, and it really made the day special. If you’re planning a family gathering or any event, and you want to offer something that everyone will love, you need to check out their kosher event catering. It’s the perfect way to make your event unforgettable.