AI Cloud & ML Ops Company Global News Industrial AI Industry United States US & Canada

Apple Reveals Design of Its On-Device ML System for Federated Evaluation and Tuning

Apple has laid out the design characteristics of a new generic system that enables federated evaluation and tuning (FE&T) systems on end-user devices.

Apple has revealed system design details for its federated task processing system, demonstrating the potential of federated evaluation and tuning for on-device ML system personalization.

Federated Learning (FL) enables model training on a large corpus of decentralized data, a de-identification approach that addresses public concerns and other issues regarding data privacy, ownership and locality. FL is seeing significant interest from both research and application perspectives, and its potential deployment on end devices such as smartphones and computers — where it can safeguard privacy and improve user experience — is highly desirable.

Now, a research team from Apple has created a generic system that enables “federated evaluation and tuning (FE&T)” systems on such end-user devices.

image.png

The objective of most current FL research is to learn a global neural network, while few FL studies have looked into the idea of model personalization in federated settings. The new Apple paper explores the application of FL systems for on-device personalization in an initial use case, automatic speech recognition (ASR). Ingesting data that is only available on-device, the system requires the evaluation and tuning of a personalization algorithm’s global parameters to create device-specific ASR language models.

The researchers studied how to improve user-specific ML system accuracy in the context of such ASR systems, especially when the personalization data remains inaccessible to the server-side.

image.png
System design overview

The proposed system’s on-device components include a data store, task scheduler and results manager. The data store provides common on-device data retention policies to ensure that the amount of application-specific data stored is limited and recent. The task scheduler periodically downloads a list of available task descriptors when the system-level preconditions for device participation are met. The results manager sends back the task results and populates an on-device database that enables end-users to inspect data that is shared with the server, and also collects and sends health-related telemetry data to the server system.

The server components include a task manager, data manager and developer interface. The task manager stores and delivers all tasks and their attachments to the content delivery network, and can also retire tasks. The data manager is responsible for dropping any sensitive data from HTTP requests and forwarding results and telemetry information to a central database. The developer interface includes a web UI for monitoring task status and inspecting telemetry data and a Python script library.

To demonstrate federated tuning’s applicability to on-device personalization, the researchers described two specific large-scale personalization use cases: news personalization and ASR personalization.

image.png
Federated tuning for news personalization
image.png
Live A/B experimentation results

The optimized parameters from the FT’s first run resulted in a 1.98 percent increase in daily article views, while FT’s second run saw a 1.87 percent increase in daily article views and a 0.90 percent increase in the daily time spent on the application.

image.png
Word error rates on utterances affected by system combination, where Server ASR refers to on-device ASR without any personalization and +Client ASR refers to a personalized client ASR.

The results show that for both user and generic cases, word error rates are lower after ASR personalization, validating the applicability of federated tuning to on-device personalization.

The paper Federated Evaluation and Tuning for On-Device Personalization: System Design & Applications is on arXiv.


Author: Hecate He | Editor: Michael Sarazen


We know you don’t want to miss any news or research breakthroughs. Subscribe to our popular newsletter Synced Global AI Weekly to get weekly AI updates.

2 comments on “Apple Reveals Design of Its On-Device ML System for Federated Evaluation and Tuning

  1. Pingback: [N] Apple Reveals Design of Its On-Device ML System for Federated Evaluation and Tuning – ONEO AI

  2. Pingback: Apple Reveals Design Of Its On-Device ML System For Federated Evaluation And Tuning - AI Summary

Leave a Reply

Your email address will not be published.

%d bloggers like this: