AI Technology & Industry Review
A research team introduces Automated Search for Artificial Life (ASAL). This novel framework leverages vision-language FMs to automate and enhance the discovery process in ALife research.
Researchers from the University of Texas at Austin and NVIDIA proposes upcycling approach, an innovative training recipe enables the development of an 8-Expert Top-2 MoE model using Llama 3-8B with less than 1% of the compute typically required for pre-training.
A DeepMind research team introduces JetFormer, a Transformer designed to directly model raw data. This model maximizes the likelihood of raw data without depending on any pre-trained components, and is capable of both understanding and generating text and images seamlessly.
An NVIDIA research team proposes the normalized Transformer, which consolidates key findings in Transformer research under a unified framework, offering faster learning and reduced training steps—by factors ranging from 4 to 20 depending on sequence length.
A research team at Meta introduces the Large Concept Model (LCM), a novel architecture that processes input at a higher semantic level. This shift allows the LCM to achieve remarkable zero-shot generalization across languages, outperforming existing LLMs of comparable size.
An NVIDIA research team proposes Hymba, a family of small language models that blend transformer attention with state space models, which outperforms the Llama-3.2-3B model with a 1.32% higher average accuracy, while reducing cache size by 11.67× and increasing throughput by 3.49×.
In a new paper Time-Reversal Provides Unsupervised Feedback to LLMs, a research team from Google DeepMind and Indian Institute of Science proposes Time Reversed Language Models (TRLMs), a framework that allows LLMs to reason in reverse—scoring and generating content in a manner opposite to the traditional forward approach.
In a new paper Navigation World Models, a research team from Meta, New York University and Berkeley AI Research proposes a Navigation World Model (NWM), a controllable video generation model that enables agents to simulate potential navigation plans and assess their feasibility before taking action.
An Apple research team introduces AIMV2, a family of vision encoders that is designed to predict both image patches and text tokens within a unified sequence. This combined objective enables the model to excel in a range of tasks, such as image recognition, visual grounding, and multimodal understanding.
In a new paper Music Foundation Model as Generic Booster for Music Downstream Tasks, a Sony research team presents SoniDo, a groundbreaking music foundation model that offers robust framework for improving the effectiveness and accessibility of music processing.
Researchers from Google DeepMind introduce the concept of “Socratic learning.” This refers to a form of recursive self-improvement in artificial intelligence that significantly enhances performance beyond the initial data or knowledge available to the system, as well as a practical framework to implement it.
Apple researchers conducted a systematic study of the computational bottlenecks and cost-efficiency of training SLMs. Their work evaluates training strategies across diverse cloud infrastructure setups, offering practical insights for improving efficiency and reducing costs.
OpenAI researchers introduces TrigFlow, a simplified theoretical framework that identifies the key causes of training instability of consistency models and addresses them with novel improvements in diffusion process parameterization, network architecture, and training objectives.
In a new paper Edify Image: High-Quality Image Generation with Pixel Space Laplacian Diffusion Models, an NVIDIA research team introduces Edify Image—a suite of pixel-based diffusion models that achieve high-resolution image synthesis with exceptional control and precision.
In a new paper Dualformer: Controllable Fast and Slow Thinking by Learning with Randomized Reasoning Traces, a Meta research team presents Dualformer, a single Transformer model that merges both fast and slow reasoning modes within a unified framework.
An NVIDIA research team introduces OMCAT: Omni Context Aware Transformer in their new paper, presenting both OCTAV, a unique dataset aimed at capturing event transitions across audio and video, and OMCAT, a model that employs RoTE (Rotary Time Embeddings).
A Stanford University research team presents Tutor CoPilot, a new model that offers expert-level guidance to tutors in real time. This study is the first of its kind—a randomized controlled trial testing a Human-AI system in live tutoring scenarios.
A research team introduces a novel approach called Induction-head ngram models (Induction-Gram). This technique merges the interpretability and efficiency of n-gram models with insights from neural LLMs to enhance language modeling performance.
A research team from DeepMind and Chicago University presents a novel approach to Reinforcement Learning from Human Feedback. The proposed eva introduces a flexible, scalable framework that leverages any RLHF algorithm to drive more effective alignment with human values
A research team from Google DeepMind and the University of Alberta presents evidence that transformer-based LLMs using autoregressive decoding can indeed support universal computation without any external adjustments or modifications to model weights.
Building on MM1’s success, Apple’s new paper, MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning, introduces an improved model family aimed at enhancing capabilities in text-rich image understanding, visual grounding, and multi-image reasoning.
A research team investigates AI self-evolution. Their work examines how models enhanced with Long-Term Memory (LTM) can adapt and evolve through interaction with their environments, a key step toward achieving more dynamic AI.
In a new paper CAX: Cellular Automata Accelerated in JAX, a research team introduces Cellular Automata Accelerated in JAX, a powerful open-source library designed to enhance CA research, which enables rapid CA simulations through extensive parallelization on various hardware accelerators, including CPUs, GPUs, and TPUs.
In a new paper Don’t Transform the Code, Code the Transforms: Towards Precise Code Rewriting using LLMs, a Meta research team proposes a novel chain-of-thought strategy to efficiently generate code transformations using LLMs. Their approach enables LLMs to derive transformations based on a small set of input/output examples.
A Google DeepMind research team proposes a biologically-inspired dual-system framework for intelligent agents. This “Talker-Reasoner” architecture aligns with Kahneman’s concept, where System 1 is fast and intuitive, while System 2 is slower and deliberative.
In a new paper Upcycling Large Language Models into Mixture of Experts, an NVIDIA research team introduces a new “virtual group” initialization technique to facilitate the transition of dense models into fine-grained MoE structures.
A research team presents a novel language-conditioned robot manipulation framework called Gen2Act, which achieves generalization to unseen tasks using publicly available web data, eliminating the need to collect specific robot data for every task.
In a new paper The Perfect Blend: Redefining RLHF with Mixture of Judges, a research team from Meta GenAI and FAIR developed Constrained Generative Policy Optimization (CGPO), which offers a more structured approach to RLHF, advancing the performance of general-purpose LLMs.
Apple introduces Depth Pro, a state-of-the-art foundation model designed for zero-shot metric monocular depth estimation. This model can generate high-resolution depth maps with exceptional clarity and fine detail, producing a 2.25-megapixel depth map in just 0.3 seconds on a standard GPU.
A joint research team from Meta and the University of Illinois Urbana-Champaign introduces CrossEval, a benchmark designed to assess both individual and cross capabilities. Their findings demonstrate that LLMs often adhere to the “Law of the Weakest Link”—where performance on complex tasks is limited by the weakest capability.
In a new paper Zero-shot Cross-lingual Voice Transfer for TTS, a Google research team presents a new VT module that seamlessly integrates into a multilingual TTS system, enabling voice transfer across languages.
In a new paper FineZip : Pushing the Limits of Large Language Models for Practical Lossless Text Compression, a research team from UC Berkeley and NYU introduces FineZip, a novel LLM-based compression system designed to significantly reduce compression time.
A Microsoft Research Asia research team introduces MarS, a financial market simulation engine powered by a Large Market Model, which addresses the unique demands of modeling the market impact of orders while enabling highly realistic, controllable simulations.
A research team presents SciAgents which aims to automate the process of scientific discovery by revealing hidden interdisciplinary relationships that traditional research methods often overlook. SciAgents operates on a scale, precision, and exploratory power that far surpasses human-driven approaches.
A Sandford U’s research team introduces an experimental framework aimed at evaluating LLMs’ ability to generate research ideas. This study, the first of its kind, compares the ideation capabilities of over 100 expert NLP researchers against an LLM-based ideation system.
A Salesforce AI Research team presents the xLAM series, a collection of large action models designed to enhance the performance of open-source LLMs for autonomous AI agents. This work aims to accelerate innovation in the field and make high-performance models for agent tasks more accessible.
A research team introduces TinyAgent, a framework designed to train and deploy small, task-specific language models capable of performing function calls for agentic systems at the edge, which outperforms larger models such as GPT-4-Turbo in this specific function-calling ability.
A Microsoft research team introduces the Fully Pipelined Distributed Transformer, which leverages the multiple memory hierarchies available in modern GPU clusters, enhancing hardware efficiency and cost-effectiveness while achieving exceptionally high Model FLOPs Utilization (MFU).
In a new paper Diffusion Models Are Real-Time Game Engines, a Google research team presents GameNGen, the first game engine powered entirely by a neural model that enables real-time interaction with complex environments over extended sequences, maintaining high-quality output.
A research team from Samsung makes a first attempt to facilitate LLM deployment on edge devices using integer-only quantization. The proposed MobileQuant, is a post-training quantization technique that reduces both inference latency and energy consumption while preserving accuracy comparable to those achieved with 16-bit activations.