AI Technology & Industry Review
In a recent technical report, OpenAI introduces Sora, a groundbreaking text-to-video model. Sora stands out for its ability to generate videos and images spanning a wide range of durations, aspect ratios, and resolutions, producing up to a minute of high-definition video content.
In the new paper SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks, a research team from the University of California and Kuaishou Technology presents SpikeGPT, the first generative spiking neural network language model. The team’s largest, 260M parameter version achieves DNN-level performance while maintaining the energy efficiency of spike-based computations.
In the new paper Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback, a Microsoft Research and Columbia University team presents LLM-Augmenter, a system that augments black-box large language models with a set of plug-and-play modules to significantly improve the factuality of their responses.
In a new paper Scaling Robot Learning with Semantically Imagined Experience, a team from Robotics at Google and Google Research Robot proposes Learning with Semantically Imagined Experience (ROSIE), a general and semantically-aware data augmentation strategy that leverages text-to-image models to obtain data for robot learning.
In the new paper DocPrompting: Generating Code by Retrieving the Docs, a research team from Carnegie Mellon University and Inspired Cognition presents DocPrompting, a natural-language-to-code generation approach. Tasked with generating code to unseen functions or libraries from a natural language intent, DocPrompting retrieves corresponding code documentation to enable the model to learn to perform the task.
Meta AI reveals the technical details of their LLaMA collection of foundation language models in the new paper LLaMA: Open and Efficient Foundation Language Models. The LLaMA models were trained on trillions of tokens and achieve performance competitive with state-of-the-art models such as GPT-3 and PaLM while being much smaller and using only publicly available training data.
In the new paper RealFusion: 360° Reconstruction of Any Object from a Single Image, an Oxford University research team leverages a diffusion model to generate 360° reconstructions of objects from a single image. Their RealFusion approach achieves state-of-the-art performance on monocular 3D reconstruction benchmarks.
Colossal-AI, as one of the hottest open-source solutions for large AI models, presents an open-source complete PyTorch-based ChatGPT equivalent implementation process that achieves 7.73 times faster compared to the original PyTorch approach with only 1.6GB GPU memory.
In the new paper Symbolic Discovery of Optimization Algorithms, a research team from Google and UCLA presents a method for formulating algorithm discovery as program search and applies it to find EvoLved Sign Momentum (Lion), a simple and effective optimization algorithm that surpasses state-of-the-art methods while reducing computation costs.
In the new paper ResMem: Learn What You Can and Memorize the Rest, a Stanford University research team proposes the residual-memorization (ResMem) algorithm, a novel approach to improving the generalization ability of neural network models by performing explicit memorization via a separate k-nearest neighbour component.
In the new paper Toolformer: Language Models Can Teach Themselves to Use Tools, a team from Meta AI Research and the Universitat Pompeu Fabra proposes Toolformer, a model that self-learns how to choose and use external tools such as search engines, calculators, and translation systems to boost performance on downstream tasks.
In the new paper Zero-Shot Image-to-Image Translation, a team from Carnegie Mellon University and Adobe Research introduces pix2pix-zero, a diffusion-based image-to-image translation method that performs structure-preserving image editing without requiring manual prompting or additional training.
A Hugging Face team collaborates with researcher Simo Ryu to provide a general approach that enables users to implement Low-Rank Adaptation (LoRA) in diffusers via both Dreambooth and full fine-tuning methods.
In the new paper Accelerating Large Language Model Decoding with Speculative Sampling, a DeepMind research team presents SpS (Speculative Sampling), an algorithm that achieves 2–2.5x decoding speedups on a 70 billion parameter Chinchilla language model. The novel approach maintains sample quality and does not require any modifications to model parameters or architecture.
In the new paper PyGlove: Efficiently Exchanging ML Ideas as Code, a Google Brain research team extends their PyGlove Python Library to leverage symbolic rule-based patches and simplify the scalable exchange of ML ideas as code.
In the new paper Dreamix: Video Diffusion Models Are General Video Editors, a team from Google Research and the Hebrew University of Jerusalem presents Dreamix, a novel approach that leverages a video diffusion model (VDM) to enable text-based motion and appearance video editing.
In the new paper Mnemosyne: Learning to Train Transformers with Transformers, a research team from Google and Columbia University presents Mnemosyne Optimizer, a learning-to-learn system for training entire neural network architectures without any task-specific optimizer tuning.
In the new paper Mathematical Capabilities of ChatGPT, an international research team tests ChatGPT’s mathematical capabilities and evaluates its suitability as an assistant to professional mathematicians. The team concludes that despite the glowing reviews in mainstream media, ChatGPT’s mathematical abilities “are significantly below those of an average mathematics graduate student.”
In the new paper DetectGPT: Zero-Shot Machine-Generated Text Detection Using Probability Curvature, a Stanford University research team presents DetectGPT, a zero-shot machine-generated text detection algorithm that uses probability curvature to predict whether a candidate passage was generated by a large language model.
In the new paper MusicLM: Generating Music From Text, a Google Research and Sorbonne University team presents MusicLM, a model for generating high-fidelity music that can be conditioned on both text and melody. MusicLM surpasses baselines in both its audio quality and adherence to the text descriptions.
In the new paper ClimaX: A Foundation Model for Weather and Climate, a team from Microsoft Autonomous Systems and Robotics Research, Microsoft Research AI4Science and the University of California at Los Angeles presents ClimaX, a foundation model for weather and climate that can be efficiently adapted for general-purpose tasks related to the Earth’s atmosphere.
A Stanford University research team presents a brain-computer interface for translating speech-related neural activity into text (speech BCI) in the new paper A High-performance Speech Neuroprosthesis. Theirs is the first speech BCI to record impulse activity from intracortical microelectrode arrays and could benefit people unable to produce clear utterances due to diseases such as stroke and ALS.
In the new paper Asynchronous Deep Double Duelling Q-Learning for Trading-Signal Execution in Limit Order Book Markets, an Oxford University research team introduces Deep Duelling Double Q-learning with the APEX architecture to train a trading agent to translate predictive signals into optimal limit order trading strategies.
In the new paper Continual Few-Shot Learning Using HyperTransformers, a Google Research team proposes Continual HyperTransformer, which modifies the recently published HyperTransformer few-shot learning method to sequentially update a convolutional neural network’s weights based on the information in a new task without forgetting the knowledge it learned from previous tasks.
In the new paper Tracr: Compiled Transformers as a Laboratory for Interpretability, a research team from ETH Zurich and DeepMind presents Tracr, a compiler that addresses the absence of ground truth explanations in deep neural network models by “compiling” human readable code to the weights of a transformer model.
In the new paper Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling, a research team from Peking University, ByteDance, and the University of Oxford presents Sparse Masked Modelling with Hierarchy (SparK), the first BERT-style pretraining approach that can be used on convolutional models without any backbone modifications.
In the new paper Memory Augmented Large Language Models are Computationally Universal, Google Brain and University of Alberta researcher Dale Schuurmans establishes computational universality for a large language model augmented with an associative read-write memory.
In the new paper DensePose From WiFi, a Carnegie Mellon University research team proposes WiFi-based DensePose, a neural network architecture capable of estimating human dense pose using only WiFi signals in scenarios with occlusion and multiple people.
In the new paper Unlocking de Novo Antibody Design With Generative Artificial Intelligence, researchers from Absci Corporation leverage the power of generative artificial intelligence for de novo antibody design in a zero-shot and controllable manner, dramatically reducing time and resource requirements for the task.
In the new paper Optimistic Meta-Gradients, a DeepMind research team explores the connection between gradient-based meta-learning and convex optimization, demonstrating that optimism in meta-learning is achievable via the Bootstrapped Meta-Gradients approach.
In the new paper Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers, a Microsoft research team presents VALL-E, the first language model-based text-to-speech (TTS) system with strong in-context learning. VALL-E achieves state-of-the-art personalized speech synthesis quality via prompting in a zero-shot setting.
Baidu, Inc. today hosted its annual flagship developer conference Baidu Create 2022. In the meeting, Baidu offered an in-depth exploration of Baidu’s research and analysis of future technology trends, covering a range of emerging technologies including artificial intelligence, autonomous driving, intelligent search, quantum computing and AI scientific computing.
In the new paper Muse: Text-To-Image Generation via Masked Generative Transformers, a Google Research team introduces Muse, a transformer-based text-to-image synthesis model that leverages masked image modelling to achieve state-of-the-art performance while being significantly faster than diffusion or autoregressive models.
Baidu research releases top 10 tech trends for 2023, e.g. big model building, digital-real convergence, virtual-real symbiosis, autonomous Driving, etc.
Colossal-AI (https://github.com/hpcaitech/ColossalAI), the widely-used open-source library for training, inference and fine-tuning of large deep learning models, has released a new automatic parallelism feature and functionality that reduces hardware costs by up to 46 times for AI-Generate Content (AIGC) solutions.
In the new paper Hungry Hungry Hippos: Towards Language Modeling with State Space Models, Stanford University and State University of New York at Buffalo researchers explore the expressivity gap between state space models and transformer language model attention mechanisms and propose FlashConv to improve state space model training efficiency on modern hardware.
In the new paper OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization, a Meta AI research team presents OPT-IML Bench, an Instruction Meta Learning benchmark comprising 2000 NLP tasks and an evaluation framework for model generalization.
In the new paper GraphCast: Learning Skillful Medium-Range Global Weather Forecasting, a research team from DeepMind and Google presents GraphCast, a machine-learning (ML)-based weather simulator that scales well with data and can generate a 10-day forecast in under 60 seconds. GraphCast outperforms the world’s most accurate deterministic operational medium-range weather forecasting system and betters existing ML-based benchmarks.
In the new paper Point-E: A System for Generating 3D Point Clouds from Complex Prompts, An OpenAI research team presents Point·E, a system for text-conditional synthesis of 3D point clouds that leverages diffusion models to generate diverse and complex 3D shapes conditioned on complex text prompts in minutes on a single GPU.
In the new paper Mu2SLAM: Multitask, Multilingual Speech and Language Models, a Google Research team presents Mu2SLAM, a multilingual sequence-to-sequence pretraining method for speech and text models that covers arbitrary tasks in over 100 languages.
In the new paper FlexiViT: One Model for All Patch Sizes, a Google Research team presents FlexiViT, a flexible ViT that performs well across a wide range of patch sizes, matching or outperforming standard fixed-patch ViT performance with no extra costs.