AI Technology & Industry Review
In the new paper ResiDual: Transformer With Dual Residual Connections, a team from Microsoft Research, Microsoft Azure Translation, and Renmin University of China proposes ResiDual, a novel transformer architecture that fuses the connections in post-layer normalization and pre-layer normalization to exploit the benefits of both while also addressing their limitations.
In the new paper Dissecting Recall of Factual Associations in Auto-Regressive Language Models, a team from Google DeepMind, Tel Aviv University and Google Research investigates how factual associations are stored and extracted internally in transformer-based language models and provides insights on how such models’ factual predictions are formed.
In the new paper Towards Realistic Generative 3D Face Models, a research team from Carnegie Mellon University and Meta proposes a 3D controllable generative model capable of generating high-resolution textures and capturing high-frequency details in facial geometry. Their proposed AlbedoGAN outperforms state-of-the-art baselines in facial shape reconstruction.
In the new paper WizardLM: Empowering Large Language Models to Follow Complex Instructions, a research team from Microsoft and Peking University presents Evol-Instruct, a novel approach that leverages LLMs to automatically generate large amounts of instruction data with varying levels of complexity. In human evaluations, the team’s resulting WizardLM model’s generated instructions were judged superior to human-created instruction datasets.
In the new paper FastRLAP: A System for Learning High-Speed Driving via Deep RL and Autonomous Practicing, a UC Berkeley research team proposes FastRLAP (Fast Reinforcement Learning via Autonomous Practicing), a system that autonomously practices in the real world and learns aggressive maneuvers to enable effective high-speed driving.
In the new paper NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers, a Microsoft team introduces NaturalSpeech 2, a TTS system with latent diffusion models for natural and strong zero-shot voice synthesis that captures expressive prosodies with superior robustness.
In the new paper DETRs Beat YOLOs on Real-Time Object Detection, a Baidu Inc. research team presents Real-Time Detection Transformer (RT-DETR), a real-time end-to-end object detector that leverages a hybrid encoder and novel IoU-aware query selection to address inference speed delay issues. RT-DETR outperforms YOLO object detectors in both accuracy and speed.
In the new paper DiffFit: Unlocking Transferability of Large Diffusion Models via Simple Parameter-Efficient Fine-Tuning, a Huawei Noah’s Ark Lab research team introduces DiffFit, a parameter-efficient fine-tuning technique that enables fast adaptation to new domains for diffusion image generation. Compared to full fine-tuning approaches, DiffFit achieves 2x training speed-ups while using only ~0.12 percent of trainable parameters.
In the new paper Meta-Learned Models of Cognition, a team from the Max Planck Institute for Biological Cybernetics (Max-Planck-Gesellschaft, MPG) and DeepMind proposes the establishment of a research program focused on meta-learned models of cognition. The team cites machine learning papers demonstrating how meta-learning can be used to construct Bayes-optimal learning algorithms and suggests it can significantly expand the scope of the rational analysis of cognition.
In the new paper SpectFormer: Frequency and Attention Is What You Need in a Vision Transformer, a research team from Microsoft and the University of Bath proposes Spectformer, a novel transformer architecture that combines spectral and multi-headed attention layers to better capture appropriate feature representations and improve performance.
In the new paper Teaching Large Language Models to Self-Debug, a Google Research and UC Berkeley team presents Self-Debugging, a framework that teaches large language models to debug their own predicted code via few-shot demonstrations and improves baseline accuracy by up to 12 percent.
In the new paper Inference with Reference: Lossless Acceleration of Large Language Models, a Microsoft research team proposes LLMA, an inference-with-reference decoding mechanism that achieves up to 2x lossless speed-ups with identical generation results by exploiting the overlaps between LLM outputs and references.
In the new paper Generative Agents: Interactive Simulacra of Human Behavior, a team from Stanford University and Google Research presents agents that draw on generative models to simulate both individual and emergent group behaviours that are humanlike and based on their changing experiences and environment.
In the new paper Pix2Video: Video Editing Using Image Diffusion, an Adobe Research and University College London team presents Pix2Video, a framework for realistic text-guided video editing using a pretrained image diffusion model.
In the new paper Instruct-NeRF2NeRF: Editing 3D Scenes With Instructions, a UC Berkeley research team presents Instruct-NeRF2NeRF, an approach for editing 3D NeRF scenes through natural language text instructions. The proposed method can edit large-scale, real-world 3D scenes with improved ease of use and realism.
In the new paper TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support, a Google Research team presents TPU v4, the company’s latest supercomputer. TPU v4 is ten times faster than v3 and 1.2–1.7x faster than Nvidia A100 GPUs while using 1.3x–1.9x less power.
In the new paper Towards Healthy AI: Large Language Models Need Therapists Too, a team from Columbia University and IBM Research proposes SafeguardGPT, a framework that incorporates psychotherapy and reinforcement learning to correct the potentially harmful behaviours of AI chatbots.
In the new paper BloombergGPT: A Large Language Model for Finance, a research team from Bloomberg and Johns Hopkins University presents BloombergGPT, a 50 billion parameter language model trained on a 700 billion token dataset that significantly outperforms current benchmark models on financial tasks.
In the new paper TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs, a Microsoft research team proposes TaskMatrix.AI, a novel ecosystem that connects foundation models with millions of existing models and system APIs to build a “super-AI” capable of addressing a wide range of digital and physical tasks.
A recent tech demo called “Living Chang’an City” has been garnering attention. In this video, AI-powered NPCs can be seen roaming the streets of Chang’an City, each possessing unique identities and short-term and long-term goals. They engage in various life-like interactions, such as chatting, shopping and even falling in love.
Colossal-AI open sources a complete RLHF pipeline that includes supervised data collection, supervised fine-tuning, reward model training, and reinforcement learning fine-tuning, based on the LLaMA pre-trained model, and shares ColossalChat, the most practical open-source project that closely resembles the original ChatGPT technical solution!
A Google Research team addresses transformers’ input sequence limitations in the new paper CoLT5: Faster Long-Range Transformers with Conditional Computation, proposing CoLT5 (Conditional LongT5), a family of models that applies a novel conditional computation approach for higher quality and faster long-input processing of up to 64,000 tokens.
In the new paper Sparks of Artificial General Intelligence: Early Experiments with GPT-4, a Microsoft Research team investigates GPT-4, demonstrating its ability to achieve human-level performance on novel and difficult tasks in domains ranging from mathematics and coding to vision, medicine, law and psychology; and proposing it as an early version of an AGI system.
In the new paper GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models, a research team from OpenAI, OpenResearch, and the University of Pennsylvania investigates the potential impact of LLMs like GPT on the US labour market, shedding light on the economic, social, and policy implications.
In the new paper UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation, a Microsoft research team introduces a novel approach that tunes a lightweight and versatile retriever to retrieve prompts for any given task input to improve the zero-shot performance of LLMs.
In the new paper X-Avatar: Expressive Human Avatars, a research team from ETH Zurich and Microsoft presents X-Avatar, an expressive implicit human avatar model designed to capture high fidelity human body and hand poses, facial expressions and other appearance characteristics in a holistic fashion.
In the new paper ViperGPT: Visual Inference via Python Execution for Reasoning, a Columbia University research team presents ViperGPT, a framework for solving complex visual queries by integrating code-generation models into vision via a Python interpreter. The proposed approach requires no further training and achieves state-of-the-art results.
In the new paper MathPrompter: Mathematical Reasoning Using Large Language Models, a Microsoft Research team presents MathPrompter, a novel approach that leverages chain-of-thought (CoT) prompting techniques to improve LLM performance on mathematical reasoning problems and increase confidence in their predictions.
In the new paper Exphormer: Sparse Transformers for Graphs, a team from the University of British Columbia, Google Research and the Alberta Machine Intelligence Institute proposes Exphormer, a class of graph transformers with improved scalability and reduced computational complexity that achieves state-of-the-art performance on graph benchmarks.
In the new paper Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models, a Microsoft Research Asia team presents Visual ChatGPT, a system that incorporates various visual foundation models to enable ChatGPT to understand, generate and edit visual information.
In the new paper Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling, a Microsoft research team presents VALL-E X, a simple yet effective cross-lingual neural codec language model that inherits strong in-context learning capabilities from VALL-E and demonstrates high-quality zero-shot cross-lingual speech synthesis performance.
In the new paper Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages, Google introduces the Universal Speech Model (USM), a scalable self-supervised training framework that extends automatic speech recognition to more than 100 languages.
Diffusion-based AI models continue to wow the world with remarkable image, audio, and video generation capabilities. This performance however comesContinue Reading
In the new paper Language Is Not All You Need: Aligning Perception with Language Models, a Microsoft research team presents KOSMOS-1, a multimodal large language model (MLLM) that can perceive general modalities, learn in context, and follow instructions.
In the new paper SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networks, a research team from the University of California and Kuaishou Technology presents SpikeGPT, the first generative spiking neural network language model. The team’s largest, 260M parameter version achieves DNN-level performance while maintaining the energy efficiency of spike-based computations.
In the new paper Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback, a Microsoft Research and Columbia University team presents LLM-Augmenter, a system that augments black-box large language models with a set of plug-and-play modules to significantly improve the factuality of their responses.
In a new paper Scaling Robot Learning with Semantically Imagined Experience, a team from Robotics at Google and Google Research Robot proposes Learning with Semantically Imagined Experience (ROSIE), a general and semantically-aware data augmentation strategy that leverages text-to-image models to obtain data for robot learning.
In the new paper DocPrompting: Generating Code by Retrieving the Docs, a research team from Carnegie Mellon University and Inspired Cognition presents DocPrompting, a natural-language-to-code generation approach. Tasked with generating code to unseen functions or libraries from a natural language intent, DocPrompting retrieves corresponding code documentation to enable the model to learn to perform the task.
Meta AI reveals the technical details of their LLaMA collection of foundation language models in the new paper LLaMA: Open and Efficient Foundation Language Models. The LLaMA models were trained on trillions of tokens and achieve performance competitive with state-of-the-art models such as GPT-3 and PaLM while being much smaller and using only publicly available training data.
In the new paper RealFusion: 360° Reconstruction of Any Object from a Single Image, an Oxford University research team leverages a diffusion model to generate 360° reconstructions of objects from a single image. Their RealFusion approach achieves state-of-the-art performance on monocular 3D reconstruction benchmarks.