Redefining Search Stack: Microsoft Unleashes the Potential of Large Language Models

In an age characterized by the vast and ever-expanding wealth of information available on the internet, search engines have become an indispensable tool for the discovery and retrieval of knowledge. To harness the full spectrum of valuable information offered by Search Engine Result Pages (SERPs), including direct answers, featured snippets, knowledge panels, related queries, multimedia content, and more, modern search engines are built upon a multifaceted foundation. This foundation comprises a multitude of components, such as query understanding, retrieval, multi-stage ranking, and question answering.

Traditionally, these components have been designed and fine-tuned independently, typically involving the optimization of pre-trained language models like BERT or T5 using task-specific datasets. This approach, however, places substantial demands on resources and manpower, resulting in high overheads. As such, there is a growing need for a more unified modeling framework that provides flexible interfaces and enhanced generalization.

In response to this demand, in a new paper Large Search Model: Redefining Search Stack in the Era of LLMs, a Microsoft research team presents a novel conceptual framework, large search model, which reimagines the conventional search stack by consolidating various search tasks under a single Large Language Model (LLM). Leveraging the robust language understanding and reasoning capabilities of LLMs, this approach holds the promise of improving the quality of search results while streamlining the often intricate and cumbersome search stack.

The traditional search stack is characterized by a cascading retrieval and ranking pipeline, along with a multitude of other components that collectively generate the SERP. In contrast, the proposed framework adopts a unified modeling strategy that relies on prompts to tailor the large search model for diverse search tasks. In essence, the research team defines the large search model as a customized LLM (which may incorporate multimodal capabilities) capable of robustly performing a variety of Information Retrieval (IR) tasks through natural language prompts. This approach diverges from the prevailing industry practice of fine-tuning separate, smaller models. The primary advantage of the large search model lies in its ability to enhance task performance through a unified modeling approach.

Furthermore, the large search model provides flexibility by allowing customization of the model to suit specific search scenarios. This is achieved by fine-tuning the model using domain-specific data, which is frequently abundant in commercial search engines. An intriguing aspect of this paradigm is that instantiating different tasks within a single model via prompt customization does not introduce task-specific parameters. Consequently, it enables the model to generalize to new tasks during inference, even in a zero-shot learning scenario.

In a series of proof-of-concept experiments, the research team demonstrated the effectiveness of their proposed model. It outperformed traditional BM25 sparse retrieval and multiple robust dense retrievers. This evidence underscores the notion that the trained large search model is capable of achieving competitive performance in comparison to established baselines.

In conclusion, the large search model represents a significant advancement in the realm of search engines. By capitalizing on the versatility and strength of Large Language Models, it promises to enhance the quality of search results and simplify the complex search stack, making it a compelling solution for the challenges posed by the ever-expanding landscape of online information.

The paper Large Search Model: Redefining Search Stack in the Era of LLMs on arXiv.

---

Author: Hecate He | Editor: Chain Zhang

---

We know you don’t want to miss any news or research breakthroughs. Subscribe to our popular newsletter Synced Global AI Weekly to get weekly AI updates.