update literature reviewHEADmaster

1 files changed, 18 insertions, 0 deletions

diff --git a/literature-review.md b/literature-review.md
index 30f0b62..be1299e 100644
--- a/literature-review.md
+++ b/literature-review.md
@@ -54,3 +54,21 @@ Speculative decoding traditionally relies on a smaller language model to generat
 
 # Retrieval Augmentation Reduces Hallucination in Conversation
 State-of-the-art dialogue models often generate responses that lack factual accuracy, resulting in hallucination, a problem exacerbated by their reliance on internal knowledge that may not cover all relevant information (Roller et al., 2021; Maynez et al., 2020). Retrieval-Augmented Generation (RAG) addresses this issue by integrating neural retrieval mechanisms with generative models, allowing for the retrieval of relevant documents from a large corpus to enhance the factual accuracy of responses (Lewis et al., 2020b). Studies have shown that models employing retrieval mechanisms achieve state-of-the-art performance on knowledge-grounded conversational tasks, significantly reducing hallucination rates (Shuster et al., 2021). Human evaluations further reveal that retrieval-augmented models demonstrate higher knowledgeability and lower hallucination rates compared to standard models, while also exhibiting improved generalization to unseen topics, thereby outperforming models that rely solely on internal knowledge (Dinan et al., 2019b; Zhou et al., 2021).
+
+# Retrieval Augmented Code Generation and Summarization
+The REDCODER framework, introduced by Parvez et al. (2021), exemplifies a retrieval augmented approach that enhances code generation and summarization by retrieving relevant code or summaries from a database through a two-step process involving a retriever and a generator, leading to significant improvements in performance metrics such as BLEU and Exact Match scores. Karpukhin et al. (2020) developed the Dense Passage Retriever (DPR), which efficiently encodes queries and passages for document retrieval, serving as a foundational model for various retrieval-based approaches in this domain. Additionally, Feng et al. (2020) created CodeBERT, a pre-trained model for understanding programming and natural languages, while Guo et al. (2021) introduced GraphCodeBERT, which incorporates data flow information for enhanced retrieval and generation tasks. Ahmad et al. (2021) presented PLBART, a sequence-to-sequence model pre-trained on extensive code and natural language data, which plays a crucial role in the REDCODER framework. The performance of these models is typically evaluated using metrics such as BLEU Score, Exact Match (EM), and CodeBLEU, which assess the overlap, accuracy, and correctness of the generated outputs.
+
+# Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
+Empirical evaluations show that Retrieval-Augmented Generation (RAG) models achieve state-of-the-art performance on various knowledge-intensive tasks, such as open-domain question answering, abstractive question answering, and fact verification, outperforming traditional extractive models by generating answers that are not necessarily verbatim from retrieved documents (Lewis et al., 2020; Karpukhin et al., 2020). This capability highlights RAG's strength in synthesizing and contextualizing information effectively. Additionally, when compared to closed-book models that rely solely on internal knowledge, RAG models demonstrate a balanced approach that combines the flexibility of generation with the accuracy of factual content, making them a more robust solution for knowledge-intensive tasks (Guu et al., 2020; Petroni et al., 2019).
+
+# Retrieval-Enhanced Machine Learning
+Retrieval-Enhanced Machine Learning (REML) consists of two main components: the prediction model, which generates queries, and the information access models, which retrieve relevant information from a knowledge repository. This framework supports various optimization strategies, including independent, conditional, and joint end-to-end optimization of the prediction and retrieval models (Zamani et al., 2022). REML has several significant applications, such as enhancing model generalization in domain adaptation and few-shot learning, improving scalability by offloading memorization to retrieval systems, facilitating dynamic updates to knowledge bases in non-stationary environments, and increasing interpretability by grounding predictions in retrieved information.
+
+# Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
+Self-RAG introduces several innovative components that set it apart from conventional Retrieval-Augmented Generation (RAG) approaches, including an adaptive retrieval mechanism that allows the model to determine the necessity of retrieving passages based on the input and preceding generations, thereby enhancing efficiency and relevance. The framework utilizes reflection tokens, which facilitate self-reflection by enabling the model to critique its own outputs and assess the relevance of retrieved information, categorized into retrieval and critique tokens to guide decision-making. Additionally, Self-RAG incorporates a structured critique mechanism that evaluates the factuality and overall quality of generated outputs in relation to retrieved information, significantly improving the reliability of responses. Empirical evaluations by Asai et al. (2024) demonstrate that Self-RAG outperforms state-of-the-art LLMs and retrieval-augmented models across various tasks, including open-domain question answering, reasoning, and fact verification, with notable enhancements in factuality and citation accuracy, particularly in long-form generation tasks, thus addressing critical limitations of existing models.
+
+# The Probabilistic Relevance Framework - BM25 and Beyond
+The Probabilistic Relevance Framework (PRF), initially formulated by Robertson and Sparck Jones (1977), established a probabilistic model that highlighted the significance of relevance weighting in search term selection, paving the way for further advancements such as the BM25 function introduced by Robertson et al. (1994), which integrates term frequency and inverse document frequency into its scoring mechanism. BM25 has become a widely recognized instantiation of the PRF, known for its effectiveness in various retrieval tasks by estimating the probability of relevance through a combination of term frequency and document length normalization, with its robustness validated by numerous empirical studies (Robertson & Zaragoza, 2009). The PRF has found applications in diverse information retrieval contexts, including ad-hoc retrieval, query expansion, and information filtering, with Sparck Jones et al. (2000) emphasizing its adaptability in relevance feedback scenarios, thereby demonstrating the framework's flexibility and potential for integrating additional features to enhance its applicability in real-world search systems.
+
+# TIARA Multi-grained Retrieval for Robust Question Answering over Large Knowledge Base
+Knowledge Base Question Answering (KBQA) systems face significant challenges, primarily in KB grounding, which involves linking questions to relevant knowledge within complex and large knowledge bases, and in logical form generation, where ensuring the semantic and syntactic correctness of generated forms is difficult (Gu et al., 2021; Ye et al., 2021). To address these issues, TIARA employs a multi-grained retrieval approach that enhances contextual understanding by retrieving relevant contexts from the knowledge base, including entities, exemplary logical forms, and schema items (Chen et al., 2021). The TIARA framework incorporates key components such as entity retrieval, exemplary logical form retrieval, schema retrieval, and constrained decoding, which collectively improve the accuracy and reliability of logical form generation. Empirical evaluations indicate that TIARA outperforms previous state-of-the-art methods on benchmarks like GrailQA and WebQuestionsSP, demonstrating significant improvements in compositional and zero-shot generalization, thereby showcasing its robustness in managing complex queries (Raffel et al., 2020; Devlin et al., 2019).