Dissertation and Thesis

Permanent URI for this communityhttp://164.52.219.250:4000/handle/10263/2146

Browse

Start date: 2020Subject: search.filters.subject.Large Language Models

Search Results

Now showing 1 - 5 of 5
  • Thumbnail Image
    Item
    On the Faithfulness of Language Model-based Data-to-Text Generation
    (Indian Statistical Institute, Kolkata, 2026-11-03) Mahapatra, Joy
    Data-to-Text Generation (D2T)—the task of transforming structured or semi-structured data such as tables, graphs, and meaning representations into readable and faithful natural language—has advanced substantially with the advent of language modeling, particularly with Large Language Models (LLMs). Despite remarkable improvements in language model-based D2T systems in terms of readability, maintaining faithfulness---defined in terms of informativeness and factual consistency---remains a persistent challenge. This thesis presents a systematic exploration of faithfulness in language model-based D2T, focusing on both its evaluation and enhancement, with particular attention to influential factors such as source–reference divergence. The thesis first introduces a structure-aware encoder-decoder framework, based on pre-LLM architectures such as Long Short-Term Memory (LSTM), which encodes input data according to its record-field hierarchy and enhances informativeness through selective gating and hierarchical encoding. It then transitions to LLM-based D2T, conducting a large-scale evaluation across multiple D2T benchmarks and LLM families. The study identifies source-reference divergence---the semantic mismatch between input data and reference text---as a primary factor influencing readability, informativeness, and overall faithfulness. To address this issue, a divergence-weighted training approach is proposed, employing an adaptive loss function that significantly improves model performance. Focusing specifically on factual consistency, the thesis presents a detailed evaluation of widely used LLMs using both automatic and human assessments, with improvements quantified through the Average Rate of Change. The findings demonstrate that source-reference divergence is a major contributor to factual inconsistency. To mitigate this effect, a divergence-aware selective generation framework is introduced, enabling models to abstain from producing factually inconsistent outputs under conditions of high divergence, thereby improving factual consistency without substantial coverage loss. Finally, empirical and mathematical analyses of scaling behavior reveal that factual inconsistency in D2T follows an exponential---rather than a power-law---scaling with respect to model size, compute, and data volume, an observation validated through a statistical framework, \textit{VaCScal}. Collectively, the findings of this research provide both conceptual and empirical insights into faithfulness in language model-based D2T, laying a foundation for the development of future models that are not only readable but also factually faithful.
  • Thumbnail Image
    Item
    Reproducing and Analyzing the “Lost in the Middle” and “The Power of Noise” Phenomenon in Retrieval-Augmented Generation
    (Indian Statistical Institute, 2026-06-16) Samanta, Kousik
    Retrieval-Augmented Generation has become the way to improve Large Language Models. They help with problems like knowledge and hallucinations. Recent studies show that these models still have limitations. One big problem is the “Lost in the Middle” phenomenon. Models can’t access information in the middle of contexts properly. Another counterintuitive observation is the “Power of Noise” paradigm, which suggests adding unrelated documents can actually make the generation better. We know these happen in extractive QA tasks, but we don’t know if they happen in tasks that need complex reasoning. This dissertation looks into how position and noise affect Long-Form Question Answering. We use the ELI5 dataset and test three models. We give them varying amounts of context and see how they do. We also change the location of the correct information and add distracting or random information to observe the effects of these perturbations. Traditional metrics for evaluating model-generated answers aren’t very effective for long-form responses. We introduce two new metrics of evaluation, Prop Score and Sentence Score. Our experiments give us three findings. First, the “Lost in the Middle” issue still happen to a certain degree in Long-Form QA. Second, we confirm that noise can actually improve generation. Third, we hypothesize the reasons of persistence of the “Lost in the Middle” phenomenon and the “power of noise” paradigm in Long-Form QA.
  • No Thumbnail Available
    Item
    Efficient Blending of Large Language Models
    (Indian Statistical Institute, Kolkata, 2025-06) Chatterjee, Sandeep
    Due tothelimitedcapabilitiesofsingleLargeLanguageModels(LLMs),multipleLLMscanbe employedintandemforbetterreliabilityofanswers.Blendingreferstocombiningthestrengths of variousLLMstomakeuseoftheircomplementarycapabilitiesforgeneratinghigh-quality responses.Itisanon-trivialproblem,andthetaskbecomesevenmoredifficultwhenaiming for minimallatencyandsupervisingtheblendingcomponents.Thestandardframework,LLM- Blender, approachesthisinthreestages:responsegeneration,candidateselectionviaranking, and responsefusionthroughsummarization.However,thispipelinefacestwocriticallimita- tions—high latencyduetorepeatedrankingsteps,andheavyrelianceonexternal,supervised componentsincludingalearnedencoderforrankingandaseparatesequence-to-sequencesum- marizer forfusion. In thisthesis,weproposenovel,efficientalternativestoovercomethesechallenges.Thisthesis comprises twoworks.First,weshowthatreducingthefrequencyofrankingwithinmulti- turn conversationssignificantlyimproveslatencywithminimaldegradationinoutputquality. Second, weintroduceapeer-review-basedresponsefusionmechanism,whereLLMscollectively evaluateandreviseeachother’sresponses,removingtheneedforanyexternallytrainedrankers or summarizers.Thiscollaborativemethodenablesfullyself-containedLLMblendingwithout additional trainingorsupervision. WeassessourproposedmethodsonthetaskofConversationalQuestionAnsweringacrossfive multi-turnconversationalbenchmarks—ConvQuestions,Atlas-Converse,CoQA,QuAC,and DoQA—using tendiverse,publiclyavailableopen-weightLLMs.Experimentalresultsdemon- strate thatourpeer-review-drivenframeworkwithreducedrankingachievesqualityonparwith existing approacheswhilebeingsubstantiallymoreefficient.Ourworkpresentsasteptoward scalable, modularLLMensemblingforreal-worldopen-domaindialoguesystems.
  • No Thumbnail Available
    Item
    Explanation and Judgement of IR Ranking using LLM
    (Indian Statistical Institute, Kolkata, 2024-06) Mondal, Santanu
    Pretrained transformer models such as BERT and T5 have significantly advanced the performance of information retrieval (IR) systems when fine-tuned with large-scale labeled datasets. However, their effectiveness diminishes notably in low-resource scenarios where annotated query-passage pairs are limited. This thesis explores an alternative supervision strategy by leveraging natural language explanations to enhance training signals during fine-tuning. We propose a novel methodology that augments traditional relevance labels with textual explanations generated by a large language model (LLM) using few-shot prompting. To achieve this, we generate explanations for 30,000 query-passage-label triples from the MS MARCO dataset using the open-source model google/gemma-2b, allowing for cost-free and scalable inference. These augmented samples are then used to fine-tune a T5-base sequence-to-sequence model, with the objective of producing both the relevance label and an accompanying explanation. During inference, the model predicts the label token, and the probability of that token is used as a soft relevance score, enabling efficient ranking. Empirical results demonstrate that our explanation-augmented retriever outperforms strong baselines, including BM25, a BERT reranker, and a T5 model trained with labels only. We further analyze the effectiveness of explanation order, training data size, and the quality of generated rationales. Our findings suggest that natural language explanations offer a powerful form of supervision, particularly valuable in data-scarce IR settings, and present a compelling direction for improving neural retrievers with minimal annotation overhead.
  • No Thumbnail Available
    Item
    Binary Document Filtering for Retrieval-Augmented Generation
    (Indian Statistical Institute, Kolkata, 2025-06) Saha, Sreyan
    Retrieval-Augmented Generation (RAG) has become a popular technique to enhance Large Language Models (LLMs) with access to external information sources. However, the success of RAG systems critically depends on the relevance and quality of the retrieved documents. In particular, supplying irrelevant or noisy context can lead to degraded downstream generation quality. To address this, our project focuses on improving the document filtering stage in a RAG pipeline through binary relevance classification — deciding whether a retrieved document is suitable to include in the final context window based on its usefulness in directly answering the user query. We explore a wide range of approaches to this task, including rule-based retrieval methods (TF-IDF, BM25), classical machine learning classifiers (logistic regression, SVM), deep neural networks, and LLM-based methods, both in zero-shot and few-shot settings. Our final pipeline leverages instruction-tuned LLMs to act as strict binary classifiers, with a focus on maximizing precision over recall, thereby ensuring that only the most relevant and high-quality documents are passed to the generation module. Experiments are conducted on a Reddit-based query-document dataset tailored to subjective and opinion-heavy queries. Our evaluations suggest that LLMs, even without fine-tuning, can outperform traditional methods in this setting, o”ering a strong foundation for further enhancement through supervised fine-tuning