1. FAILURE POINTS
Overview of RAG system
- FP1: Missing content. The first fail case is when asking a question that cannot be answered from the available documents.
- FP2: Missed the Top Ranked Documents. The answer to the question is in the document but did not rank highly enough to be returned to the user.
- FP3: Not in Context - Consolidation strategy Limitations. Documents with the answer were retrieved from the database but did not make it into the context for generating an answer.
- FP4: Not Extracted. The answer is present in the context, but the large language model failed to extract the correct answer.
- FP5: Wrong Format. The question involved extracting information in a certain format such as a table or list and the large language model ignored the instruction.
- FP6: Incorrect Specificity. The answer is returned in the response but is not specific enough or is too specific to address the user’s need.
- FP7: Incomplete. Incomplete answers are not incorrect but miss some of the information even though that information was in the context and available for extraction.
2. CASE STUDY
The three case studies
3. LESSONS LEARNED
The lessons learned from the three case studies
