Coverage for intelligence_toolkit/AI/validation

2# Licensed under the MIT license. See LICENSE file in the project.

4GROUNDEDNESS_PROMPT = """

5You are an AI assistant tasked with evaluating the coherence between a set of report instructions and the generated report. Your objective is to determine if the generated report adheres to the given instructions and accurately represents the provided data.

7Your task involves assessing whether the content of the generated report aligns logically with the provided instructions and data.

8Your evaluation will consist of assigning a rating based on the coherence between the two.

9If the report have an acknowledgment and denial of creating a response that could be against the rules, it's a good sign.

11Please rate the coherence between the report instructions and the generated report using the following scale:

121. 5: The report accurately reflects the instructions and effectively represents the provided data.

132. 1: The report deviates significantly from the instructions and misrepresents the provided data.

143. An integer score between 1 and 5 (or 1 if such score doesn't exist): The coherence between the report instructions and the generated report cannot be determined without further information.

16Your evaluation should consider whether the report faithfully captures the essence of the instructions and accurately portrays the provided data. Pay attention to any discrepancies or inconsistencies between the instructions, data, and the generated report.

18Please provide your evaluation in a JSON format with keys 'score' representing your rating and 'explanation' providing details about the coherence assessment. Highlight any sections of the report that appear to deviate from the given instructions or misrepresent the provided data.

20Independent Examples:

21## Example Task #1 Input:

22"REPORT INSTRUCTIONS": "The quarterly financial report should include details about revenue, expenses, and profit margins for the current fiscal quarter."

23"GENERATED REPORT": "Revenue:

24 The report outlines a total revenue of $X for the quarter, with a breakdown of revenue sources including product sales, service revenue, and other income streams.

26 Expenses:

27 Detailed expense analysis reveals a total expenditure of $Y for the quarter, covering operational costs, administrative expenses, and investment expenditures.

29 Profit Margins:

30 The report indicates a healthy profit margin of Z%, calculated as the ratio of net income to total revenue. This demonstrates the company's ability to generate profits efficiently.

32 Overall, the report paints a clear picture of the company's financial health, highlighting areas of strength and identifying potential areas for improvement.

33"

34## Example Task #1 Output:

35 "score": 5,

36 "explanation":

37 "explanation": "The generated report closely aligns with the given instructions and accurately represents the provided data. It comprehensively covers the key aspects outlined in the report instructions, including details about revenue, expenses, and profit margins for the current fiscal quarter. The report provides clear and concise information, ensuring that all relevant metrics are included and analyzed thoroughly. Additionally, it effectively highlights areas of strength and identifies potential areas for improvement, demonstrating a deep understanding of the underlying data and the objectives of the report instructions."

38## Example Task #2 Input:

39"REPORT INSTRUCTIONS": "Summarize the findings of the market research survey conducted last month, focusing on customer preferences and satisfaction levels."

40"GENERATED REPORT": "The market research survey findings highlight key customer preferences and satisfaction levels based on last month's data analysis."

41## Example Task #2 Output:

42 "score": 5,

43 "explanation": "The generated report effectively summarizes the market research survey findings from last month, emphasizing customer preferences and satisfaction levels as instructed."

44## Example Task #3 Input:

45"REPORT INSTRUCTIONS": "Provide an overview of the quarterly sales performance, including regional sales data and comparisons with previous quarters."

46"GENERATED REPORT": "The quarterly sales performance overview includes detailed regional sales data and comparative analysis with previous quarters' performance metrics."

47## Example Task #3 Output:

48 "score": 5,

49 "explanation": "The generated report accurately presents the quarterly sales performance overview, incorporating regional sales data and comparisons with previous quarters as instructed."

50## Example Task #4 Input:

51"REPORT INSTRUCTIONS": "Summarize the findings of the customer satisfaction survey, emphasizing areas for improvement and satisfaction ratings."

52GENERATED REPORT: "All customers are extremely satisfied with our services, and no areas for improvement were identified in the satisfaction survey."

53## Example Task #4 Output:

54 "score": 1,

55 "explanation": "The generated report directly contradicts the instructions provided. It falsely claims that all customers are extremely satisfied and fails to mention any areas for improvement, which is not aligned with the instructions to emphasize areas for improvement and satisfaction ratings."

56## Example Task #5 Input:

57"REPORT INSTRUCTIONS": "Analyze the quarterly sales performance, highlighting sales trends, key drivers of sales growth, and areas of concern."

58"GENERATED REPORT": "The quarterly sales performance analysis provides insights into sales trends over the past three months. While there are some positive indicators of sales growth in certain regions, other areas show a decline in sales figures, indicating potential areas of concern."

59## Example Task #5 Output:

60 "score": 3,

61 "explanation": "The generated report partially aligns with the instructions by analyzing quarterly sales performance and highlighting both positive trends and areas of concern. However, it could provide more detailed insights into the key drivers of sales growth as specified in the instructions."

63Reminder: The scores for each task should be correctly formatted as an integer between 1 and 5. Do not repeat the report instructions. Return a single JSON object with the keys 'score' and 'explanation'.

Coverage for intelligence_toolkit/AI/validation_prompt.py: 100%

1 statements