Skip to content

Commit a9f0f20

Browse files
imran-binhasanimran-binhasan
committed
Eval driven system design cookbook (openai#1875)
1 parent 20d0eb7 commit a9f0f20

File tree

262 files changed

+2592
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

262 files changed

+2592
-0
lines changed

‎examples/partners/eval_driven_system_design/data/ground_truth/audit_results/Gas_20240605_164059_Raven_Scan_3_jpeg.rf.e3408aa2b936afd1f1aed84fa40d454e.json‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"not_travel_related": false,
3+
"amount_over_limit": false,
4+
"math_error": false,
5+
"handwritten_x": false,
6+
"reasoning": "1. The receipt is for gas (E-85 fuel), which is a travel-related expense. Therefore, NOT_TRAVEL_RELATED is FALSE. \n2. The total amount of the receipt is $36.16, which does not exceed $50. Therefore, AMOUNT_OVER_LIMIT is FALSE. \n3. The subtotal of $36.16 matches the total of $36.16, indicating no math errors. Therefore, MATH_ERROR is FALSE. \n4. The handwritten notes do not contain an 'X'. Therefore, HANDWRITTEN_X is FALSE.",
7+
"needs_audit": false
8+
}

‎examples/partners/eval_driven_system_design/data/ground_truth/audit_results/Nissan_20250205_121534_Raven_Scan_10_jpeg.rf.5524d7df5648dc37d9bf264aa3f1e2c5.json‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"not_travel_related": false,
3+
"amount_over_limit": false,
4+
"math_error": false,
5+
"handwritten_x": false,
6+
"reasoning": "The receipt is for gasoline, which is considered a travel-related expense, so it does not meet the NOT_TRAVEL_RELATED criterion. The total amount is $49.61, which is below the $50 limit, so the AMOUNT_OVER_LIMIT criterion is not violated. There are no discrepancies in the math for computing the total, confirming that MATH_ERROR is false. There is no 'X' present in the handwritten notes, meaning HANDWRITTEN_X is also false. Since none of the criteria are violated, the receipt does not need auditing.",
7+
"needs_audit": false
8+
}

‎examples/partners/eval_driven_system_design/data/ground_truth/audit_results/Nissan_20250205_122340_Raven_Scan_2_jpeg.rf.3f26976203cf4683fc90cbbe34e876bd.json‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"not_travel_related": false,
3+
"amount_over_limit": true,
4+
"math_error": false,
5+
"handwritten_x": true,
6+
"reasoning": "The receipt is for fuel, which is classified as a travel-related expense, hence it does not satisfy the NOT_TRAVEL_RELATED criterion. The total amount of $60.91 exceeds the $50 limit, thus satisfying the AMOUNT_OVER_LIMIT criterion. There are no discrepancies in the arithmetic calculations for the total, so MATH_ERROR is false. However, there is a handwritten 'X' noted in the handwritten notes, which satisfies the HANDWRITTEN_X criterion. 1. NOT_TRAVEL_RELATED: This is a fuel purchase (travel-related), so not_travel_related=false. 2. AMOUNT_OVER_LIMIT: Total of $60.91 exceeds $50, so amount_over_limit=true. 3. MATH_ERROR: 4.199×14.5076≈60.9174, which rounds up to $60.92 or truncates to $60.91, the listed total (either would be acceptable), so math_error=false. 4. HANDWRITTEN_X: The note 'X' is written on the the receipt, so handwritten_x=true. At least one criterion is violated, so needs_audit=true.",
7+
"needs_audit": true
8+
}

‎examples/partners/eval_driven_system_design/data/ground_truth/audit_results/Nissan_20250205_122340_Raven_Scan_3_jpeg.rf.8f7d7e820eeaf8cfb2d61ef8232fc1af.json‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"not_travel_related": false,
3+
"amount_over_limit": true,
4+
"math_error": false,
5+
"handwritten_x": false,
6+
"reasoning": "The receipt is for fuel, which is considered travel-related, hence NOT_TRAVEL_RELATED is FALSE. The total amount of the receipt is $72.10, which exceeds the $50 limit making AMOUNT_OVER_LIMIT TRUE. The math adds up correctly as the subtotal matches the total, so MATH_ERROR is FALSE. There are no handwritten 'X's in the notes so HANDWRITTEN_X is FALSE. Since one of the criteria (AMOUNT_OVER_LIMIT) is violated, the receipt needs auditing.",
7+
"needs_audit": true
8+
}

‎examples/partners/eval_driven_system_design/data/ground_truth/audit_results/Retail_20240605_163628_Raven_Scan_1_jpeg.rf.c28eb1342052ef8af13d54a329c8218b.json‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"not_travel_related": true,
3+
"amount_over_limit": true,
4+
"math_error": false,
5+
"handwritten_x": false,
6+
"reasoning": "1. NOT_TRAVEL_RELATED: The receipt is for home goods and cleaning supplies, which are not travel-related expenses. Therefore, this criterion is TRUE.\n\n2. AMOUNT_OVER_LIMIT: The total of the receipt is $273.52, which is over the $50 limit, making this criterion TRUE.\n\n3. MATH_ERROR: The subtotal adds up correctly when calculated with tax. The subtotal $252.67 plus tax $20.85 equals the total $273.52. Thus, there is no math error, and this is FALSE.\n\n4. HANDWRITTEN_X: There is no 'X' in the handwritten notes provided. Thus, this is FALSE.\n\nSince two of the criteria (AMOUNT_OVER_LIMIT and MATH_ERROR) are TRUE, the final determination is that this receipt needs to be audited.",
7+
"needs_audit": true
8+
}

‎examples/partners/eval_driven_system_design/data/ground_truth/audit_results/Sequoia-gas_20241213_193919_Raven_Scan_4_jpeg.rf.b79fe0306ef298356ee020fd63572734.json‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"not_travel_related": false,
3+
"amount_over_limit": false,
4+
"math_error": false,
5+
"handwritten_x": false,
6+
"reasoning": "1. NOT_TRAVEL_RELATED: The receipt includes fuel as a purchase which is considered travel-related, therefore this criterion is FALSE.\n2. AMOUNT_OVER_LIMIT: The total amount is $49.54, which does not exceed $50, so this is FALSE.\n3. MATH_ERROR: The line item for fuel ($49.54) matches the total provided ($49.54), hence there is no math error, making this FALSE.\n4. HANDWRITTEN_X: There is no 'X' in the handwritten notes, so this is FALSE. \nSince none of the criteria are violated, the receipt does not need an audit.",
7+
"needs_audit": false
8+
}

‎examples/partners/eval_driven_system_design/data/ground_truth/audit_results/Sequoia-gas_20241213_193919_Raven_Scan_9_jpeg.rf.4756583269eace7e380051b580022a04.json‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"not_travel_related": false,
3+
"amount_over_limit": true,
4+
"math_error": false,
5+
"handwritten_x": false,
6+
"reasoning": "The expenses are for fuel, which is a travel-related cost, so the criterion for being 'not travel-related' is FALSE. The total amount of the receipt is $60.94, which exceeds the $50 limit, making the 'Amount Over Limit' criterion TRUE. There are no discrepancies in the math as the total matches the sum of the item prices, and there is no 'X' in the handwritten notes. Since one of the criteria (Amount Over Limit) is violated, the receipt needs to be audited.",
7+
"needs_audit": true
8+
}

‎examples/partners/eval_driven_system_design/data/ground_truth/audit_results/Sequoia-gas_20241213_194524_Raven_Scan_1_jpeg.rf.ea9c9e22f0e18cb64e158b64b43de2c0.json‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"not_travel_related": false,
3+
"amount_over_limit": false,
4+
"math_error": false,
5+
"handwritten_x": false,
6+
"reasoning": "1. NOT_TRAVEL_RELATED: The receipt is for fuel, which is considered a travel-related expense, so this is FALSE. 2. AMOUNT_OVER_LIMIT: The total amount is $41.46, which does not exceed $50, so this is FALSE. 3. MATH_ERROR: The total of the line item ($41.46) matches the total provided, so this is FALSE. 4. HANDWRITTEN_X: There is no 'X' in the handwritten notes, so this is FALSE. Since none of the criteria are violated, the receipt does not need auditing.",
7+
"needs_audit": false
8+
}

‎examples/partners/eval_driven_system_design/data/ground_truth/audit_results/Sequoia-gas_20241213_194919_Raven_Scan_3_jpeg.rf.842940493d72ebac96a20eed97b226fc.json‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"not_travel_related": false,
3+
"amount_over_limit": false,
4+
"math_error": false,
5+
"handwritten_x": false,
6+
"reasoning": "1. NOT_TRAVEL_RELATED: The receipt is from Shell Gasoline and includes fuel expenses, making it travel-related. Therefore, this criterion is FALSE. 2. AMOUNT_OVER_LIMIT: The total amount of $38.26 is below the $50 threshold, so this criterion is FALSE. 3. MATH_ERROR: The line item for fuel ($38.26) matches the computed total, indicating no math errors, so this criterion is FALSE. 4. HANDWRITTEN_X: There is no 'X' present in the handwritten notes on the receipt, making this criterion FALSE. Since all criteria are not violated (all FALSE), the receipt does not need auditing.",
7+
"needs_audit": false
8+
}

‎examples/partners/eval_driven_system_design/data/ground_truth/audit_results/Sequoia-gas_20241213_194920_Raven_Scan_11_jpeg.rf.5fa0c8031793626f1d87a7acb08617dd.json‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"not_travel_related": false,
3+
"amount_over_limit": true,
4+
"math_error": false,
5+
"handwritten_x": false,
6+
"reasoning": "The receipt is for 'Unleaded' fuel from a gas station, which is a travel-related expense, thus the NOT_TRAVEL_RELATED criterion is FALSE. The total amount of the receipt is $62.68, exceeding the $50 limit, making AMOUNT_OVER_LIMIT TRUE. There are no discrepancies in the math calculations presented, hence MATH_ERROR is FALSE. There is no 'X' noted in the handwritten notes, so HANDWRITTEN_X is FALSE. Since the AMOUNT_OVER_LIMIT criteria is violated, the receipt needs auditing.",
7+
"needs_audit": true
8+
}

0 commit comments

Comments
 (0)