Merge pull request #28 from jeremyLabrado/fix/migrate-to-nova-canvas

Modernize preferences UI, add new parameters and improve image generation prompt

Author: brnaba-aws

README.md

@@ -20,7 +20,7 @@ We developed this exhibit to create an interactive serverless application using
 ## Features overview
 
 - **Personalized product information**: Curious about what is in a product and if it is good for you?
-Just scan the barcode with the app for an explained list of ingredients/alergens and a personalized summary based on your preferences.
+Just scan the barcode with the app for an explained list of ingredients/allergens and a personalized summary based on your preferences, health goals, and dietary restrictions. The app provides direct allergen detection and quantitative nutritional analysis using data from Open Food Facts.
 
 - **Personalized recipe generator**: Capture a photo of the ingredients in your fridge, and the app will generate recipes based on your preferences using those ingredients.
 
@@ -61,6 +61,8 @@ The architecture of the application can be split in 4 blocks:
 #### Product Management:
 
 - **Implementation**: Using AWS Lambda for server-side logic and a database from [Open Food Facts](https://fr.openfoodfacts.org/) accessed through APIs.
+- **Data Integration**: The app retrieves allergen tags and nutritional data (calories, sugars, fats, proteins, etc.) from Open Food Facts API for accurate, data-driven recommendations.
+- **Safety Features**: Direct allergen detection from API data ensures reliable allergen warnings without relying solely on ingredient text parsing.
 
 #### Product Summary and Generative Recipe:
 
@@ -223,6 +225,31 @@ The output format is a Markdown file to faciliate the display of the recipe on t
 
 - **Solution**: The same prompt is utilized, but the LLM is instructed to generate the output in a specific language, catering to the user's language preference (English/French).
 
+**Direct Allergen Detection and Nutritional Analysis**
+
+- **Challenge**: Ensuring accurate allergen warnings and providing quantitative nutritional recommendations based on user health goals.
+
+- **Solution**: Integrated Open Food Facts API to retrieve `allergens_tags` and `nutriments` fields. The app filters key nutritional data (calories, sugars, fats, proteins, salt, fiber) and stores them in DynamoDB. Product summaries now include:
+  - Direct allergen detection with prominent warnings
+  - Specific nutritional values (e.g., "539 kcal/100g", "56.3g sugars")
+  - Health goal-specific recommendations (weight loss, muscle gain, etc.)
+  - Dietary preference compatibility (keto, low carb, low sodium)
+  
+  A custom `DecimalEncoder` handles DynamoDB Decimal type serialization to JSON, ensuring proper data formatting in API responses.
+
+**Dietary Labels and Religious Requirements**
+
+- **Challenge**: Accurately identifying vegan, vegetarian, halal, and kosher products without relying solely on ingredient text parsing.
+
+- **Solution**: Integrated `labels_tags` and `categories` fields from Open Food Facts API. The app now provides:
+  - Direct vegan/vegetarian detection from product labels
+  - Halal and kosher certification identification
+  - Category-based product context for better recommendations
+  - Religious requirement matching with clear certification status
+  - Fallback to ingredient analysis when labels are unavailable
+  
+  This reduces LLM hallucination and provides more confident dietary and religious compatibility assessments.
+
 
 
 ## Key Technical Features

lambda/barcode_image/index.py

@@ -24,8 +24,8 @@ def generate_product_summary_prompt(
 ):
     return f"""Human:
         You are a nutrition expert. I will give you a nutritional list of a product, as sold per 100 g / 100 ml.
-        What, in your opinion, is the most unhealthy component? You must imagine the quantity of the most unhealthy component in terms of the quotient so that I realize how bad it is.
-        Respond in the form of a prompt in English, which will be used to generate an image in English. Respond only with the prompt.
+        What, in your opinion, is the most notable nutritional characteristic? Create a visual representation using actual food items or ingredients.
+        Respond in the form of a prompt in English for image generation. The prompt should describe a clean, professional food photography scene WITHOUT any text, labels, or words visible in the image.
         -----------------------------------
         Example 1:  
         Nutritional list:
@@ -41,7 +41,7 @@ def generate_product_summary_prompt(
             "salt_100g": "0,107 g"
         
         Response: 
-         A jar of chocolate hazelnut spread next to 14 cubes of sugar labeled "diabetes danger".
+         A jar of chocolate hazelnut spread surrounded by sugar cubes and hazelnuts on a white background, professional food photography, no text or labels visible.
         
         -------------------------------------
         Example 2:
@@ -59,7 +59,7 @@ def generate_product_summary_prompt(
                 "proteins_100g": "12,5",
                 "salt_100g": "1,02"
 
-        Response: "an hamburger and an evil teaspoon full of salt"
+        Response: A hamburger with a small pile of salt crystals beside it, professional food photography, clean white background, no text visible.
         -------------------------------------
         Example 3
         Liste nutritionnelle:
@@ -75,7 +75,7 @@ def generate_product_summary_prompt(
                 "fiber_100g": "3.5", 
                 "proteins_100g": "6.1", 
                 "salt_100g": "1.2"
-        Response:  "barbecue potato chips and a salt shaker"
+        Response: Barbecue potato chips in a bowl with salt crystals scattered around, professional food photography, no text or labels.
 
         -------------------------------------
 
@@ -128,7 +128,8 @@ def get_image(prompt):
     body=json.dumps({
         "taskType": "TEXT_IMAGE",
         "textToImageParams": {
-            "text": f"{prompt}, professional food photography, studio lighting, clean composition, high resolution, editorial quality, commercial product photography style"
+            "text": f"{prompt}, professional food photography, studio lighting, clean composition, high resolution, editorial quality, commercial product photography style",
+            "negativeText": "text, words, letters, labels, writing, typography, captions, watermarks, logos, signs, numbers, alphabet"
         },
         "imageGenerationConfig": {
             "numberOfImages": 1,

lambda/barcode_ingredients/index.py

@@ -1,6 +1,7 @@
 import time
 import boto3
 import json
+from decimal import Decimal
 from botocore.exceptions import ClientError
 import urllib.parse
 import requests
@@ -16,6 +17,12 @@
 bedrock = boto3.client("bedrock-runtime")
 dynamodb = boto3.resource('dynamodb')
 
+class DecimalEncoder(json.JSONEncoder):
+    def default(self, obj):
+        if isinstance(obj, Decimal):
+            return float(obj)
+        return super(DecimalEncoder, self).default(obj)
+
 
 PRODUCT_TABLE_NAME = os.environ['PRODUCT_TABLE_NAME']
 OPEN_FOOD_FACTS_TABLE_NAME = os.environ['OPEN_FOOD_FACTS_TABLE_NAME']
@@ -101,7 +108,7 @@ def make_api_request(product_code):
     api_url = os.environ.get('API_URL')
     url = f'{api_url}/api/v2/product/{product_code}'
     headers = {'Accept': 'application/json'}
-    fixed_params = {'fields': 'ingredients_text,additives_tags,product_name'}
+    fixed_params = {'fields': 'ingredients_text,additives_tags,product_name,allergens_tags,nutriments,labels_tags,categories'}
     full_url = f'{url}?{urllib.parse.urlencode(fixed_params)}'
     logger.debug("Calling the API to get the product informations")
 
@@ -158,6 +165,41 @@ def call_claude_haiku(prompt_text):
     results = response_body.get("content")[0].get("text")
     return results
 
+def filter_nutriments(nutriments):
+    """
+    Filters nutriments to only include key nutritional fields.
+    Converts numeric values to Decimal for DynamoDB compatibility.
+    
+    Args:
+        nutriments (dict): Full nutriments dictionary from API
+        
+    Returns:
+        dict: Filtered nutriments with only key fields (excludes None values)
+    """
+    if not nutriments:
+        return {}
+    
+    key_fields = [
+        'energy-kcal_100g',
+        'carbohydrates_100g',
+        'sugars_100g',
+        'fat_100g',
+        'saturated-fat_100g',
+        'salt_100g',
+        'sodium_100g',
+        'proteins_100g',
+        'fiber_100g'
+    ]
+    
+    # Filter out None values and convert to Decimal for DynamoDB
+    filtered = {}
+    for k in key_fields:
+        if k in nutriments and nutriments[k] is not None:
+            # Convert to Decimal for DynamoDB compatibility
+            filtered[k] = Decimal(str(nutriments[k]))
+    
+    return filtered
+
 def clean_text_in_brackets(text):
     """
     Removes text enclosed within parentheses, square brackets, or curly braces from the given text.
@@ -257,8 +299,8 @@ def get_product_from_db(product_code, language):
         product_code (str): The code of the product to retrieve information for.
 
     Returns:
-        tuple: A tuple containing product name, ingredients, and additives if the product is found in the database;
-               otherwise, returns (None, None, None).
+        tuple: A tuple containing product name, ingredients, additives, allergens, nutriments, labels, and categories if the product is found in the database;
+               otherwise, returns (None, None, None, None, None, None, None).
     """
 
     table = dynamodb.Table(PRODUCT_TABLE_NAME)
@@ -276,19 +318,23 @@ def get_product_from_db(product_code, language):
             product_name = item.get('product_name')
             ingredients = item.get('ingredients')
             additives = item.get('additives')
+            allergens = item.get('allergens_tags', [])
+            nutriments = item.get('nutriments', {})
+            labels = item.get('labels_tags', [])
+            categories = item.get('categories', '')
 
             # Check if either ingredients or additives don't exist, then return None
             if ingredients is None or additives is None:
-                return None, None, None
-            return product_name, ingredients, additives
+                return None, None, None, None, None, None, None
+            return product_name, ingredients, additives, allergens, nutriments, labels, categories
         else:
-            return None, None, None
+            return None, None, None, None, None, None, None
     except Exception as e:
         logger.error("Error while getting the Product from database", e)
-        return None, None, None
+        return None, None, None, None, None, None, None
 
 @tracer.capture_method
-def write_product_to_db(product_code, language, product_name, ingredients, additives):
+def write_product_to_db(product_code, language, product_name, ingredients, additives, allergens, nutriments, labels, categories):
     """
     Writes product information product table.
 
@@ -297,6 +343,10 @@ def write_product_to_db(product_code, language, product_name, ingredients, addit
         product_name (str): The name of the product.
         ingredients (list): The list of ingredients of the product.
         additives (list): The list of additives of the product.
+        allergens (list): The list of allergens tags.
+        nutriments (dict): The filtered nutriments data.
+        labels (list): The list of labels tags.
+        categories (str): The product categories.
 
     Returns:
         None
@@ -316,6 +366,22 @@ def write_product_to_db(product_code, language, product_name, ingredients, addit
 
         if ingredients is not None:
             item['ingredients'] = ingredients
+            
+        # Only add allergens if list is not empty
+        if allergens and len(allergens) > 0:
+            item['allergens_tags'] = allergens
+            
+        # Only add nutriments if dict is not empty
+        if nutriments and len(nutriments) > 0:
+            item['nutriments'] = nutriments
+            
+        # Only add labels if list is not empty
+        if labels and len(labels) > 0:
+            item['labels_tags'] = labels
+            
+        # Only add categories if not empty
+        if categories:
+            item['categories'] = categories
 
         # Write item to DynamoDB table
         response = table.put_item(Item=item)
@@ -370,8 +436,8 @@ def fetch_new_product(product_code, language):
         product_code (str): The code of the product to fetch.
 
     Returns:
-        tuple: A tuple containing dictionaries of ingredients and additives, along with the product name,
-               if the product information is successfully fetched from the API; otherwise, returns (None, None, None, None).
+        tuple: A tuple containing dictionaries of ingredients, additives, allergens, nutriments, labels, categories, and product name,
+               if the product information is successfully fetched from the API; otherwise, returns (None, None, None, None, None, None, None).
     """
 
     response_data = get_product_from_open_food_facts_db(product_code)
@@ -382,6 +448,11 @@ def fetch_new_product(product_code, language):
     if response_data is not None:
 
         additives=[]
+        allergens=[]
+        nutriments={}
+        labels=[]
+        categories=''
+        
         if 'product' not in response_data or 'ingredients_text' not in response_data['product']:
             raise ValueError("Missing ingredients in Open Food Facts API. Unable to generate a personalized summary for this product.")
 
@@ -397,11 +468,27 @@ def fetch_new_product(product_code, language):
         response_additives = additives
         if additives:
             response_additives = parse_additives_description(additives, language)
+            
+        # Extract allergens
+        if 'product' in response_data and 'allergens_tags' in response_data['product']:
+            allergens = response_data['product']['allergens_tags']
+            
+        # Extract and filter nutriments
+        if 'product' in response_data and 'nutriments' in response_data['product']:
+            nutriments = filter_nutriments(response_data['product']['nutriments'])
+            
+        # Extract labels
+        if 'product' in response_data and 'labels_tags' in response_data['product']:
+            labels = response_data['product']['labels_tags']
+            
+        # Extract categories
+        if 'product' in response_data and 'categories' in response_data['product']:
+            categories = response_data['product']['categories']
 
-        return response_ingredients, response_additives, product_name
+        return response_ingredients, response_additives, product_name, allergens, nutriments, labels, categories
 
     else:
-        return None, None, None
+        return None, None, None, None, None, None, None
 
 @logger.inject_lambda_context(log_event=True)
 @tracer.capture_lambda_handler
@@ -412,18 +499,18 @@ def handler(event, context):
         product_code = fields[1]
         language = fields[2]
         logger.debug("ProductCode="+product_code)
-        product_name, response_ingredients, response_additives = get_product_from_db(product_code, language)
+        product_name, response_ingredients, response_additives, allergens, nutriments, labels, categories = get_product_from_db(product_code, language)
 
         if product_name is not None:        
             logger.debug("Product found in the database")
         else:
             logger.debug("Product not found in the database")
 
-            response_ingredients, response_additives, product_name = fetch_new_product(product_code, language)
+            response_ingredients, response_additives, product_name, allergens, nutriments, labels, categories = fetch_new_product(product_code, language)
 
 
             if  response_ingredients is not None:
-                write_product_to_db(product_code, language, product_name, response_ingredients, response_additives)
+                write_product_to_db(product_code, language, product_name, response_ingredients, response_additives, allergens, nutriments, labels, categories)
 
             if(response_ingredients is None):
                 response_ingredients = {"Ingredients Generation Error": "Description Generation Unavailable"}                
@@ -433,14 +520,18 @@ def handler(event, context):
                 "ingredients_description": response_ingredients,
                 "additives_description": response_additives,
                 "product_name": product_name,
+                "allergens_tags": allergens,
+                "nutriments": nutriments,
+                "labels_tags": labels,
+                "categories": categories,
         }
 
         logger.debug("Response", extra=response)
 
         # Return JSON response
         return {
             "statusCode": 200,
-            "body": json.dumps(response),
+            "body": json.dumps(response, cls=DecimalEncoder),
             "headers": {
                 "Access-Control-Allow-Headers": "*",
                 "Access-Control-Allow-Origin": "*",