Skip to content

Commit abb57ae

Browse files
jeremyLabradojeremyLabrado
committed
feat: Add multi-image recipe generation feature
- Enable users to capture 4-5 photos of fridge/pantry for comprehensive meal planning - Update Recipe component to support multiple image capture with array state management - Modify RecipeImageIngredients to accept images array instead of single image - Add translations for multi-image UI in English, French, Italian, and Spanish - Update README to document multi-image capability - Backend already supports multiple images via list_images_base64 parameter
1 parent e319f90 commit abb57ae

File tree

4 files changed

+96
-23
lines changed

4 files changed

+96
-23
lines changed

‎README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ We developed this exhibit to create an interactive serverless application using
2222
- **Personalized product information**: Curious about what is in a product and if it is good for you?
2323
Just scan the barcode with the app for an explained list of ingredients/allergens and a personalized summary based on your preferences, health goals, and dietary restrictions. The app provides direct allergen detection and quantitative nutritional analysis using data from Open Food Facts.
2424

25-
- **Personalized recipe generator**: Capture a photo of the ingredients in your fridge, and the app will generate recipes based on your preferences using those ingredients.
25+
- **Personalized recipe generator**: Capture multiple photos of ingredients in your fridge and pantry, and the app will generate recipes based on your preferences using those ingredients.
2626

2727

2828
## Demo
@@ -129,9 +129,9 @@ The architecture of the application can be split in 4 blocks:
129129

130130
#### Food aliment detection
131131

132-
- **Strategy**: Extract ingredients from the image, works well on fruits and vegetables.
132+
- **Strategy**: Extract ingredients from multiple images, works well on fruits and vegetables. Users can capture their entire fridge and pantry to enable comprehensive meal planning.
133133

134-
- **Implementation**: We use Anthropic Claude 3 Sonnet on Amazon Bedrock with its vision capabilities to extract only food elements from the image. This allows us to focus on the food elements and ignore the background or other elements in the image. Claude 3 is a multi-modal model that can handle both text and images. The output is a list of ingredients present in the image.
134+
- **Implementation**: We use Anthropic Claude 3 Sonnet on Amazon Bedrock with its vision capabilities to extract only food elements from the images. This allows us to focus on the food elements and ignore the background or other elements in the images. Claude 3 is a multi-modal model that can handle both text and images. The output is a list of ingredients present across all captured images. The backend processes multiple images via the `list_images_base64` array parameter.
135135

136136
- **Prompt Engineering**: To exploit the full potential of the model, we use a system prompt. A system prompt is a way to provide context, instructions, and guidelines to Claude before presenting it with a question or task. By using a system prompt, you can set the stage for the conversation, specifying Claude's role, personality, tone, or any other relevant information that will help it to better understand and respond to the user's input.
137137

‎resources/ui/src/assets/i18n/all.ts‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,10 @@ const customTranslations: Record<string, Record<string, string>> = {
4949
preference_other_placeholder: "I want to eat healthy",
5050
recipe_retake_photo: "Capture new photo",
5151
recipe_use_this: "Use selected image",
52+
recipe_add_image: "Add to collection",
53+
recipe_add_more: "Add more photos",
54+
recipe_captured_images: "Captured photos",
55+
recipe_generate_recipes: "Generate recipes",
5256
recipe_search_video_src: "Searching for video devices in progress ...",
5357
recipe_take_picture: "Capture image",
5458
image_ingredients_loading: "Identifying ingredients in image...",
@@ -137,6 +141,10 @@ const customTranslations: Record<string, Record<string, string>> = {
137141
product_name_label: "Produit",
138142
recipe_retake_photo: "Prendre une nouvelle photo",
139143
recipe_use_this: "Utiliser cette image",
144+
recipe_add_image: "Ajouter à la collection",
145+
recipe_add_more: "Ajouter plus de photos",
146+
recipe_captured_images: "Photos capturées",
147+
recipe_generate_recipes: "Générer des recettes",
140148
recipe_search_video_src: "Recherche des appareils vidéo en cours...",
141149
recipe_take_picture: "Prendre une photo",
142150
image_ingredients_loading: "Identification des ingrédients dans l'image...",
@@ -225,6 +233,10 @@ const customTranslations: Record<string, Record<string, string>> = {
225233
preference_other_placeholder: "Voglio mangiare in modo sano",
226234
recipe_retake_photo: "Riprova",
227235
recipe_use_this: "Usa",
236+
recipe_add_image: "Aggiungi alla raccolta",
237+
recipe_add_more: "Aggiungi altre foto",
238+
recipe_captured_images: "Foto acquisite",
239+
recipe_generate_recipes: "Genera ricette",
228240
recipe_search_video_src: "Ricerca di dispositivi video in corso…",
229241
recipe_take_picture: "Cattura immagine",
230242
image_ingredients_loading:
@@ -314,6 +326,10 @@ const customTranslations: Record<string, Record<string, string>> = {
314326
preference_other_placeholder: "Quiero comer saludable",
315327
recipe_retake_photo: "Capturar nueva foto",
316328
recipe_use_this: "Usar imagen seleccionada",
329+
recipe_add_image: "Añadir a la colección",
330+
recipe_add_more: "Añadir más fotos",
331+
recipe_captured_images: "Fotos capturadas",
332+
recipe_generate_recipes: "Generar recetas",
317333
recipe_search_video_src: "Buscando dispositivos de video en progreso ...",
318334
recipe_take_picture: "Capturar imagen",
319335
image_ingredients_loading: "Identificando ingredientes en la imagen...",
@@ -393,6 +409,10 @@ arabic:{
393409
"preference_other_placeholder": "أريد أن أتناول طعامًا صحيًا",
394410
"recipe_retake_photo": "التقط صورة جديدة",
395411
"recipe_use_this": "استخدم الصورة المحددة",
412+
"recipe_add_image": "أضف إلى المجموعة",
413+
"recipe_add_more": "أضف المزيد من الصور",
414+
"recipe_captured_images": "الصور الملتقطة",
415+
"recipe_generate_recipes": "إنشاء الوصفات",
396416
"recipe_search_video_src": "جارٍ البحث عن أجهزة الفيديو ...",
397417
"recipe_take_picture": "التقط صورة",
398418
"image_ingredients_loading": "جارٍ تحديد المكونات في الصورة...",

‎resources/ui/src/pages/components/recipe.tsx‎

Lines changed: 67 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,8 @@ const Recipe: React.FC = () => {
2121
const webcamRef = useRef<any>();
2222
const [imgSrc, setImgSrc] = useState<string | null>(null);
2323
const [myValue, setMyValue] = useState([]);
24-
const [selectedImgSrc, setSelectedImgSrc] = useState<string | null>(null);
24+
const [capturedImages, setCapturedImages] = useState<string[]>([]);
25+
const [selectedImgSrc, setSelectedImgSrc] = useState<string[]>([]);
2526
const [showWebcam, setShowWebcam] = useState(false);
2627
const [showOptionsButtons, setShowOptionsButtons] = useState(true);
2728
const [loadingVideoDevices, setLoadingVideoDevices] = useState(false);
@@ -131,8 +132,20 @@ function resizeBase64Image(base64Image: string, width: number, height: number):
131132
}
132133
}, [webcamRef]);
133134

135+
const addImage = () => {
136+
if (imgSrc) {
137+
setCapturedImages([...capturedImages, imgSrc]);
138+
setImgSrc(null);
139+
setShowOptionsButtons(true);
140+
}
141+
};
142+
143+
const removeImage = (index: number) => {
144+
setCapturedImages(capturedImages.filter((_, i) => i !== index));
145+
};
146+
134147
const startWebcam = () => {
135-
setSelectedImgSrc(null);
148+
setSelectedImgSrc([]);
136149
setImgSrc(null);
137150
setShowWebcam(true);
138151
enumerateDevices();
@@ -141,17 +154,17 @@ function resizeBase64Image(base64Image: string, width: number, height: number):
141154
const retake = () => {
142155
setImgSrc(null);
143156
};
144-
const useThisImage = () => {
145-
setShowOptionsButtons(false);
146-
//setShowWebcam(false);
147157

148-
setSelectedImgSrc(imgSrc);
158+
const useTheseImages = () => {
159+
const allImages = imgSrc ? [...capturedImages, imgSrc] : capturedImages;
160+
setSelectedImgSrc(allImages);
161+
setShowOptionsButtons(false);
149162
};
150163

151164
const currentTranslations = customTranslations[language];
152165

153166
const fileUploadOnChange = ({ detail }) => {
154-
setSelectedImgSrc(null);
167+
setSelectedImgSrc([]);
155168
console.log(detail.value);
156169
const files = detail.value;
157170
if (files.length > 0) {
@@ -239,6 +252,42 @@ function resizeBase64Image(base64Image: string, width: number, height: number):
239252
)}
240253

241254
{/* Conditionally render the image */}
255+
{capturedImages.length > 0 && !imgSrc && (
256+
<div style={{ textAlign: "center" }}>
257+
<h4>{currentTranslations["recipe_captured_images"]} ({capturedImages.length})</h4>
258+
<div style={{ display: "flex", flexWrap: "wrap", gap: "10px", justifyContent: "center" }}>
259+
{capturedImages.map((img, index) => (
260+
<div key={index} style={{ position: "relative" }}>
261+
<img
262+
src={img}
263+
style={{
264+
borderRadius: "5px",
265+
height: "150px",
266+
objectFit: "cover",
267+
}}
268+
/>
269+
<Button
270+
iconName="close"
271+
variant="icon"
272+
onClick={() => removeImage(index)}
273+
ariaLabel="Remove image"
274+
/>
275+
</div>
276+
))}
277+
</div>
278+
<div style={{ marginTop: "10px" }}>
279+
<SpaceBetween direction="horizontal" size="s">
280+
<Button onClick={startWebcam} variant="normal">
281+
{currentTranslations["recipe_add_more"]}
282+
</Button>
283+
<Button onClick={useTheseImages} variant="primary">
284+
{currentTranslations["recipe_generate_recipes"]}
285+
</Button>
286+
</SpaceBetween>
287+
</div>
288+
</div>
289+
)}
290+
242291
{imgSrc && (
243292
<>
244293
<div style={{ textAlign: "center", maxHeight: "70vh" }}>
@@ -263,12 +312,16 @@ function resizeBase64Image(base64Image: string, width: number, height: number):
263312
}}
264313
>
265314
<SpaceBetween direction="horizontal" size="s">
266-
{/* <Button onClick={retake} variant="primary">
315+
<Button onClick={retake} variant="normal">
267316
{currentTranslations["recipe_retake_photo"]}
268-
</Button> */}
269-
270-
<Button onClick={useThisImage} variant="primary">
271-
{currentTranslations["recipe_use_this"]}
317+
</Button>
318+
<Button onClick={addImage} variant="normal">
319+
{currentTranslations["recipe_add_image"]}
320+
</Button>
321+
<Button onClick={useTheseImages} variant="primary">
322+
{capturedImages.length > 0
323+
? currentTranslations["recipe_generate_recipes"]
324+
: currentTranslations["recipe_use_this"]}
272325
</Button>
273326
</SpaceBetween>
274327
</div>
@@ -348,10 +401,10 @@ function resizeBase64Image(base64Image: string, width: number, height: number):
348401

349402
<div id="reader"></div>
350403

351-
{selectedImgSrc && (
404+
{selectedImgSrc.length > 0 && (
352405
<div>
353406
<ImageIngredients
354-
img={selectedImgSrc}
407+
images={selectedImgSrc}
355408
language={language}
356409
onRecipePropositionsDone={() => {
357410
setShowWebcam(false);

‎resources/ui/src/pages/components/recipe_image_ingredients.tsx‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -13,29 +13,29 @@ import { on } from "events";
1313
import { FlowItems } from "./flowitems";
1414

1515
interface RecipeImageIngredientsProps {
16-
img: string;
16+
images: string[];
1717
language: string;
1818
onRecipePropositionsDone?: () => void;
1919
}
2020

2121
const RecipeImageIngredients: React.FC<RecipeImageIngredientsProps> = ({
22-
img,
22+
images,
2323
language,
2424
onRecipePropositionsDone,
2525
}) => {
2626
const currentTranslations = customTranslations[language];
27-
const [loadingImageIngredients, setLoadingImageIngredients] = useState(true); // Added loading state
27+
const [loadingImageIngredients, setLoadingImageIngredients] = useState(true);
2828
const [imageIngredientsResponse, setImageIngredientsResponse] = useState<
2929
any[]
3030
>([]);
31-
const [responseReceived, setResponseReceived] = useState(false); // Added loading state
31+
const [responseReceived, setResponseReceived] = useState(false);
3232

3333
useEffect(() => {
3434
const fetchData = async () => {
3535
try {
3636
setResponseReceived(false);
3737
const body = {
38-
list_images_base64: [img],
38+
list_images_base64: images,
3939
language: language,
4040
};
4141

@@ -54,7 +54,7 @@ const RecipeImageIngredients: React.FC<RecipeImageIngredientsProps> = ({
5454
};
5555

5656
fetchData();
57-
}, [img, language]);
57+
}, [images, language]);
5858

5959
return (
6060
<TextContent>

0 commit comments

Comments
 (0)