You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: custom_pytorch_yolov5/custom_pytorch.md
+42-38Lines changed: 42 additions & 38 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,4 +1,4 @@
1
-
# Using a Mask Model on OCI with YOLOv5: Custom PyTorch From Scratch
1
+
# YOLOv5 and OCI: Custom PyTorch Code From Scratch
2
2
3
3
## Introduction
4
4
@@ -22,19 +22,17 @@ I decided to create a "modified" version of what YOLOv5 does, by taking advantag
22
22
23
23
I believed custom PyTorch code would be great, because simply using YOLOv5's repository didn't give you 100% flexibility and responsiveness (real-time), so I decided to __very slightly__ add some *extra* functionalities (we'll talk about them below). If you're trying to use [the standard GitHub repository for YOLOv5](https://github.com/ultralytics/yolov5), you'll find that you can use their code like [this detector](https://github.com/ultralytics/yolov5/blob/master/detect.py) to post-process video or image files. You can also use it directly with a YouTube video, and an integrated youtube downloader will download frames and process them.
24
24
25
-
But what is the definition of real time? I want every frame that I see in my computer, somehow (be it either a camera frame from my webcam or a YouTube video, or even my screen) to display the results of my detection immediately.
25
+
But what is the definition of real time? I want every frame that I see in my computer, somehow (be it either a camera frame from my webcam or a YouTube video, or even my screen) to display the results of my detection immediately. This is why I created my own custom code to detect with PyTorch.
26
26
27
-
This is why I created my own custom PyTorch detector.
27
+
Finally, I'd like to mention my journey through a *very painful* road of finding *a few* bugs on the Windows Operating System and trying to *virtualize* your webcam feed. There's this great plugin that could "replicate" your camera feed into a virtual version that you could use in any program (you could give your computer any program / input and feed it into the webcam stream, so that it looked like your webcam feed was coming from somewhere else), and it was really great:
28
28
29
-
Finally, I'd like to mention my journey through a very painful road of finding __a few__ bugs on the Windows Operating System and trying to *virtualize* your webcam feed. There's this great plugin that could "replicate" your camera feed into a virtual version that you could use in any program (you could give your computer any program / input and feed it into the webcam stream, so that it looked like your webcam feed was coming from somewhere else), and it was really great:
30
-
31
-
29
+
32
30
33
31
This was an [OBS (Open Broadcaster Software)](https://obsproject.com/) plugin. OBS is the go-to program to use when you're planning to make a livestream. However, this plugin was discontinued in OBS version 28, and all problems came with this update. I prepared this bug-compilation image so you can feel the pain too:
34
32
35
-
33
+
36
34
37
-
So, once we've established that there are various impediments that prevent us from happily developing in a stable environment, let's start implementing.
35
+
So, once we've established that there are several roadblocks that prevent us from happily developing in a stable environment, we finally understand the "why" of this article. Let's begin implementing.
38
36
39
37
## Implementation
40
38
@@ -46,54 +44,53 @@ Technical requirements are Python 3.8 or higher, and PyTorch 1.7 or higher. [Her
46
44
47
45
First, we will use the `argparse` to include additional parameters to our Python script. I added three optional parameters:
48
46
49
-
50
-
51
-
> **Note**: the confidence threshold will only display detected objects if the confidence score of the model's prediction is higher than a given value (0-1).
47
+
48
+
> **Note**: the confidence threshold will only display detected objects if the confidence score of the model's prediction is higher than the given value (0.0-1.0).
52
49
53
-
These parameters' default can be modified. The frequency parameter will determine how many frames to detect (e.g. a frame step). If the user specifies any number, then only one frame every N frames will be used for detection. This can be useful if you're expecting your data to be very similar within different sequential frames, as one detection of the object will probably suffice. Specifying this frame step is also beneficial to avoid cost overages when making these predictions (electricity bill beware) or OCI costs if you're using the Cloud.
50
+
These argparse parameters' default values can always be modified. The _frequency_ parameter will determine how many frames to detect (e.g. a frame step). If the user specifies any number N, then only 1 frame every N frames will be used for detection. This can be useful if you're expecting your data to be very similar within sequential frames, as the detection of one object, in one frame, will suffice. Specifying this frame step is also beneficial to avoid cost overages when making these predictions (electricity bill beware, or OCI costs if you're using Oracle Cloud).
54
51
55
-
After this initial configuration, we're ready to load our custom model. You can find the pre-trained custom model's weights for the Mask Detection Model being featured in this article [in this link](https://www.kaggle.com/datasets/jasperan/covid-19-mask-detection?select=best.pt). You'll need to have this file within reach for your Python code to work.
52
+
After this initial configuration, we're ready to load our custom model. You can find the pre-trained custom model's weights for the Mask Detection Model being featured [in this link](https://www.kaggle.com/datasets/jasperan/covid-19-mask-detection?select=best.pt). You'll need to have this file within reach of your Python coding/execution environment to work.
56
53
57
-
So, what we do now, is load the custom weights file:
54
+
So, we now load the custom weights file:
58
55
59
-
56
+
60
57
> **Note**: we specify the model as a custom YOLO detector, and give it the model's weights file as input.
61
58
62
-
Now, we're ready to get started. We create our main loop, which infinitely gets a new image from the input (in our case, either a webcam feed or a screenshot of what we're seeing in our screen) and displays it with the bounding box detections in place:
59
+
Now, we're ready to get started. We create our main loop, which constantly gets a new image from the input source (either a webcam feed or a screenshot of what we're seeing in our screen) and displays it with the bounding box detections in place:
63
60
64
-
61
+
65
62
66
63
The most important function in that code is the `infer()` function, which returns an image and the result object, transformed into a pandas object for your convenience.
67
64
68
-
65
+
69
66
70
67
In this first part of the function, we obtain a new image from the OpenCV video capture object. You can also find a similar implementation but by taking screenshots, instead of using the webcam feed in [this file](https://github.com/oracle-devrel/devo.publishing.other/custom_pytorch_yolov5/files/lightweight_screen_torch_inference.py)
71
68
72
69
Now, we need to pass this to our Torch model, which will return bounding box results. However, there's an important consideration to make: since we want our predictions to be as fast as possible, and we know that the mask detection model has been trained with thousands of images of all shapes and sizes, we can consider a technique which will benefit Frames Per Second (FPS) on our program: **rescaling** images into a lower resolution (since my webcam feed had 1080p resolution, and I use a 2560x1440 monitor which causes screenshots to be too detailed, more than I need).
73
70
For this, I chose a `SCALE_FACTOR` variable to hold this value (between 0-1). Currently, all images will be downscaled to 640 pixels in width and the respective resolution in height, to maintain the original image's aspect ratio.
74
71
75
-
72
+
76
73
77
74
Now that we have our downscaled image, we pass it to the model, and it returns the object we wanted:
78
75
79
-
80
-
> **Note**: the `size=640` option tells the model we're going to pass it images with that width, so the model will predict results from that width too.
76
+
77
+
> **Note**: the `size=640` option tells the model we're going to pass it images with that width, so the model will predict results of those dimensions.
81
78
82
-
The last thing we do is draw the bounding boxes from
83
-
84
-
79
+
The last thing we do is draw the bounding boxes that we obtained into the image, and return the image to display it later.
85
80
81
+
86
82
87
83
## 1. Sorting Detections
88
84
89
85
This first technique is the simplest, and can be useful to add value to the standard YOLO functionality in an unique way. The idea is to quickly manipulate the PyTorch-pandas object to sort values according to one of the columns.
90
86
91
-
For this, I suggest two ideas: sorting by confidence score, or by detection coordinates. To illustrate how any of these techniques are useful, let's think of the following image:
87
+
For this, I suggest two ideas: sorting by confidence score, or by detection coordinates. To illustrate how any of these techniques are useful, let's look at the following image:
92
88
93
89
94
-
> **Note**: this image was produced by [Muhammad Moin](https://www.linkedin.com/in/muhammad-moin-7776751a0/) and illustrates how sorting detections can be useful.
90
+
> **Note**: this image illustrates how sorting detections can be useful. [(image credits)](https://www.linkedin.com/in/muhammad-moin-7776751a0/)
91
+
92
+
In the image above, an imaginary line is drawn between both sides of the roadway, in this case **horizontally**. Any object passing from one equator to the other in a specific direction is counted as an "inward" or "downward" vehicle. This can be achieved by specifying (x,y) bounds, and any item in the PyTorch-pandas object that surpasses it in any direction is detected.
95
93
96
-
In the image above, an imaginary line is drawn between both sides of the roadway, in this case _horizontally_. Any object passing from one equator to the other in a specific direction is counted as an "inward" or "downward" vehicle. This can be achieved by specifying (x,y) bounds, and any item in the PyTorch-pandas object that surpasses it in any direction is detected.
97
94
For processing purposes, sorting these values from the lowest y coordinate to the highest will return all cars in-order, from top to bottom of the image, which facilitates their processing in an ordered manner.
98
95
99
96
### 2. Cropping & Saving Detections
@@ -105,26 +102,26 @@ It's useful because we can manipulate or use the cropped image of an object, ins
105
102
2. Give this cropped image to an Optical Character Recognition (OCR) program
106
103
3. Extract the text in real-time
107
104
108
-
109
-
> **Note**: this is an example of a car's license plate being passed through an OCR in Keras. Credits to [Theophilebuyssens](https://medium.com/@theophilebuyssens).
105
+
106
+
> **Note**: this is an example of a car's license plate being passed through an OCR in Keras. [(image credits)](https://medium.com/@theophilebuyssens)
110
107
111
-
This approach wouldn't work if we gave the whole image to the OCR, as it wouldn't be able to distinguish small texts, unless the image's resolution was very high (which in most cases isn't the case, especially in surveillance cameras).
108
+
This approach wouldn't work if we gave the whole image to the OCR, as it wouldn't be able to confidently recognize such small texts (that represent only a fraction of the screen), unless the image's resolution was very high (which in most cases isn't the case, just like in surveillance cameras).
112
109
113
110
To implement this, we will base everything we do on **bounding boxes**. Our PyTorch code will return an object with bounding box coordinates for detected objects (and the detection's confidence scores), and we will use this object to create newly cropped images with the bounding box sizes.
114
-
> **Note**: you can always modify the range of pixels you want to crop in each image, by being either more permissive (getting extra pixels around the bounding box) or more restrictiv, removing the edges of the detected object.
111
+
> **Note**: you can always modify the range of pixels you want to crop in each image, by being either more **permissive** (getting extra pixels around the bounding box) or more **restrictive**, removing the edges of the detected object.
115
112
116
-
An important consideration is that, since we're passing images to our model with a width of 640 pixels, we need to keep our previously-mentioned `SCALE_FACTOR` variable. The problem is that the original image has a bigger size than the downscaled image (the one we pass the model), so bounding box detections will also be downscaled. We need to multiply these detections by the scale factor in order to _draw_ these bounding boxes over the original image; and then display it:
113
+
An important consideration is that, since we're passing images to our model with a width of 640 pixels, we need to keep our previously-mentioned `SCALE_FACTOR` variable. The problem is that the original image has a higher size than the downscaled image (the one we pass the model), so bounding box detection coordinates will also be downscaled. We need to multiply these detections by the scale factor in order to _draw_ these bounding boxes over the original image; and then display it:
117
114
118
-
115
+
119
116
120
117
We use the `save_cropped_images()` function to save images, while also accounting for the frequency parameter we set: we'll only save the cropped detections in case the frame is one we're supposed to save.
121
118
Inside this function, we will **upscale** bounding box detections. Also, we'll only save the image if the detected image is higher than (x,y) width and height:
Last thing we do is save the cropped image with OpenCV:
126
123
127
-
124
+
128
125
129
126
And we successfully implemented the functionality.
130
127
@@ -136,16 +133,23 @@ This last technique we're going to learn about is very straightforward and easy
136
133
137
134
Depending on the problem, you may want to choose one of these two options. In our case, we'll implement the second option:
138
135
139
-
136
+
140
137
> **Note**: to implement the first option, you just need to *increment* the variable every time, instead of setting it. However, you might benefit from looking at implementations like [DeepSORT](https://github.com/ZQPei/deep_sort_pytorch) or [Zero-Shot Tracking](https://github.com/roboflow/zero-shot-object-tracking), which is able to recognize the same object/detection from sequential frames, and only count them as one; not separate entities.
141
138
142
139
With our newly-created global variable, we'll hold a value of our liking. For example, in the code above, I'm detecting the _`mask`_ class. Then, I just need to draw the number of detected objects with OpenCV, along with the bounding boxes on top of the original image:
143
140
144
-
141
+
142
+
143
+
As an example, I created this GIF.
144
+
145
+
146
+
147
+
148
+
Note that I tested this on my own computer with an RTX 3080, I got about 25 FPS **with no downscaling** of images (I used their standard resolutions of 2560x1440). If you use downscaling you'll notice huge improvements in FPS.
145
149
146
150
## Conclusions
147
151
148
-
I've shown three additional features not currently present in YOLO models, by just adding a Python layer to it. PyTorch's Model Hub made this possible, as well as RoboFlow (made creating and exporting the mask detection model easy).
152
+
I've shown three additional features not currently present in YOLO models, by just adding a Python layer to it. PyTorch's Model Hub ultimately made this possible, as well as RoboFlow (made creating and exporting the mask detection model easy).
149
153
150
154
In the future, I'm planning on releasing an implementation with either DeepSORT or Zero-Shot Tracking, together with YOLO, to track objects. If you'd like to see any additional use cases or features implemented, let me know in the comments!
0 commit comments