add result images for each method
All checks were successful
Build Typst document / build_typst_documents (push) Successful in 1m1s
@ -1,6 +1,7 @@
|
||||
#import "@preview/fletcher:0.5.3" as fletcher: diagram, node, edge
|
||||
#import fletcher.shapes: rect, diamond
|
||||
#import "utils.typ": todo
|
||||
#import "@preview/subpar:0.1.1"
|
||||
|
||||
= Implementation
|
||||
The three methods described (ResNet50, CAML, P>M>F) were implemented in a Jupyter notebook and compared to each other.
|
||||
@ -8,16 +9,19 @@ The three methods described (ResNet50, CAML, P>M>F) were implemented in a Jupyte
|
||||
== Experiments
|
||||
For all of the three methods we test the following use-cases:#todo[maybe write more to each test]
|
||||
- Detection of anomaly class (1,3,5 shots)
|
||||
- Inbalanced target class prediction (5,10,15,30 good shots, 5 bad shots)
|
||||
- 2 Way classification (1,3,5 shots)
|
||||
- Inbalanced 2 Way classification (5,10,15,30 good shots, 5 bad shots)
|
||||
- Detect only anomaly classes (1,3,5 shots)
|
||||
- Inbalanced 2 Way classification (5,10,15,30 good shots, 5 bad shots)
|
||||
- Inbalanced target class prediction (5,10,15,30 good shots, 5 bad shots)
|
||||
|
||||
Those experiments were conducted on the MVTEC AD dataset on the bottle and cable classes.
|
||||
|
||||
|
||||
== Experiment Setup
|
||||
#todo[Setup of experiments, which classes used, nr of samples]
|
||||
All the experiments were done on the bottle and cable classes of the MVTEC AD dataset.
|
||||
The correspoinding number of shots were randomly selected from the dataset.
|
||||
The rest of the images were used to test the model and measure the accuracy.
|
||||
#todo[Maybe add real number of samples per classes]
|
||||
|
||||
== ResNet50
|
||||
=== Approach
|
||||
@ -83,13 +87,78 @@ After creating the embeddings for the support and query set the euclidean distan
|
||||
The class with the smallest distance is chosen as the predicted class.
|
||||
|
||||
=== Results
|
||||
This method perofrmed better than expected wich such a simple method.
|
||||
This method performed better than expected wich such a simple method.
|
||||
As in @resnet50bottleperfa with a normal 5 shot / 4 way classification the model achieved an accuracy of 75%.
|
||||
When detecting only if there occured an anomaly or not the performance is significantly better and peaks at 81% with 5 shots / 2 ways.
|
||||
Interestintly the model performed slightly better with fewer shots in this case.
|
||||
Moreover in @resnet50bottleperfa, the detection of the anomaly class only (3 way) shows a similar pattern as the normal 4 way classification.
|
||||
The more shots the better the performance and it peaks at around 88% accuracy with 5 shots.
|
||||
|
||||
#todo[Add images of graphs with ResNet50 stuff only]
|
||||
In @resnet50bottleperfb the model was tested with inbalanced class distributions.
|
||||
With [5,10,15,30] good shots and 5 bad shots the model performed worse than with balanced classes.
|
||||
The more good shots the worse the performance.
|
||||
The only exception is the faulty or not detection (2 way) where the model peaked at 15 good shots with 83% accuracy.
|
||||
|
||||
#subpar.grid(
|
||||
figure(image("rsc/resnet/ResNet50-bottle.png"), caption: [
|
||||
Normal [1,3,5] shots
|
||||
]), <resnet50bottleperfa>,
|
||||
figure(image("rsc/resnet/ResNet50-bottle-inbalanced.png"), caption: [
|
||||
Inbalanced [5,10,15,30] shots
|
||||
]), <resnet50bottleperfb>,
|
||||
columns: (1fr, 1fr),
|
||||
caption: [ResNet50 performance on bottle class],
|
||||
label: <resnet50bottleperf>,
|
||||
)
|
||||
|
||||
The same experiments were conducted on the cable class and the results are shown in @resnet50cableperfa and @resnet50cableperfb.
|
||||
The results are very similar to the bottle class.
|
||||
Generally the more shots the better the accuracy.
|
||||
But the overall reached max accuracy is lower than on the bottle class,
|
||||
but this is expected as the cable class consists of 8 faulty classes.
|
||||
|
||||
#subpar.grid(
|
||||
figure(image("rsc/resnet/ResNet50-cable.png"), caption: [
|
||||
Normal [1,3,5] shots
|
||||
]), <resnet50cableperfa>,
|
||||
figure(image("rsc/resnet/ResNet50-cable-inbalanced.png"), caption: [
|
||||
Inbalanced [5,10,15,30] shots
|
||||
]), <resnet50cableperfb>,
|
||||
columns: (1fr, 1fr),
|
||||
caption: [ResNet50 performance on cable class],
|
||||
label: <resnet50cableperf>,
|
||||
)
|
||||
|
||||
== P>M>F
|
||||
=== Approach
|
||||
=== Results
|
||||
The results of P>M>F look very promising and improve by a large margin over the ResNet50 method.
|
||||
In @pmfbottleperfa the model reached an accuracy of 79% with 5 shots / 4 way classification.
|
||||
#todo[write bit more here]
|
||||
|
||||
#subpar.grid(
|
||||
figure(image("rsc/pmf/P>M>F-bottle.png"), caption: [
|
||||
Normal [1,3,5] shots
|
||||
]), <pmfbottleperfa>,
|
||||
figure(image("rsc/pmf/P>M>F-bottle-inbalanced.png"), caption: [
|
||||
Inbalanced [5,10,15,30] shots
|
||||
]), <pmfbottleperfb>,
|
||||
columns: (1fr, 1fr),
|
||||
caption: [P>M>F performance on bottle class],
|
||||
label: <pmfbottleperf>,
|
||||
)
|
||||
|
||||
#subpar.grid(
|
||||
figure(image("rsc/pmf/P>M>F-cable.png"), caption: [
|
||||
Normal [1,3,5] shots
|
||||
]), <pmfcableperfa>,
|
||||
figure(image("rsc/pmf/P>M>F-cable-inbalanced.png"), caption: [
|
||||
Inbalanced [5,10,15,30] shots
|
||||
]), <pmfcableperfb>,
|
||||
columns: (1fr, 1fr),
|
||||
caption: [P>M>F performance on cable class],
|
||||
label: <pmfcableperf>,
|
||||
)
|
||||
|
||||
== CAML
|
||||
=== Approach
|
||||
@ -106,4 +175,18 @@ This might be caused by the fact that the model was not fine-tuned for any indus
|
||||
The model was trained on a large number of general purpose images and is not fine-tuned at all.
|
||||
It might not handle very similar images well.
|
||||
|
||||
#todo[Add images of graphs with CAML stuff only]
|
||||
Compared the the other two methods CAML performed poorly in almost all experiments.
|
||||
The normal few-shot classification reached only 40% accuracy in @camlperfa at best.
|
||||
The only test it did surprisingly well was the detection of the anomaly class for the cable class in @camlperfb were it reached almost 60% accuracy.
|
||||
|
||||
#subpar.grid(
|
||||
figure(image("rsc/caml/CAML-bottle.png"), caption: [
|
||||
Normal [1,3,5] shots - Bottle
|
||||
]), <camlperfa>,
|
||||
figure(image("rsc/caml/CAML-cable.png"), caption: [
|
||||
Normal [1,3,5] shots - Cable
|
||||
]), <camlperfb>,
|
||||
columns: (1fr, 1fr),
|
||||
caption: [CAML performance],
|
||||
label: <camlperf>,
|
||||
)
|
||||
|
BIN
rsc/caml/CAML-bottle.png
Normal file
After Width: | Height: | Size: 32 KiB |
BIN
rsc/caml/CAML-cable.png
Normal file
After Width: | Height: | Size: 36 KiB |
BIN
rsc/pmf/P>M>F-bottle-inbalanced.png
Normal file
After Width: | Height: | Size: 34 KiB |
BIN
rsc/pmf/P>M>F-bottle.png
Normal file
After Width: | Height: | Size: 38 KiB |
BIN
rsc/pmf/P>M>F-cable-inbalanced.png
Normal file
After Width: | Height: | Size: 31 KiB |
BIN
rsc/pmf/P>M>F-cable.png
Normal file
After Width: | Height: | Size: 35 KiB |
BIN
rsc/resnet/ResNet50-bottle-inbalanced.png
Normal file
After Width: | Height: | Size: 35 KiB |
BIN
rsc/resnet/ResNet50-bottle.png
Normal file
After Width: | Height: | Size: 37 KiB |
BIN
rsc/resnet/ResNet50-cable-inbalanced.png
Normal file
After Width: | Height: | Size: 28 KiB |
BIN
rsc/resnet/ResNet50-cable.png
Normal file
After Width: | Height: | Size: 34 KiB |