add result images for each method
All checks were successful
Build Typst document / build_typst_documents (push) Successful in 1m1s

This commit is contained in:
lukas-heiligenbrunner 2025-01-01 20:50:52 +01:00
parent 24118dce93
commit fe9f4433b3
11 changed files with 89 additions and 6 deletions

View File

@ -1,6 +1,7 @@
#import "@preview/fletcher:0.5.3" as fletcher: diagram, node, edge
#import fletcher.shapes: rect, diamond
#import "utils.typ": todo
#import "@preview/subpar:0.1.1"
= Implementation
The three methods described (ResNet50, CAML, P>M>F) were implemented in a Jupyter notebook and compared to each other.
@ -8,16 +9,19 @@ The three methods described (ResNet50, CAML, P>M>F) were implemented in a Jupyte
== Experiments
For all of the three methods we test the following use-cases:#todo[maybe write more to each test]
- Detection of anomaly class (1,3,5 shots)
- Inbalanced target class prediction (5,10,15,30 good shots, 5 bad shots)
- 2 Way classification (1,3,5 shots)
- Inbalanced 2 Way classification (5,10,15,30 good shots, 5 bad shots)
- Detect only anomaly classes (1,3,5 shots)
- Inbalanced 2 Way classification (5,10,15,30 good shots, 5 bad shots)
- Inbalanced target class prediction (5,10,15,30 good shots, 5 bad shots)
Those experiments were conducted on the MVTEC AD dataset on the bottle and cable classes.
== Experiment Setup
#todo[Setup of experiments, which classes used, nr of samples]
All the experiments were done on the bottle and cable classes of the MVTEC AD dataset.
The correspoinding number of shots were randomly selected from the dataset.
The rest of the images were used to test the model and measure the accuracy.
#todo[Maybe add real number of samples per classes]
== ResNet50
=== Approach
@ -83,13 +87,78 @@ After creating the embeddings for the support and query set the euclidean distan
The class with the smallest distance is chosen as the predicted class.
=== Results
This method perofrmed better than expected wich such a simple method.
This method performed better than expected wich such a simple method.
As in @resnet50bottleperfa with a normal 5 shot / 4 way classification the model achieved an accuracy of 75%.
When detecting only if there occured an anomaly or not the performance is significantly better and peaks at 81% with 5 shots / 2 ways.
Interestintly the model performed slightly better with fewer shots in this case.
Moreover in @resnet50bottleperfa, the detection of the anomaly class only (3 way) shows a similar pattern as the normal 4 way classification.
The more shots the better the performance and it peaks at around 88% accuracy with 5 shots.
#todo[Add images of graphs with ResNet50 stuff only]
In @resnet50bottleperfb the model was tested with inbalanced class distributions.
With [5,10,15,30] good shots and 5 bad shots the model performed worse than with balanced classes.
The more good shots the worse the performance.
The only exception is the faulty or not detection (2 way) where the model peaked at 15 good shots with 83% accuracy.
#subpar.grid(
figure(image("rsc/resnet/ResNet50-bottle.png"), caption: [
Normal [1,3,5] shots
]), <resnet50bottleperfa>,
figure(image("rsc/resnet/ResNet50-bottle-inbalanced.png"), caption: [
Inbalanced [5,10,15,30] shots
]), <resnet50bottleperfb>,
columns: (1fr, 1fr),
caption: [ResNet50 performance on bottle class],
label: <resnet50bottleperf>,
)
The same experiments were conducted on the cable class and the results are shown in @resnet50cableperfa and @resnet50cableperfb.
The results are very similar to the bottle class.
Generally the more shots the better the accuracy.
But the overall reached max accuracy is lower than on the bottle class,
but this is expected as the cable class consists of 8 faulty classes.
#subpar.grid(
figure(image("rsc/resnet/ResNet50-cable.png"), caption: [
Normal [1,3,5] shots
]), <resnet50cableperfa>,
figure(image("rsc/resnet/ResNet50-cable-inbalanced.png"), caption: [
Inbalanced [5,10,15,30] shots
]), <resnet50cableperfb>,
columns: (1fr, 1fr),
caption: [ResNet50 performance on cable class],
label: <resnet50cableperf>,
)
== P>M>F
=== Approach
=== Results
The results of P>M>F look very promising and improve by a large margin over the ResNet50 method.
In @pmfbottleperfa the model reached an accuracy of 79% with 5 shots / 4 way classification.
#todo[write bit more here]
#subpar.grid(
figure(image("rsc/pmf/P>M>F-bottle.png"), caption: [
Normal [1,3,5] shots
]), <pmfbottleperfa>,
figure(image("rsc/pmf/P>M>F-bottle-inbalanced.png"), caption: [
Inbalanced [5,10,15,30] shots
]), <pmfbottleperfb>,
columns: (1fr, 1fr),
caption: [P>M>F performance on bottle class],
label: <pmfbottleperf>,
)
#subpar.grid(
figure(image("rsc/pmf/P>M>F-cable.png"), caption: [
Normal [1,3,5] shots
]), <pmfcableperfa>,
figure(image("rsc/pmf/P>M>F-cable-inbalanced.png"), caption: [
Inbalanced [5,10,15,30] shots
]), <pmfcableperfb>,
columns: (1fr, 1fr),
caption: [P>M>F performance on cable class],
label: <pmfcableperf>,
)
== CAML
=== Approach
@ -106,4 +175,18 @@ This might be caused by the fact that the model was not fine-tuned for any indus
The model was trained on a large number of general purpose images and is not fine-tuned at all.
It might not handle very similar images well.
#todo[Add images of graphs with CAML stuff only]
Compared the the other two methods CAML performed poorly in almost all experiments.
The normal few-shot classification reached only 40% accuracy in @camlperfa at best.
The only test it did surprisingly well was the detection of the anomaly class for the cable class in @camlperfb were it reached almost 60% accuracy.
#subpar.grid(
figure(image("rsc/caml/CAML-bottle.png"), caption: [
Normal [1,3,5] shots - Bottle
]), <camlperfa>,
figure(image("rsc/caml/CAML-cable.png"), caption: [
Normal [1,3,5] shots - Cable
]), <camlperfb>,
columns: (1fr, 1fr),
caption: [CAML performance],
label: <camlperf>,
)

BIN
rsc/caml/CAML-bottle.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 32 KiB

BIN
rsc/caml/CAML-cable.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 36 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 34 KiB

BIN
rsc/pmf/P>M>F-bottle.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 38 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 31 KiB

BIN
rsc/pmf/P>M>F-cable.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 35 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 35 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 37 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 28 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 34 KiB