diff --git a/implementation.typ b/implementation.typ index 2e0742d..670f668 100644 --- a/implementation.typ +++ b/implementation.typ @@ -1,6 +1,7 @@ #import "@preview/fletcher:0.5.3" as fletcher: diagram, node, edge #import fletcher.shapes: rect, diamond #import "utils.typ": todo +#import "@preview/subpar:0.1.1" = Implementation The three methods described (ResNet50, CAML, P>M>F) were implemented in a Jupyter notebook and compared to each other. @@ -8,16 +9,19 @@ The three methods described (ResNet50, CAML, P>M>F) were implemented in a Jupyte == Experiments For all of the three methods we test the following use-cases:#todo[maybe write more to each test] - Detection of anomaly class (1,3,5 shots) -- Inbalanced target class prediction (5,10,15,30 good shots, 5 bad shots) - 2 Way classification (1,3,5 shots) -- Inbalanced 2 Way classification (5,10,15,30 good shots, 5 bad shots) - Detect only anomaly classes (1,3,5 shots) +- Inbalanced 2 Way classification (5,10,15,30 good shots, 5 bad shots) +- Inbalanced target class prediction (5,10,15,30 good shots, 5 bad shots) Those experiments were conducted on the MVTEC AD dataset on the bottle and cable classes. == Experiment Setup -#todo[Setup of experiments, which classes used, nr of samples] +All the experiments were done on the bottle and cable classes of the MVTEC AD dataset. +The correspoinding number of shots were randomly selected from the dataset. +The rest of the images were used to test the model and measure the accuracy. +#todo[Maybe add real number of samples per classes] == ResNet50 === Approach @@ -83,13 +87,78 @@ After creating the embeddings for the support and query set the euclidean distan The class with the smallest distance is chosen as the predicted class. === Results -This method perofrmed better than expected wich such a simple method. +This method performed better than expected wich such a simple method. +As in @resnet50bottleperfa with a normal 5 shot / 4 way classification the model achieved an accuracy of 75%. +When detecting only if there occured an anomaly or not the performance is significantly better and peaks at 81% with 5 shots / 2 ways. +Interestintly the model performed slightly better with fewer shots in this case. +Moreover in @resnet50bottleperfa, the detection of the anomaly class only (3 way) shows a similar pattern as the normal 4 way classification. +The more shots the better the performance and it peaks at around 88% accuracy with 5 shots. -#todo[Add images of graphs with ResNet50 stuff only] +In @resnet50bottleperfb the model was tested with inbalanced class distributions. +With [5,10,15,30] good shots and 5 bad shots the model performed worse than with balanced classes. +The more good shots the worse the performance. +The only exception is the faulty or not detection (2 way) where the model peaked at 15 good shots with 83% accuracy. + +#subpar.grid( + figure(image("rsc/resnet/ResNet50-bottle.png"), caption: [ + Normal [1,3,5] shots + ]), , + figure(image("rsc/resnet/ResNet50-bottle-inbalanced.png"), caption: [ + Inbalanced [5,10,15,30] shots + ]), , + columns: (1fr, 1fr), + caption: [ResNet50 performance on bottle class], + label: , +) + +The same experiments were conducted on the cable class and the results are shown in @resnet50cableperfa and @resnet50cableperfb. +The results are very similar to the bottle class. +Generally the more shots the better the accuracy. +But the overall reached max accuracy is lower than on the bottle class, +but this is expected as the cable class consists of 8 faulty classes. + +#subpar.grid( + figure(image("rsc/resnet/ResNet50-cable.png"), caption: [ + Normal [1,3,5] shots + ]), , + figure(image("rsc/resnet/ResNet50-cable-inbalanced.png"), caption: [ + Inbalanced [5,10,15,30] shots + ]), , + columns: (1fr, 1fr), + caption: [ResNet50 performance on cable class], + label: , +) == P>M>F === Approach === Results +The results of P>M>F look very promising and improve by a large margin over the ResNet50 method. +In @pmfbottleperfa the model reached an accuracy of 79% with 5 shots / 4 way classification. +#todo[write bit more here] + +#subpar.grid( + figure(image("rsc/pmf/P>M>F-bottle.png"), caption: [ + Normal [1,3,5] shots + ]), , + figure(image("rsc/pmf/P>M>F-bottle-inbalanced.png"), caption: [ + Inbalanced [5,10,15,30] shots + ]), , + columns: (1fr, 1fr), + caption: [P>M>F performance on bottle class], + label: , +) + +#subpar.grid( + figure(image("rsc/pmf/P>M>F-cable.png"), caption: [ + Normal [1,3,5] shots + ]), , + figure(image("rsc/pmf/P>M>F-cable-inbalanced.png"), caption: [ + Inbalanced [5,10,15,30] shots + ]), , + columns: (1fr, 1fr), + caption: [P>M>F performance on cable class], + label: , +) == CAML === Approach @@ -106,4 +175,18 @@ This might be caused by the fact that the model was not fine-tuned for any indus The model was trained on a large number of general purpose images and is not fine-tuned at all. It might not handle very similar images well. -#todo[Add images of graphs with CAML stuff only] +Compared the the other two methods CAML performed poorly in almost all experiments. +The normal few-shot classification reached only 40% accuracy in @camlperfa at best. +The only test it did surprisingly well was the detection of the anomaly class for the cable class in @camlperfb were it reached almost 60% accuracy. + +#subpar.grid( + figure(image("rsc/caml/CAML-bottle.png"), caption: [ + Normal [1,3,5] shots - Bottle + ]), , + figure(image("rsc/caml/CAML-cable.png"), caption: [ + Normal [1,3,5] shots - Cable + ]), , + columns: (1fr, 1fr), + caption: [CAML performance], + label: , +) diff --git a/rsc/caml/CAML-bottle.png b/rsc/caml/CAML-bottle.png new file mode 100644 index 0000000..2e3ac14 Binary files /dev/null and b/rsc/caml/CAML-bottle.png differ diff --git a/rsc/caml/CAML-cable.png b/rsc/caml/CAML-cable.png new file mode 100644 index 0000000..5358191 Binary files /dev/null and b/rsc/caml/CAML-cable.png differ diff --git a/rsc/pmf/P>M>F-bottle-inbalanced.png b/rsc/pmf/P>M>F-bottle-inbalanced.png new file mode 100644 index 0000000..0fd4e9f Binary files /dev/null and b/rsc/pmf/P>M>F-bottle-inbalanced.png differ diff --git a/rsc/pmf/P>M>F-bottle.png b/rsc/pmf/P>M>F-bottle.png new file mode 100644 index 0000000..d938c87 Binary files /dev/null and b/rsc/pmf/P>M>F-bottle.png differ diff --git a/rsc/pmf/P>M>F-cable-inbalanced.png b/rsc/pmf/P>M>F-cable-inbalanced.png new file mode 100644 index 0000000..fa60b3d Binary files /dev/null and b/rsc/pmf/P>M>F-cable-inbalanced.png differ diff --git a/rsc/pmf/P>M>F-cable.png b/rsc/pmf/P>M>F-cable.png new file mode 100644 index 0000000..8904efc Binary files /dev/null and b/rsc/pmf/P>M>F-cable.png differ diff --git a/rsc/resnet/ResNet50-bottle-inbalanced.png b/rsc/resnet/ResNet50-bottle-inbalanced.png new file mode 100644 index 0000000..9a6033e Binary files /dev/null and b/rsc/resnet/ResNet50-bottle-inbalanced.png differ diff --git a/rsc/resnet/ResNet50-bottle.png b/rsc/resnet/ResNet50-bottle.png new file mode 100644 index 0000000..5679028 Binary files /dev/null and b/rsc/resnet/ResNet50-bottle.png differ diff --git a/rsc/resnet/ResNet50-cable-inbalanced.png b/rsc/resnet/ResNet50-cable-inbalanced.png new file mode 100644 index 0000000..595ca4e Binary files /dev/null and b/rsc/resnet/ResNet50-cable-inbalanced.png differ diff --git a/rsc/resnet/ResNet50-cable.png b/rsc/resnet/ResNet50-cable.png new file mode 100644 index 0000000..18ea1fd Binary files /dev/null and b/rsc/resnet/ResNet50-cable.png differ