\section{Experimental Results}\label{sec:experimental-results}

\subsection{Does Active-Learning benefit the learning process?}\label{subsec:does-active-learning-benefit-the-learning-process?}

A test series was performed inside a Jupyter notebook.
The active learning loop starts with a untrained RESNet-18 model and a random selection of samples.
The muffin and chihuahua dataset was used for this binary classification task.
The dataset is split into training and test set which contains $\sim4750$ train- and $\sim1250$ test-images.\cite{muffinsvschiuahuakaggle}
(see subsection~\ref{subsubsec:muffinvschihuahua} for more infos)

As a loss function CrossEntropyLoss was used and the Adam optimizer with a learning rate of $0.0001$.

$\mathcal{B}$ samples are selected from the $\mathcal{S}$ samples and labeled by an oracle.
Here the oracle is just labeling the samples with the correct class because the dataset is synthetic and the labels are known.
No real human annotator was used because of huge time consumption and the goal is to benchmark the active learning process itself.
Afterwards the model is trained with this labeled samples and the loop starts again with predicting $\mathcal{B}$ samples from the $\mathcal{S}$ drawn samples.

Several different batch sizes $\mathcal{B} = \left\{ 2,4,6,8 \right\}$ and sample sizes $\mathcal{S} = \left\{ 2\mathcal{B}_i,4\mathcal{B}_i,5\mathcal{B}_i,10\mathcal{B}_i \right\}$
dependent on the selected batch size were used.
We define the baseline (passive learning) AUC curve as the supervised learning process without any active learning.
The following graphs are only a subselection of the test series which give the most insights.

\begin{figure}
    \centering
    \hspace*{-0.1\linewidth}\includegraphics[width=1.2\linewidth]{../rsc/AUC_normal_lowcer_2_10}
    \caption{AUC with $\mathcal{B} = 2$ and $\mathcal{S}=10$}
    \label{fig:auc_normal_lowcer_2_10}
\end{figure}

\begin{figure}
    \centering
    \hspace*{-0.1\linewidth}\includegraphics[width=1.2\linewidth]{../rsc/AUC_normal_lowcer_2_20}
    \caption{AUC with $\mathcal{B} = 2$ and $\mathcal{S}=20$}
    \label{fig:auc_normal_lowcer_2_20}
\end{figure}

\begin{figure}
    \centering
    \hspace*{-0.1\linewidth}\includegraphics[width=1.2\linewidth]{../rsc/AUC_normal_lowcer_2_50}
    \caption{AUC with $\mathcal{B} = 2$ and $\mathcal{S}=50$}
    \label{fig:auc_normal_lowcer_2_50}
\end{figure}

\begin{figure}
    \centering
    \hspace*{-0.1\linewidth}\includegraphics[width=1.2\linewidth]{../rsc/AUC_normal_lowcer_4_16}
    \caption{AUC with $\mathcal{B} = 4$ and $\mathcal{S}=16$}
    \label{fig:auc_normal_lowcer_4_16}
\end{figure}

\begin{figure}
    \centering
    \hspace*{-0.1\linewidth}\includegraphics[width=1.2\linewidth]{../rsc/AUC_normal_lowcer_4_24}
    \caption{AUC with $\mathcal{B} = 4$ and $\mathcal{S}=24$}
    \label{fig:auc_normal_lowcer_4_24}
\end{figure}

\begin{figure}
    \centering
    \hspace*{-0.1\linewidth}\includegraphics[width=1.2\linewidth]{../rsc/AUC_normal_lowcer_8_16}
    \caption{AUC with $\mathcal{B} = 8$ and $\mathcal{S}=16$}
    \label{fig:auc_normal_lowcer_8_16}
\end{figure}

\begin{figure}
    \centering
    \hspace*{-0.1\linewidth}\includegraphics[width=1.2\linewidth]{../rsc/AUC_normal_lowcer_8_32}
    \caption{AUC with $\mathcal{B} = 8$ and $\mathcal{S}=32$}
    \label{fig:auc_normal_lowcer_8_32}
\end{figure}

Generally a pattern can be seen: The lower the batch size $\mathcal{B}$ the more benefits are gained by active learning.
This may be caused by the fast model convergence.
The lower $\mathcal{B}$ the more pre-prediction decision points are required.
This helps directing the learning with better samples of the selected metric.
When the batch size is higher the model already converges to a good AUC value before the same amount of pre-predictions is reached.

Moreover, when increasing the sample-space $\mathcal{S}$ from where the pre-predictions are drawn generally the performance improves.
This is because the selected subset $\pmb{x} \sim \mathcal{X}_U$ has a higher chance of containing relevant elements corresponding to the selected metric.
But keep in mind this improvement comes with a performance penalty because more model evaluations are required to predict the ranking scores.

Figures~\ref{fig:auc_normal_lowcer_2_10};\ref{fig:auc_normal_lowcer_2_20};\ref{fig:auc_normal_lowcer_2_50} show the AUC curve with a batch size of 2 and a sample size of 10, 20, 50 respectively.
On all three graphs the active learning curve outperforms the passive learning curve in all four scenarios.
Generally the higher the sample space $\mathcal{S}$ the better the performance.

Figures~\ref{fig:auc_normal_lowcer_4_16};\ref{fig:auc_normal_lowcer_4_24} show the AUC curve with a batch size of 4 and a sample size of 16, 24 respectively.
The performance is already much worse compared to the results from above with a batch size of 2.
Only the low certainty first approach outperforms the passive learning in both cases.
The other methods are as good or worse than the passive learning curve.

Figures~\ref{fig:auc_normal_lowcer_8_16};\ref{fig:auc_normal_lowcer_8_32} show the AUC curve with a batch size of 8 and a sample size of 16, 32 respectively.
The performance is even worse compared to the results from above with a batch size of 4.
This might be the case because the model already converges to a good AUC value before the same amount of pre-predictions is reached.

\subsection{Is Dagster and Label-Studio a proper tooling to build an AL
Loop?}\label{subsec:is-dagster-and-label-studio-a-proper-tooling-to-build-an-al
loop?}

The combination of Dagster and Label-Studio is a good choice for building an active-learning loop.
Dagster provides a clean way to build pipelines and to keep track of the data in the Web UI\@.
Label-Studio provides a great api which can be used to update the predictions of the model from the dagster pipeline.

Overall this option should just be chosen if the solution needs to be scalable and deployed in the cloud.
For smaller projects a simpler solution just in a notebook or as a simple python script might be more appropriate.

\subsection{Does balancing the learning samples improve performance?}\label{subsec:does-balancing-the-learning-samples-improve-performance?}

The previous process was improved by balancing the classes to give the oracle for labelling.
The idea is that it might happen that the low certainty samples might always be of one class and thus lead to an imbalanced learning process.
The sample selection was modified as described in paragraph~\ref{par:furtherimprovements}.

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{../rsc/AUC_balanced__4_24}
    \caption{Dagster asset graph}
    \label{fig:balancedauc}
\end{figure}

Unfortunately it didn't improve the convergence speed and it seems to make no difference compared to not balancing and seems mostly even worse.
This might be the case because the uncertainty sampling process balances the draws itself pretty well.
Figure~\ref{fig:balancedauc} shows the AUC curve with a batch size $\mathcal{B}=4$ and a sample size $\mathcal{S}=24$ for both, balanced and unbalanced low certainty sampling.
The result looks similar for the other batch sizes and sample sizes.