PWAI/src/experimentalresults.tex

\section{Experimental Results}
\subsection{Does Active-Learning benefit the learning process?}

With the test setup described in~\ref{sec:implementation} a test series was performed.
Several different batch sizes $\mathcal{B} = \left\{ 2,4,6,8 \right\}$ and sample sizes $\mathcal{S} = \left\{ 2\mathcal{B}_i,4\mathcal{B}_i,5\mathcal{B}_i,10\mathcal{B}_i \right\}$
dependent on the selected batch size were selected.
We define the baseline (passive learning) AUC curve as the supervised learning process without any active learning.
The following graphs are only a subselection of the test series which give the most insights.

\begin{figure}
    \label{fig:auc_normal_lowcer_2_10}
    \centering
    \hspace*{-0.1\linewidth}\includegraphics[width=1.2\linewidth]{../rsc/AUC_normal_lowcer_2_10}
    \caption{AUC with $\mathcal{B} = 2$ and $\mathcal{S}=10$}
\end{figure}

\begin{figure}
    \label{fig:auc_normal_lowcer_2_20}
    \centering
    \hspace*{-0.1\linewidth}\includegraphics[width=1.2\linewidth]{../rsc/AUC_normal_lowcer_2_20}
    \caption{AUC with $\mathcal{B} = 2$ and $\mathcal{S}=20$}
\end{figure}

\begin{figure}
    \label{fig:auc_normal_lowcer_2_50}
    \centering
    \hspace*{-0.1\linewidth}\includegraphics[width=1.2\linewidth]{../rsc/AUC_normal_lowcer_2_50}
    \caption{AUC with $\mathcal{B} = 2$ and $\mathcal{S}=50$}
\end{figure}

\begin{figure}
    \label{fig:auc_normal_lowcer_4_16}
    \centering
    \hspace*{-0.1\linewidth}\includegraphics[width=1.2\linewidth]{../rsc/AUC_normal_lowcer_4_16}
    \caption{AUC with $\mathcal{B} = 4$ and $\mathcal{S}=16$}
\end{figure}

\begin{figure}
    \label{fig:auc_normal_lowcer_4_24}
    \centering
    \hspace*{-0.1\linewidth}\includegraphics[width=1.2\linewidth]{../rsc/AUC_normal_lowcer_4_24}
    \caption{AUC with $\mathcal{B} = 4$ and $\mathcal{S}=24$}
\end{figure}

\begin{figure}
    \label{fig:auc_normal_lowcer_8_16}
    \centering
    \hspace*{-0.1\linewidth}\includegraphics[width=1.2\linewidth]{../rsc/AUC_normal_lowcer_8_16}
    \caption{AUC with $\mathcal{B} = 8$ and $\mathcal{S}=16$}
\end{figure}

\begin{figure}
    \label{fig:auc_normal_lowcer_8_32}
    \centering
    \hspace*{-0.1\linewidth}\includegraphics[width=1.2\linewidth]{../rsc/AUC_normal_lowcer_8_32}
    \caption{AUC with $\mathcal{B} = 8$ and $\mathcal{S}=32$}
\end{figure}

Generally a pattern can be seen: The lower the batch size the more benefits are gained by active learning.
This may be caused by the fast model convergence.
The lower the batch size the more pre-prediction decision points are required.
This helps directing the learning with better samples of the selected metric.
When the batch size is higher the model already converges to a good AUC value before the same amount of pre-predictions is reached.

Moreover, when increasing the sample-space $\mathcal{S}$ from where the pre-predictions are drawn generally the performance improves.
This is because the selected subset $\pmb{x} \sim \mathcal{X}_U$ has a higher chance of containing relevant elements corresponding to the selected metric.
But keep in mind this improvement comes with a performance penalty because more model evaluations are required to predict the ranking scores.

% todo
\ref{fig:auc_normal_lowcer_2_10} shows the AUC curve with a batch size of 2 and a sample size of 10.
Todo add some references to the graphs.


\subsection{Is Dagster and Label-Studio a proper tooling to build an AL
Loop?}\label{subsec:is-dagster-and-label-studio-a-proper-tooling-to-build-an-al
loop?}

The combination of Dagster and Label-Studio is a good choice for building an active-learning loop.

\subsection{Does balancing the learning samples improve performance?}\label{subsec:does-balancing-the-learning-samples-improve-performance?}

Not really.