> ## Documentation Index > Fetch the complete documentation index at: https://aegean.ai/llms.txt > Use this file to discover all available pages before exploring further. # The Unsupervised Approach > Anomaly detection using unsupervised learning methods with pretrained models ## Problem Statement Assume that we have data (images) $\mathbf{x}$ *with* or *without* explicit knowledge of their labels $y$. We want to learn a model that can create a representation of $\mathbf{x}$ such that it can assign an image that was never seen before, to either the *nominal* category or the *anomalous* category ($k=1$), correctly with high probability. This setting is known as *Anomaly Detection* (AD). In the AD literature we meet a category of problems that is associated with the *localization* of the anomaly. Localization is achieved by producing an image with a bounding box around the anomalous area. This problem is not in scope for this project that aims to produce a classification of the image as *nominal* or *anomalous*. Having said that, some of the models we have evaluated have inherent localization capabilities that can be used in the future. We start by establishing some definitions and mathematical notation so that we understand the developed modeling approach. We need to extract a vector $\mathbf{z}$ from the image $\mathbf{x}$ that can be used to subsequently assign to it a score that will flag the AD event. The vector $\mathbf{z}$ is called the *representation* of the input image $\mathbf{x}$. A synonymous way of referring to the representation is *embedding* where the image is *embedded* into a vector space with dimensions smaller (e.g., $< 2048$) than the raw image space. In the Seamagine data, the raw image vector space for the grayscale cropped images is $1 \times 224 \times 224 = 50,176$ dimensions for one of the channels. We also refer to representations as *features* since they represent the *features* of the image needed for the task at hand. [Qdrant](https://qdrant.tech/) is used for storing such embeddings. ## Unsupervised Anomaly Detection When we have a fully or partially annotated dataset we have a *supervised* or *semi-supervised* classification problem respectively. The Seamagine dataset is a collection of images taken at different times from a production machine grouped in $M=12$ machine settings. Because of the grouped collection, it may appear on the first glance as a *supervised* dataset but it is *not*. The main reason is that multiple machine settings are used to only *vary* the data distribution $p_{data}(\mathbf{x})$ such that *some* products are to fail when a physical stress test is performed. Despite that we have no direct mapping from a machine setting to a stress test outcome for all the images in our dataset, which is needed if one can claim supervision for our AD problem, we have seen some evidence that some machine settings lead to a higher number of failures than others. This is the only information we can use and we use it to only *benchmark* our models using industry-standard AD evaluation protocols. Since we do not use labels during training we have an *unsupervised* problem that is also known as *one-class* (OC) classification or out-of-distribution (OOD) detection. Let $\mathcal{X}_T$ denote the set of all nominal images ($\forall x_i \in \mathcal{X}_T : y_i = 0$) available at training time, with $y_i \in \{0, 1\}$ denoting if an image $x_i$ is nominal ($y_i=0$) or anomalous ($y_i=1$). Accordingly, we define $\mathcal{X}_V$ to be the set of samples provided at test time, with $\forall x_i \in \mathcal{X}_V : y_i \in \{0, 1\}$. Since we need the labels only for evaluation / benchmarking purposes, we can use the labels in the validation dataset to evaluate the performance of our models. **It is important to emphasize the unsupervised problem setting - to develop a model we do not use any anomaly detection labels (`PASS` or `FAIL`) associated with the images, which is in direct contrast to a supervised classification setting.** **The images with anomaly detection label equal to `FAIL` are present only in the validation dataset during testing and should not be present in the training dataset. This means that our training dataset contains 11 machine configurations while the validation dataset contains 12 machine configurations.** Most unsupervised AD methods attempt to create a model of the underlying *nominal* data distribution $p_{nom}(\mathbf{x})$ - as a direct consequence of having a single class present in our dataset. We call such model $p_{model}(\mathbf{x})$ and we aim to make it as close as possible to the true nominal data distribution. We distinguish two parts in the construction of the model: 1. The embeddings / features extractor that produces the representation $\mathbf{z}$ from the image $\mathbf{x}$. 2. The distribution estimator that learns the distribution $p_{model}(\mathbf{x})$. ### Embeddings With respect to the embeddings we can broadly divide the ways that we obtain them into two categories: *generative* and *discriminative*. In the following we outline the relative merits of the methods and attempt to bet on the most promising one that solves the *real-time* unsupervised AD problem we are facing. 1. **Reconstruction-based**: One of the main generative approaches is the set of models that are reconstruction based. Here we learn to reconstruct the *domain-specific* nominal input images and use the reconstruction error as a score to determine if the image is anomalous or not. During training only nominal samples are seen by the network which means that when an anomalous image is seen during testing, the network is unable to reconstruct it with high quality lifting the reconstruction error above a threshold. Reconstruction based methods learn embedding from the domain-specific dataset itself and require an explicit training stage before the encoder can be used in production. Two main representatives in this category include Convolutional AutoEncoders (CAE) and Generative Adversarial Networks (GANs). Advances in the form of Masked Autoencoders (MAE) have been found to offer foundational modeling but are computationally heavy especially during inference. 2. **Representation-based**: One of the main discriminative approaches is the set of models that produce the embeddings using a *pretrained* model - a vision transformer or a (residual) CNN. Such models are pretrained using a discriminative dataset such as ImageNet. There are a couple of promising approaches that we decided to focus on based on the excellent evaluation results of the MVTec dataset in combination with the distribution estimation methods. The main representatives in this category include OpenCLIP and ResNet-50. We focus on the representation learning category since it is the most widely used in the industry and it is the most interpretable. ### Distribution Estimation 1. **Patchcore**: A method that uses a patch-based approach to learn the distribution of the dataset. The method is based on the idea that the distribution of the patches of the nominal images is different from the distribution of the patches of the anomalous images. The method uses a pretrained model to extract the features from the images and then applies a nearest-neighbor algorithm to detect anomalies at the patch level. It relies on representative features that come from the mid sections of pre-trained backbone CNN and uses coreset sampling to reduce memory requirements. 2. **Efficient-AD**: Introduces a fast patch descriptor and trains a student network to predict the features computed by a pretrained teacher network on nominal training images. Because the student is not trained on anomalous images, it generally fails to mimic the teacher on these. A large distance between the outputs of the teacher and the student thus enables the detection of anomalies at test time. ## Anomaly Detection Scoring Irrespective of the approach used and unless the method inherently scores the anomaly, a Nearest neighbors algorithm (kNN) is employed to assign an anomaly score to the validation image and a thresholding operation determines if the image is anomalous or not. ## Evaluation Protocol For the anomaly detection task we have adopted the industry standard evaluation protocol and metrics. During training we use only the `PASS` class images while during the validation we use both `PASS` and `FAIL` class images and report key performance metrics: the Precision-Recall curve, the associated Area Under the Curve (AUC) score and also the validation confusion matrix. *** [Edit this page on GitHub](https://github.com/aegean-ai/eaia/edit/main/src/products/applications/anomaly-detection/manufacturing/unsupervised-learning-models/index.mdx) or [file an issue](https://github.com/aegean-ai/eaia/issues/new/choose).