This post is summary of the “Anomaly Detection : A Survey”. Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behavior. These non-conforming patterns are often referred to as anomalies, outliers, discordant observations, exceptions, aberrations, surprises, peculiarities or contaminants in different application domains.

Anomalies are patterns in data that do not conform to a well defined notion of normal behavior.
  • Interesting to analyze
  • Unwanted noise in the data also can be found in there.
  • Novelty detection which aims at detecting previously unobserved (emergent, novel) patterns in the data
Challenges for Anomaly Detection
  • Drawing the boundary between normal and anomalous behavior
  • Availability of labeled data
  • Noisy data

Type of Anomaly
Anomalies can be classified into following three categories
  1. Point Anomalies - An individual data instance can be considered as anomalous with respect to the rest of data
  2. Contextual Anomalies - A data instance is anomalous in a specific context (but not otherwise), then it is termed as a contextual anomaly (also referred as conditional anomaly). Each data instance is defined using following two sets of attributes
    • Contextual attributes. The contextual attributes are used to determine the context (or neighborhood) for that instance
      eg:
      In time- series data, time is a contextual attribute which determines the position of an instance on the entire sequence
    • Behavioral attributes. The behavioral attributes define the non-contextual characteristics of an instance
      eg:
      In a spatial data set describing the average rainfall of the entire world, the amount of rainfall at any location is a behavioral attribute
      • To explain this we will look into "Exchange Rate History For Converting United States Dollar (USD) to Sri Lankan Rupee (LKR)"[1]
image
Contextual anomaly t2 in a exchange rate time series. Note that the exchange rate at time t1 is same as that at time t2 but occurs in a different context and hence is not considered as an anomaly
     3.    Collective Anomalies - A collection of related data instances is anomalous with respect to the entire data set

Data Labels
The labels associated with a data instance denote if that instance is normal or anomalous. Depending labels availability, anomaly detection techniques can be operated in one of the following three modes
  1. Supervised anomaly detection - Techniques trained in supervised mode assume the availability of a training data set which has labeled instances for normal as well as anomaly class
  2. Semi-Supervised anomaly detection - Techniques that operate in a semi-supervised mode, assume that the training data has labeled instances for only the normal class. Since they do not require labels for the anomaly class
  3. Unsupervised anomaly detection - Techniques that operate in unsupervised mode do not require training data, and thus are most widely applicable. The techniques  implicit assume that normal instances are far more frequent than anomalies in the test data. If this assumption is not true then such techniques suffer from high false alarm rate

Output of Anomaly Detection
Anomaly detection have two types of output techniques
  1. Scores. Scoring techniques assign an anomaly score to each instance in the test data depending on the degree to which that instance is considered an anomaly
  2. Labels. Techniques in this category assign a label (normal or anomalous) to each test instance

Applications of Anomaly Detection
Intrusion detection
Intrusion detection refers to detection of malicious activity. The key challenge for anomaly detection in this domain is the huge volume of data. Thus, semi-supervised and unsupervised anomaly detection techniques are preferred in this domain.Denning[3] classifies intrusion detection systems into host based and net-
work based intrusion detection systems.
  • Host Based Intrusion Detection Systems  - This deals with operating system call traces
  • Network Intrusion Detection Systems - These systems deal with detecting intrusions in network data. The intrusions typically occur as anomalous patterns (point anomalies) though certain techniques model[4] the data in a sequential fashion and detect anomalous subsequences (collective anomalies). A challenge faced by anomaly detection techniques in this domain is that the nature of anomalies keeps changing over time as the intruders adapt their network attacks to evade the existing intrusion detection solutions.
Fraud Detection

Fraud detection refers to detection of criminal activities occurring in commercial organizations such as banks, credit card companies, insurance agencies, cell phone companies, stock market, etc. The organizations are interested in immediate detection of such frauds to prevent economic losses.  Detection techniques used for credit card fraud and network intrusion detection as below.
  • Statistical Profiling using Histograms
  • Parametric Statistical Modeling
  • Non-parametric Statistical Modeling Bayesian Networks
  • Neural Networks
  • Support Vector Machines
  • Rule-based
  • Clustering Based
  • Nearest Neighbor based
  • Spectral
  • Information Theoretic
Here are some domain in fraud detections
  • Credit Card Fraud Detection
  • Mobile Phone Fraud Detection
  • Insurance Claim Fraud Detection
  • Insider Trading Detection
Medical and Public Health Anomaly Detection
Anomaly detection in the medical and public health domains typically work with pa- tient records. The data can have anomalies due to several reasons such as abnormal patient condition or instrumentation errors or recording errors. Thus the anomaly detection is a very critical problem in this domain and requires high degree of accuracy.
Industrial Damage Detection
Such damages need to be detected early to prevent further escalation and losses.
Fault Detection in Mechanical Units
Structural Defect Detection
Image Processing
Anomaly detection techniques dealing with images are either interested in any changes in an image over time (motion detection) or in regions which appear ab- normal on the static image. This domain includes satellite imagery.
Anomaly Detection in Text Data
Anomaly detection techniques in this domain primarily detect novel topics or events or news stories in a collection of documents or news articles. The anomalies are caused due to a new interesting event or an anomalous topic.
Sensor Networks
Since the sensor data collected from various wireless sensors has several unique characteristics.

References
[1] http://themoneyconverter.com/USD/LKR.aspx
[2] Varun Chandola, Arindam Banerjee, and Vipin Kumar. 2009. Anomaly detection: A survey. ACM Comput. Surv. 41, 3, Article 15 (July 2009), 58 pages. DOI=10.1145/1541880.1541882 http://doi.acm.org/10.1145/1541880.1541882
[3] Denning, D. E. 1987. An intrusion detection model. IEEE Transactions of Software Engineer-ing 13, 2, 222–232.
[4]Gwadera, R., Atallah, M. J., and Szpankowski, W. 2004. Detection of significant sets of episodes in event sequences. In Proceedings of the Fourth IEEE International Conference on Data Mining. IEEE Computer Society, Washington, DC, USA, 3–10.
0

Add a comment

Prevent the breaking of a Singleton Class Pattern
Prevent the breaking of a Singleton Class Pattern
13
Design Patterns for Microservices
Design Patterns for Microservices
12
API Monetization Models
API Monetization Models
1
Kubernetes command-line tool for Windows
2
WSO2 Enterprise Integrator with message broker profile
1
Messaging Patterns on Enterprise integration
Messaging Patterns on Enterprise integration
2
Writing Micro Services with msf4j
SMPP to wso2 ESB / EI
1
WSO2 APIM - Deployment Patterns and Profiles
SMS with WSO2 ESB
5
Reading Value from uri-template in WS02 ESB
1
Estimation for Software project development
Estimation for Software project development
1
JAVA8 Stream API and New Class Optional
Lifecycle of a Book in WSO2 Greg
1
Enterprise Data integration Directions
Enterprise Data integration Directions
Handling BigDecimal in Talend
Vehicles registration services - Part 01– PayloadFactory and Validate with JSON
1
Handling simple denormalized data from Talend
WSO2 ESB with JavaScript Object Notation
Cleaning OSSIM Alarms
Syscheck in OSSEC
Syscheck in OSSEC
Triggering action or email over the event occurrence in OSSIM
Adding More user data field for Event
Connecting to OSSEC rule from OSSIM
Creating New Rule set for OSSEC Server
1
OSSEC Rule Testing
Sending Brute force attack
DiskPart in window (Fdisk in windows 8)
Uncomplicated Firewall
Uncomplicated Firewall
Grep quotes in Linux
Grep quotes in Linux
OSSEC Decoder
How access log work with OSSIM
HIDS Agentless in AlienVault USM
Install OSSIM
OSSEC configure to new log file
OSSEC configure to new log file
Testing Log forwarding in OSSEC
Host based firewall in Linux
Adding OSSEC client to OSSEC Server
Creating Correlation Rules and Alarms in AlienVault
Advance Tutorial in OSSIM Directive
Simple OSSIM Directives
Making OSSIM Alarm from Event
Reading a custom log file from OSSIM
2
OSSIM components
module.exports VS exports
Adding agent for OSSIM from OSSEC
OSSEC service for Centos7
1
JavaScript references with setTimeout() and setInterval()
JavaScript references with setTimeout() and setInterval()
CROS in Node
CROS in Node
Node.JS with Express Session
Node.JS with Express Session
dhis2-android-dashboard Build from Source
Python make life easy
Python make life easy
Installing NodeJS in CentOS
Packaging and Distributing Python Projects
Zeppelin Data Validation Service
Zeppelin Data Validation Service
Introducing New Chart Library and Types for Apache Zeppelin
1
Tutorial with Map Visualization in Apache Zeppelin
Zeppelin Docs
Data validation
Generate a AngularJS application with grunt and bower
7
Git simple Feature Branch Workflow
Git simple Feature Branch Workflow
Workflows for Git
Workflows for Git
1
Chart Types and Data Models in Google Charts
Options for Google Charts
Google Chart with AngularJS
3
Grammar induction
Adding Configuration file for Python
NLTK tutorial–03 (n-gram)
NLTK tutorial–02 (Texts as Lists of Words / Frequency words)
Natural Language Toolkit (NLTK) sample and tutorial - 01
AffinityPropagation Clustering Algorithm
AffinityPropagation Clustering Algorithm
Building Zeppelin in windows 8
1
Zeppelin Note for load data and Analyzing
Zeppelin NoteBook
Data Binding in Angular
5
AngularJS and Angular Directives
AngularJS and Angular Directives
19
Density-based clustering algorithm (DBSAN) and Implementation
scikit-learn to generate isotropic Gaussian blobs
1
CouchDB 2.0 (Developer Preview) with HTTP APIs
CouchDB-fauxton introduction
2
Building Apache Zeppelin
CouchDB with Fauxton in windows 8
Installing Flask in Windows8
1
Basic Functionality of Series or DataFrame in Pandas
Pandas for Data Manipulation and Analysis
Pandas for Data Manipulation and Analysis
Install python 2.7.X on Ubuntu
Install python 2.7.X on Ubuntu
Python For Beginners
Python with CSV
Python Class
Python Class
Maven 3.3.x for Mint
Bower: Front-end Package Manager
Predictive modeling
Predictive modeling
Gaussian function
Installing external package for canopy / (python)
Regular Expressions with Python
2
Python Code for File reading
Estimation Theory
AngularJS and solr
3
Apache Solr and Search
I am
I am
Archives
Total Pageviews
Total Pageviews
2 0 5 8 0 4 4
Categories
Categories
Loading