This post is summary of the “Anomaly Detection : A Survey”. Anomaly detection refers to the problem of finding patterns in data that do not conform to expected behavior. These non-conforming patterns are often referred to as anomalies, outliers, discordant observations, exceptions, aberrations, surprises, peculiarities or contaminants in different application domains.

Anomalies are patterns in data that do not conform to a well defined notion of normal behavior.
  • Interesting to analyze
  • Unwanted noise in the data also can be found in there.
  • Novelty detection which aims at detecting previously unobserved (emergent, novel) patterns in the data
Challenges for Anomaly Detection
  • Drawing the boundary between normal and anomalous behavior
  • Availability of labeled data
  • Noisy data

Type of Anomaly
Anomalies can be classified into following three categories
  1. Point Anomalies - An individual data instance can be considered as anomalous with respect to the rest of data
  2. Contextual Anomalies - A data instance is anomalous in a specific context (but not otherwise), then it is termed as a contextual anomaly (also referred as conditional anomaly). Each data instance is defined using following two sets of attributes
    • Contextual attributes. The contextual attributes are used to determine the context (or neighborhood) for that instance
      eg:
      In time- series data, time is a contextual attribute which determines the position of an instance on the entire sequence
    • Behavioral attributes. The behavioral attributes define the non-contextual characteristics of an instance
      eg:
      In a spatial data set describing the average rainfall of the entire world, the amount of rainfall at any location is a behavioral attribute
      • To explain this we will look into "Exchange Rate History For Converting United States Dollar (USD) to Sri Lankan Rupee (LKR)"[1]
image
Contextual anomaly t2 in a exchange rate time series. Note that the exchange rate at time t1 is same as that at time t2 but occurs in a different context and hence is not considered as an anomaly
     3.    Collective Anomalies - A collection of related data instances is anomalous with respect to the entire data set

Data Labels
The labels associated with a data instance denote if that instance is normal or anomalous. Depending labels availability, anomaly detection techniques can be operated in one of the following three modes
  1. Supervised anomaly detection - Techniques trained in supervised mode assume the availability of a training data set which has labeled instances for normal as well as anomaly class
  2. Semi-Supervised anomaly detection - Techniques that operate in a semi-supervised mode, assume that the training data has labeled instances for only the normal class. Since they do not require labels for the anomaly class
  3. Unsupervised anomaly detection - Techniques that operate in unsupervised mode do not require training data, and thus are most widely applicable. The techniques  implicit assume that normal instances are far more frequent than anomalies in the test data. If this assumption is not true then such techniques suffer from high false alarm rate

Output of Anomaly Detection
Anomaly detection have two types of output techniques
  1. Scores. Scoring techniques assign an anomaly score to each instance in the test data depending on the degree to which that instance is considered an anomaly
  2. Labels. Techniques in this category assign a label (normal or anomalous) to each test instance

Applications of Anomaly Detection
Intrusion detection
Intrusion detection refers to detection of malicious activity. The key challenge for anomaly detection in this domain is the huge volume of data. Thus, semi-supervised and unsupervised anomaly detection techniques are preferred in this domain.Denning[3] classifies intrusion detection systems into host based and net-
work based intrusion detection systems.
  • Host Based Intrusion Detection Systems  - This deals with operating system call traces
  • Network Intrusion Detection Systems - These systems deal with detecting intrusions in network data. The intrusions typically occur as anomalous patterns (point anomalies) though certain techniques model[4] the data in a sequential fashion and detect anomalous subsequences (collective anomalies). A challenge faced by anomaly detection techniques in this domain is that the nature of anomalies keeps changing over time as the intruders adapt their network attacks to evade the existing intrusion detection solutions.
Fraud Detection

Fraud detection refers to detection of criminal activities occurring in commercial organizations such as banks, credit card companies, insurance agencies, cell phone companies, stock market, etc. The organizations are interested in immediate detection of such frauds to prevent economic losses.  Detection techniques used for credit card fraud and network intrusion detection as below.
  • Statistical Profiling using Histograms
  • Parametric Statistical Modeling
  • Non-parametric Statistical Modeling Bayesian Networks
  • Neural Networks
  • Support Vector Machines
  • Rule-based
  • Clustering Based
  • Nearest Neighbor based
  • Spectral
  • Information Theoretic
Here are some domain in fraud detections
  • Credit Card Fraud Detection
  • Mobile Phone Fraud Detection
  • Insurance Claim Fraud Detection
  • Insider Trading Detection
Medical and Public Health Anomaly Detection
Anomaly detection in the medical and public health domains typically work with pa- tient records. The data can have anomalies due to several reasons such as abnormal patient condition or instrumentation errors or recording errors. Thus the anomaly detection is a very critical problem in this domain and requires high degree of accuracy.
Industrial Damage Detection
Such damages need to be detected early to prevent further escalation and losses.
Fault Detection in Mechanical Units
Structural Defect Detection
Image Processing
Anomaly detection techniques dealing with images are either interested in any changes in an image over time (motion detection) or in regions which appear ab- normal on the static image. This domain includes satellite imagery.
Anomaly Detection in Text Data
Anomaly detection techniques in this domain primarily detect novel topics or events or news stories in a collection of documents or news articles. The anomalies are caused due to a new interesting event or an anomalous topic.
Sensor Networks
Since the sensor data collected from various wireless sensors has several unique characteristics.

References
[1] http://themoneyconverter.com/USD/LKR.aspx
[2] Varun Chandola, Arindam Banerjee, and Vipin Kumar. 2009. Anomaly detection: A survey. ACM Comput. Surv. 41, 3, Article 15 (July 2009), 58 pages. DOI=10.1145/1541880.1541882 http://doi.acm.org/10.1145/1541880.1541882
[3] Denning, D. E. 1987. An intrusion detection model. IEEE Transactions of Software Engineer-ing 13, 2, 222–232.
[4]Gwadera, R., Atallah, M. J., and Szpankowski, W. 2004. Detection of significant sets of episodes in event sequences. In Proceedings of the Fourth IEEE International Conference on Data Mining. IEEE Computer Society, Washington, DC, USA, 3–10.
0

Add a comment

We used have  Singleton Design Pattern in our applications whenever it is needed. As we know that in singleton design pattern we can create only one instance and can access in the whole application. But in some cases, it will break the singleton behavior.

13

Microservices can have a positive impact on your enterprise. Therefore it is worth to know that, how to handle Microservice Architecture (MSA) and some Design Patterns for Microservices. General goals or principles for a microservice architecture.

12

Last few years has been a great year for API Gateways and API companies. APIs (Application Programming Interfaces) are allowing businesses to expand beyond their enterprise boundaries to drive revenue through new business models.

1

kubectl (Kubernetes command-line tool) is to deploy and manage applications on Kubernetes. Using kubectl, you can inspect cluster resources; create, delete, and update components.

NOTE

You must use a kubectl version that is within one minor version difference of your cluster.

2

WSO2 Enterprise Integrator is shipped with a separate message broker profile (WSO2 MB). In this Post I will be using message broker profile in EI (6.3.0).

1

When two or many applications want to exchange data, they do so by sending the data through a channel that connects the each others.

2

Microservices are going completely over the enterprise and changed the way people write software within an enterprise ecosystem.

Let build you microservices with msf4j for Auto Mobile.

The SMPP inbound endpoint allows you to consume messages from SMSC via WSO2 ESB OR EI.

1.  Start SMSC

2.  Create custom inbound end point with below parameter. (Make sure you pick correct system-id and password correct for your SMSC)

3. Create Sequence for Inbound EP.

4. Once ESB or EI start.

1

WSO2 APIM Components

WSO2 API Manager includes five main components as the Publisher, Store, Gateway, Traffic Manager and Key Manager.

API Gateway - responsible for securing, protecting, managing, and scaling API calls.

There is REST Back-End end-point in Vehicle registration services as below

GET /car?name=prius HTTP/1.1

Host: localhost:8080

color: White

Company need to expose it 3rd part companies and above End-point should not change as internal services are using it.

1

Estimation for Software project development is the process of predicting the most realistic amount of effort (expressed in terms of person-hours or money) required to develop or maintain software based on incomplete, uncertain and noisy input.

1

In this post give some basic on JAVA Stream API which is added in Java 8. It works very well in conjunction with lambda expressions. Pipeline of stream operations can manipulate data by performing operations like search, filter, count, sort, etc.

The Lifecycle Management(LCM) plays a major role in SOA Governance. WSO2 Governance Registry Lifecycle Management supports access control at multiple levels in lifecycle state.

1.

1

Enterprise Data Integration is a broad term used in the integration landscape to connect multiple Enterprise  applications and hardware systems within an organization. All these enterprise data integration lead to achieve to remove the complexity by simplifying data management as a whole.

Post is very basic one, Since Talend is all about data integration. Finding a BigDecimal [1] in such data set is very common.

BigDecimal VS Doubles

A BigDecimal is an exact way of representing numbers. A Double has a certain precision.

Vehicles registration services using REST services on government TAX department system. That REST services give the TAX information for the Vehicle.

{"Tax": {"Amount": 58963}}

Vehicles registration Depart planning to extend the service and expose as below.

1

Data integration is the combination of technical and business processes used to combine data from disparate sources into meaningful and valuable information. Today some systems may store data in a denormalized form and data integration tools able handle those.

There is few thing that make my work enjoyable with WSO2 ESB as it provides support for JavaScript Object Notation (JSON) payloads in messages. It is not very new feature and it old feature.

Working on an Alienvault IDS system or OSSIM you can come across over huge amount of alarms are created will system migrations.

If you’re familiar with SEIM tools or OSSEC, then you know syscheck. Syscheck is the integrity checking daemon within OSSEC. It’s purpose is simple, identify and report on changes within the system files.

Triggering action over the event occurrence in OSSIM is going to explain in this article.

There is agent in the system with IP, 192.168.80.22. Email is to be send to server admins whenever this agent disconnect and reconnect to SEIM server.

We need to have extra user data field on our security event. We need to know

event occurred time Host Server IP Editing particular event on ‘/etc/ossim/agent/plugins/ossec-single-line.cfg’. We can achieve it. We are interest on Web group and ID 0030. We added below line as our need.

Pre request

Test OSSEC new log from ‘ossec-logtest’

Here is the custom created rules.

In here I am using well known decoder in OSSEC if you need new OSSEC decoder you can write new decoder also [1]. Add new file to  rules directory in OSSEC.

Creating new OSSEC rule set

$ vi var/ossec/rules/custom_access_rules.xml

In here I am interest to monitor web user behavior model.

1

Introductions

In OSSEC, the rules are classified in multiple levels from the lowest (00) to the maximum level 16. But some levels are not used right now and below explain level details.

A brute-force attack consists of an attacker trying many passwords or passphrases with the hope of eventually guessing correctly. The attacker systematically checks all possible passwords and passphrases until the correct one is found.

Unfortunately Windows does not support Fdisk anymore. But there is another good command line tool to solve this problem. DiskPart in windows is useful format unallocated spaces in USB pen.

1. Enter ‘diskpart’ in cmd

Then disk part will start

2. List down storage in PC by

list disk

3.

The Linux kernel in Ubuntu provides a packet filtering system called netfilter, and the traditional interface for manipulating netfilter are the iptables suite of commands. The Uncomplicated Firewall (ufw) is a frontend for iptables and is particularly well-suited for host-based firewalls.

Count line when words has been matched

$ grep -c 'word' /path/to/file

Pass the -n option to precede each line of output with the number of the line in the text file

$ grep -n 'root' /etc/passwd

Ignore word case

$ grep -i 'word' /path/to/file

Use grep recursively under each directory

$ grep -r

Each application contains it's own log record format.

Access log moves to sensor / data source then I mapping to event id with considering the rules in ossim.

Data sources can be found in “ossim ->configuration –> threat_intelligence –> data_source” and search for source as below. Pick “AlienVault HIDS-accesslog” and it reads the access log.

It provides the SSH authentication to the host you want to access. For Cisco devices (PIX, routers, etc), you need to provide an additional parameter for the enable password. The same thing applies if you want to add support for “su”, it must be the additional parameter.

1. Log into AlienVault USM.

1. Download the image file of OSSIM

2. Make bootable pen with OSSIM ISO file

3. Boot drive

Make sure you have internet connection

4. Select OSSIM server to install

5. Just follow the the wizard

6. Add the net work details correctly with unique new IP for OSSIM server.

7.

Finding the logs in my server. I generally use lsof to list what is my server.

lsof | grep log

I check which log are reading by OSSEC

Check cat /var/ossec/etc/ossec.conf  |grep "<location>/"

Add new access log to OSSCE.

I am
I am
Archives
Total Pageviews
Total Pageviews
2 0 5 8 0 4 4
Categories
Categories
Loading