Using affinity perturbations to detect web traffic anomalies

The initial training phase of machine learning algorithms is usually computationally expensive as it involves the processing of huge matrices. Evolving datasets are challenging from this point of view because changing behavior requires updating the training. We propose a method for updating the training profile efficiently and a sliding window algorithm for online processing of the data in smaller fractions. This assumes the data is modeled by a kernel method that includes spectral decomposition. We demonstrate the algorithm with a web server request log where an actual intrusion attack is known to happen. Updating the kernel dynamically using a sliding window technique, prevents the proble…

Hilbert-Huang versus Morlet wavelet transformation on mismatch negativity of children in uninterrupted sound paradigm

Background Compared to the waveform or spectrum analysis of event-related potentials (ERPs), time-frequency representation (TFR) has the advantage of revealing the ERPs time and frequency domain information simultaneously. As the human brain could be modeled as a complicated nonlinear system, it is interesting from the view of psychological knowledge to study the performance of the nonlinear and linear time-frequency representation methods for ERP research. In this study Hilbert-Huang transformation (HHT) and Morlet wavelet transformation (MWT) were performed on mismatch negativity (MMN) of children. Participants were 102 children aged 8–16 years. MMN was elicited in a passive oddball parad…

Gear classification and fault detection using a diffusion map framework

This article proposes a system health monitoring approach that detects abnormal behavior of machines. Diffusion map is used to reduce the dimensionality of training data, which facilitates the classification of newly arriving measurements. The new measurements are handled with Nyström extension. The method is trained and tested with real gear monitoring data from several windmill parks. A machine health index is proposed, showing that data recordings can be classified as working or failing using dimensionality reduction and warning levels in the low dimensional space. The proposed approach can be used with any system that produces high-dimensional measurement data. peerReviewed

Applying Hilbert-Huang transform to mismatch negativity

Dimensionality reduction framework for detecting anomalies from network logs

Dynamic web services are vulnerable to multitude of intrusions that could be previously unknown. Server logs contain vast amounts of information about network traffic, and finding attacks from these logs improves the security of the services. In this research features are extracted from HTTP query parameters using 2-grams. We propose a framework that uses dimensionality reduction and clustering to identify anomalous behavior. The framework detects intrusions from log data gathered from a real network service. This approach is adaptive, works on the application layer and reduces the number of log lines that needs to be inspected. Furthermore, the traffic can be visualized. peerReviewed

Gear classification and fault detection using a diffusion map framework

Online anomaly detection using dimensionality reduction techniques for HTTP log analysis

Modern web services face an increasing number of new threats. Logs are collected from almost all web servers, and for this reason analyzing them is beneficial when trying to prevent intrusions. Intrusive behavior often differs from the normal web traffic. This paper proposes a framework to find abnormal behavior from these logs. We compare random projection, principal component analysis and diffusion map for anomaly detection. In addition, the framework has online capabilities. The first two methods have intuitive extensions while diffusion map uses the Nyström extension. This fast out-of-sample extension enables real-time analysis of web server traffic. The framework is demonstrated using …

Key issues in decomposing fMRI during naturalistic and continuous music experience with independent component analysis

Background: Independent component analysis (ICA) has been often used to decompose fMRI data mostly for the resting-state, block and event-related designs due to its outstanding advantage. For fMRI data during free-listening experiences, only a few exploratory studies applied ICA.New method: For processing the fMRI data elicited by 512-s modern tango, a FFT based band-pass filter was used to further pre-process the fMRI data to remove sources of no interest and noise. Then, a fast model order selection method was applied to estimate the number of sources. Next, both individual ICA and group ICA were performed. Subsequently, ICA components whose temporal courses were significantly correlated …

Combining conjunctive rule extraction with diffusion maps for network intrusion detection

Network security and intrusion detection are important in the modern world where communication happens via information networks. Traditional signature-based intrusion detection methods cannot find previously unknown attacks. On the other hand, algorithms used for anomaly detection often have black box qualities that are difficult to understand for people who are not algorithm experts. Rule extraction methods create interpretable rule sets that act as classifiers. They have mostly been combined with already labeled data sets. This paper aims to combine unsupervised anomaly detection with rule extraction techniques to create an online anomaly detection framework. Unsupervised anomaly detectio…

Knowledge Discovery from Network Logs

Modern communications networks are complex systems, which facilitates malicious behavior. Dynamic web services are vulnerable to unknown intrusions, but traditional cyber security measures are based on fingerprinting. Anomaly detection differs from fingerprinting in that it finds events that differ from the baseline traffic. The anomaly detection methodology can be modelled with the knowledge discovery process. Knowledge discovery is a high-level term for the whole process of deriving actionable knowledge from databases. This article presents the theory behind this approach, and showcases research that has produced network log analysis tools and methods. peerReviewed

Anomaly Detection Framework Using Rule Extraction for Efficient Intrusion Detection

Huge datasets in cyber security, such as network traffic logs, can be analyzed using machine learning and data mining methods. However, the amount of collected data is increasing, which makes analysis more difficult. Many machine learning methods have not been designed for big datasets, and consequently are slow and difficult to understand. We address the issue of efficient network traffic classification by creating an intrusion detection framework that applies dimensionality reduction and conjunctive rule extraction. The system can perform unsupervised anomaly detection and use this information to create conjunctive rules that classify huge amounts of traffic in real time. We test the impl…

Knowledge discovery using diffusion maps

Musicianship can be decoded from magnetic resonance images

AbstractLearning induces structural changes in the brain. Especially repeated, long-term behaviors, such as extensive training of playing a musical instrument, are likely to produce characteristic features to brain structure. However, it is not clear to what extent such structural features can be extracted from magnetic resonance images of the brain. Here we show that it is possible to predict whether a person is a musician or a non-musician based on the thickness of the cerebral cortex measured at 148 brain regions en-compassing the whole cortex. Using a supervised machine-learning technique, we achieved a significant (κ = 0.321, p < 0.001) agreement between the actual and predicted par…

Adaptive framework for network traffic classification using dimensionality reduction and clustering

Information security has become a very important topic especially during the last years. Web services are becoming more complex and dynamic. This offers new possibilities for attackers to exploit vulnerabilities by inputting malicious queries or code. However, these attack attempts are often recorded in server logs. Analyzing these logs could be a way to detect intrusions either periodically or in real time. We propose a framework that preprocesses and analyzes these log files. HTTP queries are transformed to numerical matrices using n-gram analysis. The dimensionality of these matrices is reduced using principal component analysis and diffusion map methodology. Abnormal log lines can then …

Research literature clustering using diffusion maps

We apply the knowledge discovery process to the mapping of current topics in a particular field of science. We are interested in how articles form clusters and what are the contents of the found clusters. A framework involving web scraping, keyword extraction, dimensionality reduction and clustering using the diffusion map algorithm is presented. We use publicly available information about articles in high-impact journals. The method should be of use to practitioners or scientists who want to overview recent research in a field of science. As a case study, we map the topics in data mining literature in the year 2011. peerReviewed

Anomaly Detection from Network Logs Using Diffusion Maps

The goal of this study is to detect anomalous queries from network logs using a dimensionality reduction framework. The fequencies of 2-grams in queries are extracted to a feature matrix. Dimensionality reduction is done by applying diffusion maps. The method is adaptive and thus does not need training before analysis. We tested the method with data that includes normal and intrusive traffic to a web server. This approach finds all intrusions in the dataset. peerReviewed

Diffusion map for clustering fMRI spatial maps extracted by Indipendent Component Analysis

Functional magnetic resonance imaging (fMRI) produces data about activity inside the brain, from which spatial maps can be extracted by independent component analysis (ICA). In datasets, there are n spatial maps that contain p voxels. The number of voxels is very high compared to the number of analyzed spatial maps. Clustering of the spatial maps is usually based on correlation matrices. This usually works well, although such a similarity matrix inherently can explain only a certain amount of the total variance contained in the high-dimensional data where n is relatively small but p is large. For high-dimensional space, it is reasonable to perform dimensionality reduction before clustering.…

Concatenated trial based Hilbert-Huang transformation on event-related potentials

Time-frequency analysis is critical to study event-related potentials (ERPs) now. ERPs are usually generated through averaging over a number of trials, and such averaging limits the application of a nonlinear time-frequency analysis method—Hilbert-Huang transformation (HHT). This is because HHT usually requires very long recordings to sufficiently decompose the complicated signal into oscillations and the averaged ERP trace tends to possess only hundreds of samples. Thus, this study designs the concatenated trial based HHT to release the limitation on the decomposition. Such a paradigm may reveal better temporal and spectral properties of an ERP than the conventional wavelet transformation …

One-Pixel Attack Deceives Computer-Assisted Diagnosis of Cancer

Computer vision and machine learning can be used to automate various tasks in cancer diagnostic and detection. If an attacker can manipulate the automated processing, the results can be devastating and in the worst case lead to wrong diagnosis and treatment. In this research, the goal is to demonstrate the use of one-pixel attacks in a real-life scenario with a real pathology dataset, TUPAC16, which consists of digitized whole-slide images. We attack against the IBM CODAIT's MAX breast cancer detector using adversarial images. These adversarial examples are found using differential evolution to perform the one-pixel modification to the images in the dataset. The results indicate that a mino…

Hilbert-Huang versus morlet wavelet transformation on mismatch negativity of children in uninterrupted sound paradigm

Background. Compared to the waveform or spectrum analysis of event-related potentials (ERPs), time-frequency representation (TFR) has the advantage of revealing the ERPs time and frequency domain information simultaneously. As the human brain could be modeled as a complicated nonlinear system, it is interesting from the view of psychological knowledge to study the performance of the nonlinear and linear time-frequency representation methods for ERP research. In this study Hilbert-Huang transformation (HHT) and Morlet wavelet transformation (MWT) were performed on mismatch negativity (MMN) of children. Participants were 102 children aged 8–16 years. MMN was elicited in a passive oddbal…