On-The-Fly Processing of continuous high-dimensional data streams

Abstract

[EN] A novel method and software system for rational handling of time series of multi-channel measurements is presented. This quantitative learning tool, the On-The-Fly Processing (OTFP), develops reduced-rank bilinear subspace models that summarise massive streams of multivariate responses, capturing the evolving covariation patterns among the many input variables over time and space. Thereby, a considerable data compression can be achieved without significant loss of useful systematic information.

The underlying proprietary OTFP methodology is relatively fast and simple it is linear/bilinear and does not require a lot of raw data or huge cross-correlation matrices to be kept in memory. Unlike conventional compression methods, the approach allows the high-dimensional data stream to be graphically interpreted and quantitatively utilised in its compressed state. Unlike adaptive moving-window methods, it allows all past and recent time points to be reconstructed and displayed simultaneously.

This new approach is applied to four different case-studies: (i) multi-channel Vis-NIR spectroscopy of the Belousov Zhabotinsky reaction, a complex, ill understood chemical process; (ii) quality control of oranges by hyperspectral imaging; (iii) environmental monitoring by airborne hyperspectral imaging; (iv) multi-sensor process analysis in the petrochemical industry. These examples demonstrate that the OTFP can automatically develop high-fidelity subspace data models, which simplify the storage/transmission and the interpretation of more or less continuous time series of high-dimensional measurements to the extent there are covariations among the measured variables.