CENIIT Project 17.12:
Applied Research Platform for
Sensor Fusion

The aim of this project is to create a research group that using probabilistic learning methods addresses the needs introduced by the surge of less expensive sensors, and in the process to complement the existing group with competence in computer science and more knowledge of hardware. The goal of the proposal is to perform agile development of theory for emerging applications with early demonstrators and active involvement of students.

Background and Industrial Relevance

Sensor and information fusion is a mature area with its own society (ISIF) and conference (FUSION). The subject has advanced in stages. In the 1960's, the Apollo program and aerospace applications drove the development. It resulted in algorithms such as the Kalman filter (KF) and the extended Kalman filter (EKF), which are still important corner stones in the field. The next advancement was in the 1990's, when the development of faster computers led to better methods to handle nonlinear problems. The unscented Kalman filter (UKF), a development of the Kalman filter methodology, and the simulation based particle filter (PF), are examples of methods developed during this time. At the same time the focus shifted towards automotive industry, which in many ways can be considered the driving force in the early 2000's.

The basis for this development is the system description \[ \begin{align} x(t+T_s) &= f\bigl(x(t), v(t)\!\bigr)\\ % % y(t) &= h\bigl(x(t)\!\bigr)+e(t) , \end{align} \] where the state \(x(t)\), in a state-space model, should be estimated given the measurements \(y(t)\). Typical assumptions in the classical setting are: Known time, \(t\). The sensor model, \(h(\cdot)\), is idealized. The measurements are well structured and the noise \(e(\cdot)\) independent. And, relatively few measurements are available.

LiU, and the Division of Automatic Control in particular, has a strong position in the field developing both theory and applications. Sensor fusion has been an explicit research area in large research projects such as ISIS, MOVII, LINK-SIC, CADICS, and Security Link since 1995, and more than 200 MSc projects have been preformed in the area. Since 2010 there is an undergraduate course called Sensor Fusion with approximately 40 students attending each year. A success factor has been the ability to combine high quality theoretical work with relevant and successful applications, e.g., in the aerospace and automotive settings. The result has been a number of well cited journal articles and successful spin-offs, such as NIRA Dynamics (market leading in indirect tire pressure monitoring) and Senion (indoor localization solutions).

Since 2010 there has been another shift in attention in the field towards smart devices, including smartphones, augmented and virtual reality (AR, VR), drones, intelligent hearing aids, etc. In the future internet of things (IoT) devices and Industry 4.0 are expected to gain popularity. An enabler is the availability of an abundance of inexpensive sensors and computational resources.

Research challenges

Typically, the price paid with these new sensors is worse quality. The model above must be adapted to reflect the imperfect measurement model, which include unknown parameters, \(\theta\), and uncertain measurement times, \(\tau\), \[ \begin{align} x(t+T_s) &= f(x(t), w(t))\\ % % y\bigl(\tau(t; \theta)\!\bigr) &= h\bigl(x(\tau(t; \theta)\!); \theta\bigr) + e\bigl(\tau(t; \theta); \theta\bigr) . \end{align} \] The problem formulation now includes uncalibrated sensor parameters, \(\theta\), (exact location, time and amplitude wrapping, measurement quantizations and saturation, clock parameters, etc). The measurement, \(y\), is in many cases high-dimensional. And, the data collected often exhibit the `four V' aspects associated with Big Data (volume, velocity, variability and veracity).

Algorithms targeting this type of applications are required to be efficient, as computing power and batteries are limited, and robust, to compensate for individual sensor variations. At the same time, it might be necessary to identify sensor parameters on the fly or create sensor models from data alone using machine learning inspired methods. Offloading computations to the cloud is one option to address limited computing power. To provide general solutions requires skills in sensor fusion and computer science, as well as, familiarity with hardware. Short of this, many of the solutions in use today are rather ad hoc, designed to address a specific problem using a specific sensor.

Project description

The aim of this proposal is to initialize the creation of a research group that performs basic research focusing on challenges introduced by imperfect sensor models and their implications to state estimation, and in parallel demonstrates the results using relevant hardware. The inspiration comes from the PI's experiences working at DFKI and FOI, experiencing the problems described above first hand. The topic is important in order not to miss out on the potential of the new technology, and as a reaction to the increasing importance of being able to demonstrate results in demonstrators to attract funding.

To deal with the imperfect sensor models requires both development of new theory and solving practical problems. The project aims to develop a theoretical framework for probabilistic learning of the sensor models. Data driven machine learning algorithms adapted to the stochastic signal processing framework will be considered as a mean to acquire the necessary models. In some cases, big data aspects must be considered, e.g., efficient methods for offloading some of the computations to the cloud.

The goal of the proposal is to perform agile development of theory for emerging applications with early demonstrators and active involvement of students, e.g., utilizing the Sensor Fusion app to achieve this.

The following areas have been identified as highly relevant and are currently pursued:

• Condition monitoring in robotics and transportation systems: Sounds and vibrations in machinery and transportation systems can tell a lot about the integrity of it. Sensors measuring vibrations and sound can therefore be used for condition monitoring. Areas with ongoing activities:
  1. Synchronization of inertial data for almost-periodic events.

     Gait can be characterized using measurements of acceleration, from low cost sensors such as cellphones with the Sensor Fusion app. The gait cycle is first identified, and then time is reparameterized (or \(\tau(t; \theta) \) determined), to localize the measurements relative to the gait cycle. This can be used to learn the gait cycle, and based on this classify gait with high accuracy. We also hope to, as an extension of this, be able to detect gait deviations such as limping at an early stage. Work by a PhD student on the topic is presented in the PhD thesis [PhD2].

  2. Vibrations in cars. In a PhD project, vibrations data from cars have been collected and shown to provide speed information. This has attracted attention from NIRA Dynamics as part of a GPS-free navigation solution.

     The sound spectrum of a moving vehicle depends on engine speed, wheel speed, and other rotating components of the drive line. The exact dependence is impossible to model perfectly, so learning strategies can be developed from massive data with speed reference signals to improve simpler methods such as estimating the speed from peaks in the spectrum. In this case, \(h(x; \theta)\) models how the sound spectrum is affected by the current speed and gear level. Initial results include methods for robust extraction of relevant frequency peaks developed in cooperation with Rickard Karlsson, NIRA Dynamics.

• Robust localization: An area with several on-going research projects where cheap networked sensors can replace costly hardware and provide a more flexible demonstrator environment, e.g.:
  1. Small scale MEMS microphone array. A small scale MEMS microphone array, that has the potential to revolutionize hearing aids by providing directional information about sound sources. This has attracted the attention of the hearing aid manufacturer Oticon, who has provided funding for a postdoc to continue developing the idea.

     MEMS microphones have complicated noise properties, and cannot be be assumed independent when placed tightly together. The idea here is to learn the interconnections \(h(x; \theta)\).

  2. Magnetic arrays for tracking and odometry. Magnetometers can be used for tracking magnetic objects if the local magnetic field is known, and also for magnetic odometry, where a sensor matrix estimates the velocity vector from variations in the local magnetic field [C2]. In both cases, the local magnetic field needs to be estimated accurately, and this becomes part of the function \(h(x; \theta)\).

     The former is of interest to the spin-off company Styleaero and Niklas Wahlström at Uppsala University, and the latter to FOI and Isaac Skog, LiU/FOI.

  3. Network of BLE and WiFi sniffers. A network of smartphones can be used to measure radio signal strength (RSS) from cellular networks, WiFi, and/or Bluetooth to localize transmitters. The Sensor Fusion app is currently used with this purpose in PhD and MSc courses.

     The RSS in free space mainly depends on the distance to the transmitter, but in reality the RSS is a complicated function \(h(x; \theta)\) of the position of the receiver. There are many proposals in literature for how to learn this "fingerprint", but not so many academic publications on how massive data can be used. Further, the literature focuses on indoor applications. We want to develop accurate models for outdoor applications for accurate tracking of animals and people, including search and rescue scenarios.

  4. Distributed estimation. Networks with an abundance of (cheap) sensors, can produce an abundance of data. Distributed algorithms are then a way to deal with the inhibiting high computational complexity otherwise resulting from the big data.

     Distributed filtering is also a prerequisite for robust sensor networks that scale well in terms of both computational and communication complexity. One of the key issues here is how deal with potentially correlated information, a property of \(e(\tau(t; \theta), \theta)\), which can seriously hurt the integrity of common methods if not properly handled. The problem formulation has received attention from both the Swedish Defence Research Agency (FOI) [C1, MSc1] and SAAB Aeronautics (which has a industrial PhD student co-supervised by the PI [C3, C6]).

An important prerequisite to achieve this is to have access to realistic measurement data, which the Sensor Fusion app provides.

Visions and plans

The vision is to establish a research group that performs basic research on these new challenges and in parallel demonstrates the results using relevant hardware, this way ensuring that LiU keeps its strong position also in emerging applications where low-cost hardware and cloud solutions are increasingly important. A key component is working with real problems and data, which inevitably reveals bottlenecks of existing theory and sparks new research directions.

Dissemination will include papers in the FUSION conference for basic research, application orientated conferences for demonstrators, journal articles for the basic theory with applications, and not the least apps that are spread using Google Play.

Research environment and industrial cooperation

The research project is based on relevant industrial problem formulations. This is manifested in collaborations with representatives of local organizations.

One strategy to ensure relevant problem formulations and to establish contacts with industrial partners is to involve in MSc thesis projects with them. This establishes contacts with relevant people and identifies common interests that can evolve into long-term research collaborations. It is also a good way to test feasibility of ideas. The PI has since the beginning of the project been examiner for 11 MSc theses at FOI, 5 at Veoneer (former Autoliv, currently Arriver, in Linköping), 3 at Saab Dynamics, and one each at ABB, Einride, Husqvarna AB, Oticon, Saab Aeronatics, SIC IVP, Syntronic, UMS Skeldar, UniVRses, and Öhlins Racing AB. Interesting collaborations and problem formulations have resulted from these MSc projects, e.g., joint applications for funding for projects and industrial PhD student.

Major collaborations are outlined below:

Methods to better deal with the computational and communication complexity associated with estimation in large sensor networks using decentralized algorithms are pursued with the Swedish Defence Research Agency (FOI, Doc. Jonas Nygårds). The cooperation includes both joint work which has resulted in a conference paper published at FUSION 2018 [C2] and a jointly supervised MSc thesis project including two students [MSc1]. The FOI side of the project is financed by a FOI internal Security Link project, in which cooperation with the PI on this topic is a key component. Similar problem formulations are of interest to SAAB Aeronatics, and the industrial PhD student Robin Forsling (co-supervised by the PI). He studies the implications of limited communication ability in sensor networks and how to ensure consistent estimates under these conditions. The initial work has result in three papers presented at FUSION 2019, 2020 and 2022 [C3, C5, C10], a journal article in IEEE Transactions on Signal Processing [J7] and a licentiate thesis [Lic1].

Magnetic odometry as a method to support positioning in GPS denied environments is being pursued in collaboration with FOI (contact: Doc. Isaac Skog). First results were reported at FUSION 2018 [C2]. Additional funding has been acquired within Security Link to build a full scale prototype of the magnetometer array. Initial experiments with a first version of his hardware were conducted in collaboration between LiU and FOI in the spring 2020. Results have been published in [C8] and disseminated as part of a tutorial at IPIN2021 and at FOI. During fall of 2021 a second version of the prototype has been developmed, financed by FOI. This work has futhermore led to two new contacts at FOI in Kista, Felix Trulsson and Doc. Magnus Lundberg Nordenvaad with which there are ongoing discussions about continued collaborations outside this project.

A new collaboration with Saab AB (contact: Dr. Zoran Sjanic) has been initiated, in which robust navigation in GPS denied environments will be studied. Odometric measurements from vision sensors, will be used to support inertial navigation. Funding has been aqcuired for one PostDoc Jeong Min Kang who currently works on the project.

In collaborating with FOI methods for detection and mitigation of GPS jamming and spoofing are studied (contact: Jouni Rantakokko). This work was initiated as a MSc project [MSc2], supported by a FOI internal Security Link project based around the cooperation with the PI on the topic. The MSc student now works at FOI, and has continued the work started in the MSc thesis. The collaboration continues and has so far resulted in a joint conference paper [C4], and a follow up journal article [J8]. The article resulted in a well-attended webinar hosted by Institute of Navigation (ION). Furthermore, another MSc thesis student is currently working on the topic at FOI.

Algorithms to robustly detect harmonics in measurements of chassis vibrations have been developed by the PhD student Martin Lindfors, who was jointly supervised by Doc. Rickard Karlsson (NIRA Dynamics) and the PI. This has resulted in two conference papers [R‑C2, R‑C3], the licentiate thesis [R‑Lic1] the fall of 2018, and a follow up journal article [R‑J3]. WASP has provided the main funding for Lindfors work, whereas this project has provided means to extend the supervision and to deepen the cooperation with R. Karlsson. This cooperation funded by this project has, after the end of Lindfors' project, resulted in the development of an alternative method for speed estimation [J5], and joint work on how to use the results in a stand alone positioning system.

The Sensor Fusion app has been used by Mats Amundin (Kolmården Wildlife Park) and his MSc students to collect data used to analyze animal behavior. Their usage has been a driving force for the development of the Sensor Fusion app, and resulted in several challenging datasets exhibiting many of the issues which are within the focus of this project and which will be used to drive the algorithm development forward. The Sensor Fusion app was also used for rapid prototyping in the Vinnova funded project "Digital skallgång" (Digitalized Search and Rescue) to locate people in a search and rescue situation using their electronic signatures. It is also one of the data collection units in the Norrköping financed projecte "SenZoor", which aims to analyze visitor behavioral patterns at Kolmården Wildlife Park.

Initial contacts have been made and discussions been initiated with VitalSigns (contact: Jens Ohlsson), a start up company working with digital solutions for first aid and emergency care. A common interest in using mobile devices, e.g., a mobile phone, as sensor platform to improve the quality of performed first aid activities has been identified. Different ways to collaborate are currenly explored.

Altogether, the project intends to strengthen the current research group by introducing a new research platform for sensor fusion utilizing the applicants previous experiences of both theoretical development and industry relevant practical results.

Project Result and Status

The project is progressing on several fronts. Results up until now includes:

• A conference paper [C2], in which a method to obtain odometric information from only an array of magnetometers is derived and experimentally evaluated. The method relies on being able to online learn an instantaneous magnetic field map, and use it to estimate the speed based on the changes experienced in the magnetic field. Based on these initial results:

  • Money from Security Link have been acquired together with FOI to continue the theoretical work and to build a full scale prototype sensor platform. FOI has further more shown interest for additional common projects if the results of this prototype are promising. The work, based on work with the first prototype has resulted in a conference paper [C8]. And based on the experiences a second prototype has been produced.

  • A PhD student, Albin Andersson Jagesten, was been hired to look at practical and theoretical aspects of using the magnetometer array to provide odometry and as supporting sensor for inertial navigation. The position and supervision was partially funded by this project and partially by the ELLIIT "Local Positioning Systems" project. Unfortunately, Anderson Jagesten decide to end his PhD studies prematurely after just a few months. A new PhD (Chuan Huang) student has now been hired for this project, this time supported by a VR grant with the PI as co-applicant.

  • The PI, together with researchers from LiU, Delft University, the Netherlands, and University of Exeter, the UK, organized a tutorial "Indoor Localization using Magnetic-Fields" at the 11th International Conference on Indoor Positioning and Indoor Navigation, Lloret de Mar, Spain. The results produced within this project is a core component of the tutorial.

  • The PI, together with I. Skog, organized a half day tutorial on how to use magnetic-fields measurements for GNSS denied localization at FOI, Linköping, in the beginning of 2022. Again with the results produced within this project is a core component.

  • A conference paper to FUSION 2022 [C9] in which C. Huang explores the potential of magnetic odomertry as a supporting sensor to reduce the drift in an inertial navigation system.

• Work on the topic of distributed/decentralized estimation has resulted in a number of publication and a MSc.

  • A grant from Security Link has been acquired together, with J. Nygårds (FOI), to study aspects of decentralized estimation.

  • A conference paper presented at FUSION 2018 [C1] and a MSc thesis [MSc1] in cooperation with FOI (J. Nygårds), where methods for decentralized tracking are evaluated. Methods for decentralized estimation are key components needed to deal with the computational and communication complexity of large sensor networks, as well as provide robustness against loss of sensor nodes.

  • A conference paper at FUSION 2019 [C3], in which R. Forsling presented new methods to, in a consistent way, fuse estimates which are potentially correlated, where not all available information is communicated due to limited bandwidth. The results makes it possible to guarantee the consistency of results obtained in these situations.

  • A conference paper at FUSION 2020 [C6] by R. Forsling dealing with how to determine what information to communicate in a distributed sensor network with bandwidth limitations. Methods to approximate the information in the network are presented, and shown to provide good results compared to standard methods in simulations.

  • A licentiate thesis [Lic1] by R. Forsling including and extending the results in [C3, C6] and an initial version of [J7].

  • A conference paper at FUSION 2022 [C10] by R. Forsling on how eigen value decompositions can be used to choose dimension reductions in a decentrailed estimation scheme, this way minimize the need for communication.

  • A journal article [J7] by R. Forsling, in which a framework for conservative estimation is defined. It is shown how several famous methods of conservative estimation are special cases of this framework, and how advances in robust optimization can be used to solve for even more general cases.

• Work with FOI (J. Rantakkoko) about aspects on how to deal with GPS spoofing. The outcome of this includes:

  • A joint MSc project [MSc2], with the PI as examiner, studying how to detect and mitigate GPS spoofing.
  • A conference paper [C4] and a journal article [J8], extending the work in [MSc2] with new theory and actual open sky spoofing experiments.
  • A well attended webinar (100+ participants) organized by the Institute of Navigation (ION) on the topic of GNSS spoofing and mitigation.
  • An ongoing joint MSc project, with the PI as examiner, studying how to how to detect and mitigate GPS spoofing with a single antenna combined with measurements from an inertial measurement unit (IMU).
  • The related licentiate thesis [R‑Lic2] (the PI co-supervisor), which studies the problem of positioning based on radio signals which exhibit multi-path aspects which also has bearing on the GPS spoofing problem.
  • Two joint MSc projects [R‑MSc1, R‑MSc3] with the PI as examiner studying how to obtain autonomously land UAV in a GPS denied environment. A problem in which positioning without GPS is a key aspect.

• The work on combining classic model based estimation techniques with elements from machine learning has resulted in a number of publications and two PhD student projects.

  • An article [J1] together with the PhD student C. Veibäck (supervised by the PI) on how Gaussian process regression can be used to learn (part of) the dynamic model online in multi-target applications. This allows for improved performance in situations where a good dynamic model cannot be derived a priori.

  • An article [J2] together with the PhD student Y. Zhao which explores fundamental limits for RSS based positioning using pre-trained Gaussian process models in the estimation problem. This gives an indication of the potential of this type of system modeling.

  • The PI has acquired a WASP project financing one academic PhD student (Anton Kullberg) including supervision to study how to integrate machine learning components into classical model based estimation techniques. The results in [J1, J2] helped strengthen the application.

  • A conference paper [C5] at FUSION 2020, in which A. Kullberg presented a continuation of [J1] applied to a traffic monitoring scenario. The approach improved tracking in an intersection, as well as provided insights into the traffic patterns.

  • A journal paper [J6] in which A. Kullberg consider aspect related to the computational complexity of the methods in [J1, C5]. Methods to reduce the computational complexity and make the method more practical are provided.

  • A conference paper [C7] at FUSION 2021, in which A. Kullberg uses the results in [J1] to detect anomalous vessel behavior in a harbor.

  • A Licentiate thesis [Lic2] written by A. Kullberg on the topic of joint state estimation and model learning using Gaussian processes.

  • Based on the successful result by A. Kullberg, a follow up WASP project financing one academic PhD student (Seyedfarhad Mirkazemi) including supervision, has been acquired.

• An article [R‑J2] on how to deal with measurements with uncertain timestamps in estimation problems, together with the PhD student Clas Veibäck (supervised by the PI). Uncertain timestamps may arise from using unsynchronized sensors or inexpensive hardware to collect data, as outlined above.

• A PhD thesis [PhD1] by Clas Veibäck (with the PI as main supervisor). The project has been used to partially fund the PI's supervision of the student.

• Synchronization of inertial data for almost-periodic events for gait analysis has been studied with P. Kasebzadeh (co-supervised by the PI, and supported by this project for part of the supervision). The outcome of this includes:

  • A PhD thesis [PhD2] by Parinaz Kasebzadeh.

  • Two articles in respected journals [J3, J4].

• A licentate thesis [R‑Lic1] by Martin Lindfors on the topic of estimation of harmonics. The licentate thesis is co-supervised by Rickard Karlsson (NIRA Dynamics) and the PI. The derived methods are expected to serve as a component in GPS-free car positioning, a common interest of R. Karlsson and the PI. The following publication have also come out of this work:

  • The confernce papers [R‑C2, R‑C3].

  • The journal article [R‑J3].

• The journal article [J5], joint work with R. Karlsson, in which an alternative to the method presented in [R‑Lic1] to estimate the harmonics based on a deep convolutional network (CNN) is pursued. The results are very promising, and additional work to further robustify the method and to enable its use in a Bayesian framework is in progress.

• Work with a small scale MEMS microphone array for sound source localization has been conducted by C. Veibäck (former PhD student of the PI), who has a PostDoc position funded by the Danish hearing-aid manufacturer Oticon. The work has so far resulted in two conference paper [R‑C4, R‑C5] and a MSc thesis [R‑MSc2].

• The Sensor Fusion app has received a number of updates as a result of the collaboration with Mats Amundin (Kolmården Wildlife Park). As a result, it has been an integral part of research projects and in teaching. Teaching aspecs are covered in [R‑C1, R‑J1].

• The PI did during 2018 conduced a prestudy on an infrastructure for collecting and making datasets commonly available within WASP and the WASP research arenas (WARA). As part of this two students worked, supervised by the PI, during the summer at Ericsson developing a prototype of a portal to collect and distribute datasets. The suggested system will provide access to relevant data for the project; and hence simplify the collection and availability of measurements relevant to this project. Work within this project helped secure the prestudy.

• The PI has, together with SAAB, secured a Vinnova project "Multi Sensor Image-Based Navigation" worth MSEK 2.4 that will finance a PostDoc and university supervision at LiU. On top of this SAAB has also commited to support the postdoc with in-kind contributions.

• The PI was general chair for the 25th IEEE International Conference on Information Fusion (FUSION) 2022 in Linköping. The conference had 170 accepted papers and more than 300 participants from all around the world came to visit Linköping, Sweden. Organizing the conference lead to many new and stonger connetions to key players in the community of information fusion.

• The PI is as of October 2021 Navigation and Localization Area Cluster Leader in WASP.

Publications

Project publications

Publications related to the project

Publications tightly coupled with the project, but formally not funded by the project.