Gustaf Hendeby

Gustaf Hendeby

Associate Professor and Docent in Automatic Control

(Swe: Universitetslektor och docent i reglerteknik)

LiU
On-Line Appearance:
Gustaf Hendeby's Google Scholar profile Gustaf Hendeby's ResearchGate profile View Gustaf Hendeby's LinkedIn profile

©2021. All rights reserved.

CENIIT Project 17.12:
Applied Research Platform for
Sensor Fusion

Project leader: Gustaf Hendeby

The aim of this project is to create a research group that using probabilistic learning methods addresses the needs introduced by the surge of less expensive sensors, and in the process to complement the existing group with competence in computer science and more knowledge of hardware. The goal of the proposal is to perform agile development of theory for emerging applications with early demonstrators and active involvement of students.

Background and Industrial Relevance

Sensor and information fusion is a mature area with its own society (ISIF) and conference (FUSION). The subject has advanced in stages. In the 1960's, the Apollo program and aerospace applications drove the development. It resulted in algorithms such as the Kalman filter (KF) and the extended Kalman filter (EKF), which are still important corner stones in the field. The next advancement was in the 1990's, when the development of faster computers led to better methods to handle nonlinear problems. The unscented Kalman filter (UKF), a development of the Kalman filter methodology, and the simulation based particle filter (PF), are examples of methods developed during this time. At the same time the focus shifted towards automotive industry, which in many ways can be considered the driving force in the early 2000's.

The basis for this development is the system description \[ \begin{align} x(t+T_s) &= f\bigl(x(t), v(t)\!\bigr)\\ % % y(t) &= h\bigl(x(t)\!\bigr)+e(t) , \end{align} \] where the state \(x(t)\), in a state-space model, should be estimated given the measurements \(y(t)\). Typical assumptions in the classical setting are: Known time, \(t\). The sensor model, \(h(\cdot)\), is idealized. The measurements are well structured and the noise \(e(\cdot)\) independent. And, relatively few measurements are available.

LiU, and the Division of Automatic Control in particular, has a strong position in the field developing both theory and applications. Sensor fusion has been an explicit research area in large research projects such as ISIS, MOVII, LINK-SIC, CADICS, and Security Link since 1995, and more than 200 MSc projects have been preformed in the area. Since 2010 there is an undergraduate course called Sensor Fusion with approximately 40 students attending each year. A success factor has been the ability to combine high quality theoretical work with relevant and successful applications, e.g., in the aerospace and automotive settings. The result has been a number of well cited journal articles and successful spin-offs, such as NIRA Dynamics (market leading in indirect tire pressure monitoring) and Senion (indoor localization solutions).

Since 2010 there has been another shift in attention in the field towards smart devices, including smartphones, augmented and virtual reality (AR, VR), drones, intelligent hearing aids, etc. In the future internet of things (IoT) devices and Industry 4.0 are expected to gain popularity. An enabler is the availability of an abundance of inexpensive sensors and computational resources.

Research challenges

Typically, the price paid with these new sensors is worse quality. The model above must be adapted to reflect the imperfect measurement model, which include unknown parameters, \(\theta\), and uncertain measurement times, \(\tau\), \[ \begin{align} x(t+T_s) &= f(x(t), w(t))\\ % % y\bigl(\tau(t; \theta)\!\bigr) &= h\bigl(x(\tau(t; \theta)\!); \theta\bigr) + e\bigl(\tau(t; \theta); \theta\bigr) . \end{align} \] The problem formulation now includes uncalibrated sensor parameters, \(\theta\), (exact location, time and amplitude wrapping, measurement quantizations and saturation, clock parameters, etc). The measurement, \(y\), is in many cases high-dimensional. And, the data collected often exhibit the `four V' aspects associated with Big Data (volume, velocity, variability and veracity).

Algorithms targeting this type of applications are required to be efficient, as computing power and batteries are limited, and robust, to compensate for individual sensor variations. At the same time, it might be necessary to identify sensor parameters on the fly or create sensor models from data alone using machine learning inspired methods. Offloading computations to the cloud is one option to address limited computing power. To provide general solutions requires skills in sensor fusion and computer science, as well as, familiarity with hardware. Short of this, many of the solutions in use today are rather ad hoc, designed to address a specific problem using a specific sensor.

Project description

The aim of this proposal is to initialize the creation of a research group that performs basic research focusing on challenges introduced by imperfect sensor models and their implications to state estimation, and in parallel demonstrates the results using relevant hardware. The inspiration comes from the PI's experiences working at DFKI and FOI, experiencing the problems described above first hand. The topic is important in order not to miss out on the potential of the new technology, and as a reaction to the increasing importance of being able to demonstrate results in demonstrators to attract funding.

To deal with the imperfect sensor models requires both development of new theory and solving practical problems. The project aims to develop a theoretical framework for probabilistic learning of the sensor models. Data driven machine learning algorithms adapted to the stochastic signal processing framework will be considered as a mean to acquire the necessary models. In some cases, big data aspects must be considered, e.g., efficient methods for offloading some of the computations to the cloud.

The goal of the proposal is to perform agile development of theory for emerging applications with early demonstrators and active involvement of students, e.g., utilizing the Sensor Fusion app to achieve this.

The following areas have been identified as highly relevant and are currently pursued:

An important prerequisite to achieve this is to have access to realistic measurement data, which the Sensor Fusion app provides.

Visions and plans

The vision is to establish a research group that performs basic research on these new challenges and in parallel demonstrates the results using relevant hardware, this way ensuring that LiU keeps its strong position also in emerging applications where low-cost hardware and cloud solutions are increasingly important. A key component is working with real problems and data, which inevitably reveals bottlenecks of existing theory and sparks new research directions.

Dissemination will include papers in the FUSION conference for basic research, application orientated conferences for demonstrators, journal articles for the basic theory with applications, and not the least apps that are spread using Google Play.

Research environment and industrial cooperation

The research project is based on relevant industrial problem formulations. This is manifested in collaborations with representatives of local organizations.

One strategy to ensure relevant problem formulations and to establish contacts with industrial partners is to involve in MSc thesis projects with them. This establishes contacts with relevant people and identifies common interests that can evolve into long-term research collaborations. It is also a good way to test feasibility of ideas. The PI have since the beginning of the project been examiner for 9 MSc theses at FOI, 5 at Veoneer (former Autoliv, currently Arriver, in Linköping), 2 at Saab Dynamics, and one each at Husqvarna AB, Oticon, Saab Aeronatics, Syntronic, UMS Skeldar, UniVRses, and Öhlins Racing AB. Interesing collaborations and problem formulations have resulted from these MSc projects, e.g., joint applications for funding for projects and industrial PhD student.

Major collaborations are outlined below:

Methods to better deal with the computational and communication complexity associated with estimation in large sensor networks using decentralized algorithms are pursued with the Swedish Defence Research Agency (FOI, Doc. Jonas Nygårds). The cooperation includes both joint work which has resulted in a conference paper published at FUSION 2018 [C2] and a jointly supervised MSc thesis project including two students [MSc1]. The FOI side of the project is financed by a FOI internal Security Link project, in which cooperation with the PI on this topic is a key component. Similar problem formulations are of interest to SAAB Aeronatics, and the industrial PhD student Robin Forsling (co-supervised by the PI). He studies the implications of limited communication ability in sensor networks and how to ensure consistent estimates under these conditions. The initial work has result in two papers presented at FUSION 2019 and 2020 [C3, C5] and a licentiate thesis [Lic1].

Magnetic odometry as a method to support positioning in GPS denied environments is being pursued in collaboration with FOI (contact: Doc. Isaac Skog). First results were reported at FUSION 2018 [C2]. Additional funding has been acquired within Security Link to build a full scale prototype of the magnetometer array. Initial experiments with a first version of his hardware were conducted in collaboration between LiU and FOI in the spring 2020. Results have been published in [C8], and a second prototype is under development, financed by FOI. This work has futhermore led to two new contacts at FOI in Kista, Felix Trulsson and Doc. Magnus Lundberg Nordenvaad with which there are ongoing discussions about continued collaborations outside this project.

A new collaboration with Saab AB (contact: Dr. Zoran Sjanic) has been initiated, in which robust navigation in GPS denied environments will be studied. Odometric measturemetns from vision sensors, will be used to support inertial navigation. Funding has been aqcuired for one PostDoc, and the recruitment process is in progress.

In collaborating with FOI methods for detection and mitigation of GPS jamming and spoofing are studied (contact: Jouni Rantakokko). This work was initiated as a MSc project [MSc2], supported by a FOI internal Security Link project based around the cooperation with the PI on the topic. The MSc student now works at FOI, and has continued the work started in the MSc thesis. The collaboration continues and has so far resulted in a joint conference paper [C4], a follow up journal article that has been submitted, as well as yet another MSc thesis student working on the topic at FOI.

Algorithms to robustly detect harmonics in measurements of chassis vibrations have been developed by the PhD student Martin Lindfors, who was jointly supervised by Doc. Rickard Karlsson (NIRA Dynamics) and the PI. This has resulted in two conference papers [R‑C2, R‑C3], the licentiate thesis [R‑Lic1] the fall of 2018, and a follow up journal article [R‑J3]. WASP has provided the main funding for Lindfors work, whereas this project has provided means to extend the supervision and to deepen the cooperation with R. Karlsson. This cooperation funded by this project has, after the end of Lindfors' project, resulted in the development of an alternative method for speed estimation [J5], and joint work on how to use the results in a stand alone positioning system.

The Sensor Fusion app has been used by Mats Amundin (Kolmården Wildlife Park) and his MSc students to collect data used to analyze animal behavior. Their usage has been a driving force for the development of the Sensor Fusion app, and resulted in several challenging datasets exhibiting many of the issues which are within the focus of this project and which will be used to drive the algorithm development forward. The Sensor Fusion app was also used for rapid prototyping in the Vinnova funded project "Digital skallgång" (Digitalized Search and Rescue) to locate people in a search and rescue situation using their electronic signatures. It is also one of the data collection units in the Norrköping financed projecte "SenZoor", which aims to analyze visitor behavioral patterns at Kolmården Wildlife Park.

Altogether, the project intends to strengthen the current research group by introducing a new research platform for sensor fusion utilizing the applicants previous experiences of both theoretical development and industry relevant practical results.

Project Result and Status

The project is progressing on several fronts. Results up until now includes:

Publications

Project publications

[J1]   Y. Zhao, C. Fritsche, G. Hendeby, F. Yin, T. Chen, and F. Gunnarsson. Cramér-Rao Bounds for Filtering Based on Gaussian Process State-Space Models. In IEEE Transactions on Signal Processing, 67(23), December 2019.
[J2]   C. Veibäck, J. Olofsson, T. Lauknes, and G. Hendeby. Learning target dynamics while tracking using Gaussian processes. In IEEE Transactions on Aerospace and Electronic Systems, 56(4):2591-2601, August 2020.
[J3]   P. Kasebzadeh, K. Radnosrati, G. Hendeby, and F. Gustafsson. Joint pedestrian motion state and device pose classification. IEEE Transactions on Instrumentation and Measurement, 69(8):5862-5874, August 2020.
[J4]   P. Kasebzadeh, G. Hendeby, and F. Gustafsson. Asynchronous averaging of gait cycles for classification of gait and device modes. IEEE Sensors Journal, 21(1):529-538, January 2021.
[J5]   R. Karlsson and G. Hendeby. Speed Estimation From Vibrations Using a Deep Learning CNN Approach. IEEE Sensors Letters., March 2021.
[C1]   J. Nygårds, V. Deleskog, G. Hendeby. "Decentralized Tracking in Sensor Networks with Varying Coverage". In Proceedings of the 21st International Conference on Information Fusion, 1661-1667, July 2018.
[C2]   I. Skog, G. Hendeby, F. Gustafsson. "Magnetic Odometry - A Model-Based Approach Using A Sensor Array". In Proceedings of 21st International Conference on Information Fusion, 794-798, July 2018.
[C3]   R. Forsling, Z. Sjanic, F. Gustafsson, and G. Hendeby, "Consistent Distributed Track Fusion Under Communication Constraints". In Proceedings of the 22nd International Conference on Information Fusion, July 2019.
[C4]   N. Stenberg, Erik Axell, J. Rantakokko, and G. Hendeby. GNSS spoofing mitigation using multiple receivers. In Proceedings of IEEE Position Location and Navigation Symposium, Portland, OR, April 2020. Virtual conference (Covid-19 pandemic).
[C5]   A. Kullberg, I. Skog, and G. Hendeby. Learning driver behaviors using a Gaussian process augmented state-space model. In Proceedings of 23th IEEE International Conference on Information Fusion, Virtual conference (Covid-19 pandemic), July 6-9 2020.
[C6]   R. Forsling, Z. Sjanic, F. Gustafsson, and G. Hendeby. Communication efficient decentralized track fusion using selective information extraction. In Proceedings of 23th IEEE International Conference on Information Fusion, Virtual conference (Covid-19 pandemic), July 6-9 2020.
[C7]   A. Kullberg, I. Skog, and G. Hendeby. Learning motion patterns in AIS data and detecting anomalous vessel behavior. In Proceedings of 24th IEEE International Conference on Information Fusion, Sun City, South Africa, Nov 1-4 2021.
[C8]   I. Skog, G. Hendeby, and F. Trulsson. Magnetic-field based odometry - an optical flow inspired approach. In Proceedings of Eleventh International Conference on Indoor Positioning and Indoor Navigation, Lloret de Mar, Spain, November 29–December 2 2021.
[PhD1]   C. Veibäck. Tracking the Wanders of Nature. Dissertations no 1958, Linköping Studies in Science and Technology, SE-581 83 Linköping, Sweden, December 2018.
[PhD2]   P. Kasebzadeh. Learning Human Gait. Dissertations no 2012, Linköping Studies in Science and Technology, SE-581 83 Linköping, Sweden, September 2019. (G. Hendeby co-supervisor.)
[Lic1]   R. Forsling. Decentralized Estimation Using Conservative Information Extraction. Licentiate Thesis 1897, Linköpings tekniska högskola. 2020. (G. Hendeby co-supervisor.)
[MSc1]   T. Fornell and J. Holmberg. Target Tracking in Decentralised Networks with Bandwidth Limitations. Master's Thesis No. LiTH-ISY-EX--18/5172--SE, Department of Electrical Engineering, Linköpings universitet, 2018. (Performed at Swedish Defence Research Agency (FOI).)
[MSc2]   N. Stenberg. Spoofing Mitigation Using Multiple GNSS-Receivers. Master's Thesis No. LiTH-ISY-EX--18/5238--SE, Department of Electrical Engineering, Linköpings universitet, 2019. (Performed at Swedish Defence Research Agency (FOI).)

Publications related to the project

Publications tightly coupled with the project, but formally not funded by the project.

[R‑J1]   G. Hendeby, F. Gustafsson, N. Wahlström, and S. Gunnarsson. Platform for teaching sensor fusion using a smartphone. International Journal of Engineering Education, 33(2B): 781-789, 2017. Special issue on: "Engineering behind technology-based educational innovations".
[R‑J2]   C. Veibäck, G. Hendeby, and F. Gustafsson. Uncertain timestamps in linear state estimation. In IEEE Transactions on Aerospace and Electronic Systems, 55(3):1334-1346, June 2019.
[R‑J3]   M. Lindfors, G. Hendeby, F. Gustafsson, and R. Karlsson. Frequency tracking of wheel vibrations. IEEE Transactions on Control Systems Technology, April 2020. Early access.
[R‑C1]   G. Hendeby, F. Gustafsson, and N. Wahlström. Teaching sensor fusion and Kalman filtering using a smartphone. In Proceedings of 19th IFAC World Congress, Cape Town, South Africa, Aug. 2014.
[R‑C2]   M. Lindfors, G. Hendeby, F. Gustafsson, and R. Karlsson, "Vehicle Speed Tracking Using Chassis Vibrations". In Proceedings of the 2016 IEEE Intelligent Vehicles Symposium, 214-219, June 2016.
[R‑C3]   M. Lindfors, G. Hendeby, F. Gustafsson, and R. Karlsson. "On Frequency Tracking in Harmonic Acoustic Signals". In Proceedings of the 2th International Conference on Information Fusion, July 2017.
[R‑C4]   C. Veibäck, M. A. Skoglund, F. Gustafsson, and G. Hendeby. Sound source localization and reconstruction using a wearable microphone array and inertial sensors. In Proceedings of 23th IEEE International Conference on Information Fusion, Virtual conference (Covid-19 pandemic), July 6-9 2020.
[R‑Lic1]   M. Lindfors. Frequency Tracking for Speed Estimation. Licentiate Thesis 1815, Linköpings tekniska högskola. 2018. (R. Karlsson supervisor and G. Hendeby co-supervisor.)
[R‑Lic2]   A. Bergstöm. Timing-Based Localization using Multipath Information. Licentiate Thesis 1867, Linköpings tekniska högskola. 2020. (G. Hendeby co-supervisor.)
[R‑MSc1]   Andersson Jagesten, A. and Källström, A. Autonomous Landing of an Unmanned Aerial Vehicle on an Unmanned Ground Vehicle in a GNSS-denied scenario. Master's Thesis No. LiTH-ISY-EX--20/5327--SE, Department of Electrical Engineering, Linköpings universitet, 2020. (Performed at the Swedish Defence Research Agency (FOI).)
[R‑MSc2]   Fredriksson, A. and Wallin, J. Mapping of an Auditory Scene Using Eye Tracking Glasses. Master's Thesis No. LiTH-ISY-EX--20/5330--SE, Department of Electrical Engineering, Linköpings universitet, 2020. (Performed at Oticon.)