Aircraft Localization Competition

2019 Aircraft Localization Competition

Call for Contesters

Air traffic control (ATC) is the backbone of what is arguably the key means of personal transport in the modern world. The key issue in controlling the airspace is to know where an aircraft is at any given time. Recent technological developments and stricter separation needs have given rise to new methods of aircraft localization, most notably the automatic dependent surveillance–broadcast protocol (ADS-B).

In ADS-B, aircraft simply report their exact location (determined with onboard GPS sensors) to ground stations periodically. While this approach has many advantages, the transfer of control over the reported location to the aircraft, however, brings a number of safety and security issues along. To mitigate these issues, complementary or redundant localization methods are needed that are independent of the aircraft.

At the same time, crowdsourced air traffic communication networks have gained importance over the past decade. Large private companies such as FlightRadar24 and FlightAware, research networks such as the OpenSky Network, and increasingly flight authorities themselves rely less and less on planned deployments of ATC receivers. Instead, they use distributed networks that are randomly deployed. Contrary to traditional, carefully planned receiver networks, this crowdsourced use of cheap sensors provides a number of new challenges to existing localization algorithms.

This competition is about finding the best methods to localize aircraft based on crowdsourced air traffic control communication data. The data is collected by the OpenSky Network, a large-scale ADS-B sensor network for research. OpenSky was first presented at the IEEE/ACM IPSN conference in Berlin in 2014 in this paper. As of today, the OpenSky Network continuously collects air traffic control data from thousands of aircraft. This data is received and streamed to Opensky over the Internet by a crowd operating more than 1000 sensors. The goal of the competition is to determine the positions of all aircraft which do not have position reporting capabilities or may report wrong locations. To do so, competitors will rely on time of arrival and signal strength measurements reported by many different sensors. Although methods like multilateration are long known, this data poses new challenges because most of the low-cost sensors are not time synchronized or calibrated. Competitors will therefore have to face different kinds of noise, ranging from clock drifts, inaccurate sensor locations, or broken timestamps due to software bugs.

How does the competition work?

Execution

We will provide labelled training datasets which include all aircraft locations well before the conference. These labelled data sets can be used by the participating teams to train their models. At the conference, each team has to send at least 1 team member to the conference where they will get access to a non-labelled evaluation data sets. The teams have then 9h to find all locations of aircraft that are missing location information in the data sets. Every 3 hours, the teams have to submit their intermediate results (as a CSV file) to the organizers. The organizers will then calculate an indicator of the accuracy of their solution and provide an intermediate ranking. After 9 hours, the teams submit their final results and the final ranking is determined.

Who are we looking for?

Eligibility

We encourage both individuals and teams from academia and industry to register and participate. We strongly emphasize our openness towards novel approaches (such as machine learning) but also allow competitors to adapt their "traditional" localization models to the peculiarities of the crowdsourced measurement data. The localization algorithms should be able to produce decent results from a fresh 1h data set (~1 GB CSV) in under 3 hours.

The competition will take place if at least 5 teams respond to this preliminary call for competition. Exempted from participation are the organizers and members of the OpenSky Network Association.

Are there different levels of difficulty?

Categories

Participants will get labelled training data to prepare for the competition day. For both training phase and competition day, we will prepare different data sets with varying challenges. For instance, some of these data sets will contain information on the height of the aircraft (barometric altitude) whereas others are missing these information. In addition, we will provide data sets containing data from GPS-synchronized receivers only. Given these differences, we will have the following four categories with varying levels of difficulty.

Category 1: Synchronized Receivers + Altitude Information

In this category, no effort has to be put into sensor time synchronization and the rough geometric height of the aircraft can be estimated based on the barometric altitude. These conditions simplify things a lot and the prizes in this category will therefore be much lower than in the other categories.

Category 2: Synchronized Receivers + No Altitude Information

Still, no effort has to be put into sensor time synchronization. However, in this category, competitors will have no information on the height of the aircraft. Due to the mostly bad vertical dilution of precision, achieving good results here will be more difficult. Therefore, prizes will be higher than in the first category.

Category 3: Unsynchronized Receivers + Altitude Information

In this category, only some receivers will provide GPS synchronized timestamps while others experience clock drifts or even broken timestamps. The competitors will have to put efforts into synchronizing the timestamps of all the unsynchronized sensors and filtering of broken timestamps to get decent results. However, they can improve their localization algorithms by incorporating the barometric altitude information. This category is considered harder than categories 1 and 2 and prizes will therefore be higher.

Category 4: Unsynchronized Receivers + No Altitude Information

In this most challenging category, only some receivers will provide GPS synchronized timestamps while others experience clock drifts or even broken timestamps. The competitors will have to put efforts into synchronizing the timestamps of all the unsynchronized sensors and filtering of broken timestamps to get decent results. This is considered to be the hardest category, since no information about the altitude of the aircraft will be provided. Therefore, prizes will be the highest in this category.

Important note: Teams can compete in up to two categories at the same time. However, categories will only be eligible if there are at least 5 competing teams. We might ask teams to join other categories if there are not enough participants.

What's in there for the best solutions?

Awards

On the competition day (April 15), all teams have to submit their intermediate results to the organizers every three hours. These intermediate results will then be rated using an objective error metric and the scores will be published. The teams can then continue to improve their results, e.g., by further pre-filtering the data or improving their models ad hoc.

The leading teams of each intermediate evaluation (i.e. after 3h and 6h) will receive increasing cash awards. The awards will reach their maximum at the 9h deadline, when all teams have to submit their final results. The best 3 teams of each category will also receive cash prizes for their final results.

The cash prizes are scaled with the level of competition and the difficulty of the category. This means that the more competitors and the higher the difficulty in a category, the higher the prizes. The specific prizes are as follows:

The teams with the best final results (after 9h) of each category will have to present their solutions in a short presentation at the conference. In addition, all teams who won cash awards (intermediate or final prizes) will have to publish their code under the GNU GPLv3 on the OpenSky Network's github account. Teams that do not want to publish their code are not eligible for awards. This means that closed-source solution can also compete but they will not be eligible for cash prizes.

More details about the specific evaluation metric will follow soon.

What kind of data will I work with?

Data Sets

The data sets will be provided as CSV files. Each CSV file contains the data that was recorded by the OpenSky Network over a duration of 1h and has a (uncompressed) size of about 1GB. Each row in the CSV file represents the reception of one aircraft position report and contains the following information:

1. A unique aircraft identifier
2. Unix timestamp indicated when the message was received by OpenSky
3. Unique identifiers of all sensors which received this signal
4. Nanosecond timestamps from each of the sensors
5. Signal strength measurements from each of the sensors
6. The position of the aircraft (latitude, longitude, height)
7. The barometric altitude of the aircraft

The position of the aircraft (6.) will be empty for those aircraft that need to be localized. However, it will be present for many other aircraft and can be used to synchronize the receiver clocks. The barometric altitude (7.) will only be available to category 1 and 3 competitors (see categories). In the training data sets, all columns will be present.

In addition to this set of measurement data, we will provide meta data for all sensors. The meta data includes:

1. A unique sensor identifier
2. The position of the sensor (latitude, longitude, height)
3. The type of hard- and software

It is worth noting here that the positions of the sensors (2.) are of varying accuracy. The sensor positions have only been entered by the user when the sensor was added to the network and there is no guarantee for correctness or accuracy. While some users report accurate positions for their antennas (e.g., measured with their smartphone), others just provide a rough estimate based on services like Google Maps. Some might even report wrong locations intentionally for privacy reasons.

More detailed information on the data that is provided for the competition can be found here.

Note: the organizers reserve the right to change these rules at any time. In case such a modification is necessary, we will notify all registered participants via email. Any violation to these rules leads to disqualification and immediate loss of eligibility for the awards.

Competition Entry

In order to participate in the competition, you need to register before February 22, 2019. You can participate alone or in a team. If you want to compete in a team, all team members need to register for the competition using the same team name. Individual participants can only be a member of one team at a time.

Categories

Each team (or individual) can compete in up to two of the four categories. The categories have to be chosen on registration and cannot be changed afterwards. Categories will only be eligible if there are at least 5 competing teams. In case a category does not have enough participants, we will ask the registered participants to join another category.

Ranking

Each team (or individual) will have to submit their results (the aircraft positions) for each category during the competition day in a pre-defined format. The results will be ranked by an objective evaluation metric such as the average 3D error. The evaluation metric will be announced prior to the evaluation day. The winners are determined by that ranking.

Winner License

The winning solutions will only be eligible for cash awards if all source codes and additional data sets used to generate the results from the measurement data are published under the GNU GPLv3 license on the OpenSky Network's github account. In addition, sufficient documentation must be provided to understand and reproduce the results.

External Data Sets

Usage of any external data sets (e.g., weather data or tracking data from other sources) requires explicit permission by the organizers. In addition, to ensure fairness, any external data set that will be allowed for the competition must be shared with all other teams at least 1 month prior to the evaluation day, i.e. before March 15, 2019.

Original Solution

Teams (or individuals) are only allowed to use their own solutions. Re-using any existing solution is only allowed if the original authors grant you the rights to use their solution and if you made significant modifications to the algorithm or model. In particular, simply re-using existing code and rewriting the data input and output mechanism is not sufficient. Adding parameters to the model and modifying filters to match the specific pecularities of the data, however, can be considered sufficient. Nevertheless, in case you re-used an existing localization solution, please contact the organizers prior to the evaluation day to get written approval.

• 

  Matthias Schäfer (Main Contact)

  TU Kaiserslautern / SeRo Systems GmbH, Germany

  Dr. Matthias Schäfer is currently a lecturer and researcher at distributed computer systems lab (DISCO) at University of Kaiserslautern, Germany. Until 2018, he was a PhD student supervised by Prof. Dr.-Ing. Jens B. Schmitt. Before that, he worked for the Information Technology and Cyberspace group of armasuisse S+T and visited the University of Oxford (hosted by Prof. Ivan Martinovic) in 2012. He is also a co-founder and board member of the OpenSky Network association and managing director of SeRo Systems GmbH. His current research interests are security of air traffic control systems and crowdsourcing.

• 

  Vincent Lenders

  armasuisse, Switzerland

  Dr. Vincent Lenders is a Research Director for Cyber and Information and the Head of the C4I Networks group at armasuisse. He received the M.Sc. and Ph.D. degrees in electrical engineering and information technology from ETH Zurich. He was postdoctoral research fellow at Princeton University. Dr. Vincent Lenders is the co-founder and in the board of the OpenSky Network and Electrosense associations. His current research interests are in the fields of cyber defense, information management, big data, and crowdsourcing.

• 

  Mauro Leonardi

  University of Rome Tor Vergata, Italy

  Dr. Mauro Leonardi received his Ph.D. in October 2003. Since 2004, he is assistant professor at Tor Vergata University in Rome where he teaches “Fundamentals Of Radar And Localization” and “Satellite Navigation and Surveillance systems”. From 2015 to 2017, he was also an Adjunct Professor in the Academy of the Italian Finance Police (Accademia della Guardia di Finanza), teaching radar fundamentals and navigation in the course "Radar and radio techniques". His main research activities are focused on Radar, Air Traffic Control, Satellite Navigation, Positioning and Localization. These activities involve also research on signal processing, estimation theory, and statistical models.

• 

  Martin Strohmeier

  University of Oxford, UK / armasuisse, Switzerland

  Dr. Martin Strohmeier received the M.Sc. degree in Engineering and Management from TU Kaiserslautern, Germany, in 2012 and his Ph.D. from the University of Oxford, UK in 2016. He was a visiting researcher with Lancaster University’s InfoLab21 and Deutsche Lufthansa AG. He holds a Junior Research Fellowship at the University of Oxford and is a Scientific Project Manager at armasuisse in Switzerland. His research interests include wireless security, air traffic communication, and establishing trust in legacy critical infrastructures.

• 

  Fabio Ricciato

  OpenSky Network, Switzerland

  Dr. Fabio Ricciato received the PhD degree from University La Sapienza, Italy, in 2003. In 2004, he joined the Telecommunications Research Center Vienna, first as a senior researcher and then as a key researcher. Between 2007 and 2013 he was assistant professor with the University of Salento, Italy, and later served as head of the Business Unit in the Austrian Institute of Technology. Between 2015 and 2017 he was a professor in the Faculty of Computer and Information Science, University of Ljubljana. His research interests include radio localization, software-defined radio and privacy-preserving computation. He is currently a civil servant for the statistical office of the European Commission. Since 2016, he is an honorary member of the OpenSky Network Association, where he acts as a scientific advisor on a voluntary basis.

Credits & Contact

Icons created by Freepik, Macrovector, brgfx, and GraphiqaStock and published on freepik.
Website design by HTML5 UP.

For questions and comments regarding this website, please contact Matthias Schäfer.