time_position data downloaded from OpenSky appears to be corrupted

2 years 9 months ago #1534 by w3sz
I am seeing that in the downloaded json object for each aircraft the timestamp ("time_position") is corrupted, being offset from the correct value by ~ +60 seconds, thus placing the timestamps for recently acquired aircraft in the future, with the time stamp indicating that the aircraft generated the data at a time in the future that has not yet occurred.

I see this result both when viewing the data downloaded from OpenSky with my own software and when I use an online epoch converter to convert time_position value received in the json object from opensky-network.org/api/states to human-readable form.

My installation is NIST-synchronized, and I have done the test using other NIST-synchronized platforms independent of my installation, and all tests give the same result, that the OpenSky time_position values are corrupted, with an offset of ~+60 seconds.  So the issue is not due to my installation having incorrect time synchronization.

So, for example, if I am querying the state of an aircraft whose report was actually generated at UTC 14:00:00, then the OpenSky state data reports time_position (index 3) values for that aircraft to be ~14:01:00 UTC.

I do not know when this offset into the future started, as I had not been actively reviewing the data here for some months due to involvement with other projects.  I first noted this on June 22, 2021 as I reinitiated work on a project involving querying the live database.

You can easily prove that this error is present by doing the following:

1. Copy the most recent timestamp contained in the jquery resulting from:
opensky-network.org/api/states/all?lamin...7.8229&lomax=10.5226

2.  Paste the timestamp into the textbox under the heading "Convert epoch to human-readable date and vice versa" at:
www.epochconverter.com/  (Obviously you need to have picked a "recent" timestamp from the query and not a stale one).

3.  Click the "Timestamp to Human date" button next to the textbox into which you put the timestamp.

4.  You will see in the result, "Relative:  In a minute" indicating that the downloaded timestamp is approximately one minute into the future.
File Attachment:


The incorrect time offset in the data downloaded from OpenSky is a corruption of uploaded aircraft data that occurs at the level of the OpenSky server; it is not present in the time data that observers are uploading to the OpenSky server.

Although I first noticed this problem on data that did not originate at my installation, I have since confirmed the fact that the data corruption is occurring at the level of the OpenSky server by using the plane data that I upload to OpenSky.  The timestamps are correct on the plane data that I generate here but when that data returns to me from the OpenSky server, the time data returning from the OpenSky server has been corrupted, with the time offset into the future applied, so that the timestamps are no longer correct when they are received from the OpenSky server.  

This is a major issue for those using the data in a manner that requires accurate timestamps, and it is an issue that should be addressed.

Please Log in or Create an account to join the conversation.

2 years 9 months ago #1535 by w3sz
And as three quick examples of the effect of this corrupted time_position in the OpenSky data,

1. at 20:16:02 a download of data for flight number ACA525 shows:
For FlightAware, time_position 20:15:50 Lat 42.2795 Lng -73.2187
For OpenSky, time_position 20:17:09 Lat 42.275 Lng -73.2179
offset +89 seconds

2. at 20:30:38 a download of data for flight number ABX3163 shows:
For FlightAware, time_position 20:30:14 Lat 40.6882 Lng -75.3831
For OpenSky, time_position 20:31:39 Lat 40.6867 Lng -75.3856
For FlightAware, time_position 20:30:30 Lat 40.6815 Lng -75.3941
offset +69 to +85 seconds

3. at 20:36:36 a download of data for flight number ABX7355 shows:
For FlightAware, time_position 20:36:05 Lat 40.6829 Lng -75.3917
For OpenSky, time_position 20:37:39 Lat 40.6777 Lng -75.4004
For FlightAware, time_position 20:36:21 Lat 40.6765 Lng -75.4023
offset +78 to +94 seconds

You can compare datasets for aircraft of your choosing and find the same result.

Please Log in or Create an account to join the conversation.

Powered by Kunena Forum
This website uses cookies to offer you the best experience of our services. By using this website you agree to our privacy policy!