Are the aircraft database dumps working?

4 months 4 days ago #1626 by fquad
Hi, I noticed that the aircraft data here is perhaps being cut short, as all months have exactly 460,000 entries. All monthly files also have their "date modified" field updated every day, so perhaps they are also (or only?) being constantly overwritten. Looking at an archived snapshot from July 2018, the file had 465583 entries then. I just hope the history of the database isn't lost.

Please Log in or Create an account to join the conversation.

3 months 4 weeks ago #1632 by strohmeier
Honestly, I never looked at the files in detail. They are definitely not the same but if you are looking into it more deeply, let me know.

Please Log in or Create an account to join the conversation.

3 months 4 weeks ago - 3 months 4 weeks ago #1634 by fquad
It sure looks like the files are being cropped at 460,000 entries, every snapshot has this exact number. They are indeed not same, as the plot below demonstrates, so the overwrite theory should not be true.

 

Comparing the '2020-12' snapshot with the '2021-09' one, 20,413 icao24 codes are present only in the older snapshot. I looked at the 16 type B738 aircraft among those (listed below), and they all appear to still be in OpenSky database accessed through the web (opensky-network.org/aircraft-database), even though they are not present in the current database dump. Some of those are currently flying, so their omission really looks like a bug.

'a05dbc', '424905', 'ab62f8', '4aca54', '4cab9e', '71c232', '750202', '0c20b6', '0c205b', 'a356f6', '4ca891', 'a47d05', '781047', 'e07583', '78044b', 'a9d712'
Attachments:

Please Log in or Create an account to join the conversation.

3 months 2 weeks ago #1640 by jluebbe
I experienced something similar. I added about 50 aircraft but most of them did not appear in the dump even a few days later. However the data is not lost. It stays available in the online version of the database.

Please Log in or Create an account to join the conversation.

1 month 1 week ago #1697 by strohmeier
So, looking into this, a (random) part of the database doesn't get written, likely running out of memory.

Definitely some bad news in that we won't be able to restore that historically. :/ The actual database itself was never affected, including the access via the website interface.

Hope we can fix this soon. For the time being, here's a complete manual dump of the database from today, 520000 lines instead of the capped 460000.

www.dropbox.com/s/ka61swsxtmrzfd8/aircra...atabase.csv.zip?dl=0

Please Log in or Create an account to join the conversation.

Powered by Kunena Forum
This website uses cookies to offer you the best experience of our services. By using this website you agree to our privacy policy!