Are the aircraft database dumps working?

Hi, I noticed that the aircraft data here is perhaps being cut short, as all months have exactly 460,000 entries. All monthly files also have their "date modified" field updated every day, so perhaps they are also (or only?) being constantly overwritten. Looking at an archived snapshot from July 2018, the file had 465583 entries then. I just hope the history of the database isn't lost.

It sure looks like the files are being cropped at 460,000 entries, every snapshot has this exact number. They are indeed not same, as the plot below demonstrates, so the overwrite theory should not be true.

 

Comparing the '2020-12' snapshot with the '2021-09' one, 20,413 icao24 codes are present only in the older snapshot. I looked at the 16 type B738 aircraft among those (listed below), and they all appear to still be in OpenSky database accessed through the web (opensky-network.org/aircraft-database), even though they are not present in the current database dump. Some of those are currently flying, so their omission really looks like a bug.

I experienced something similar. I added about 50 aircraft but most of them did not appear in the dump even a few days later. However the data is not lost. It stays available in the online version of the database.

Where is the broken script performing the dump?

The manual dumps are missing some columns, for example:

$ grep 01002c aircraft-database-2022-04.csv 
"01002c","SU-GBT","AIRBUS","Airbus","A321 231","A321","680","","L2J","Egyptair","EGYPTAIR","MSR","MS","Egypt Air","No ADS-B Emitter Category Information"
$ grep 01002c ~/Downloads/aircraftDatabase.csv 
"01002c","SU-GBT","AIRBUS","Airbus","A321 231","A321","680","","L2J","Egyptair","EGYPTAIR","MSR","MS","Egypt Air","D-AVZB","1997-05-30","","","1997-01-01","","","","false","false","false","","No ADS-B Emitter Category Information"
 

It seems this is now working again (file has grown from 80MB to 130MB), however, I'm now seeing duplicate entries with different information, for the same icao24, but just differing in case. E.g:

"aa6b14","N770X","LEARJET","Learjet Inc","60","LJ60","60-336","","L2J","","","","","Mds Aviation Llc","","","2029-07-31","","2008-01-01","","","P&W CANADA PW305A","false","false","false","",""
...
"AA6B14","N770X","LEARJET","","","LJ60","","","L2J","","","","","","","","","","","","","","false","false","false","",""

In the old dumps, there didn't appear to be any icao24s in upper case.

Should these be merged in the db with case being enforced for new entries?