Missing historic data - 31/Aug/2023 onwards.

Hi All,

I would like to report an issue with the Impala state_vectors_data4 table.  I.e. WARNINGS: Disk I/O error: Failed to open HDFS file hdfs://nameservice1/user/opensky/tables_v4/state_vectors/hour=1693796400/part-r-00169-da57d494-1a00-45e3-9ecc-6a49055dbd16.snappy.parquet
Error(2): No such file or directory

Currently it's not possible to can't obtain any reliable data beyond the 30-Aug-2023

Thanks all.

Richard - Founder of  www.AircraftTrafficSurvey.com  , a Not-For-Profit organisation serving our UK communities.

Proof: 

The following dates (dd-mm-yyyy) appear incomplete within the state_vectors_data4 Impala table.

31-08-2023 (i/o issue)
01-09-2023 (i/o issue)
02-09-2023 (i/o issue)
03-09-2023 (i/o issue)
04-09-2023 (i/o issue)
05-09-2023 (i/o issue)
06-09-2023 (i/o issue)
07-09-2023 (i/o issue)
08-09-2023 (i/o issue)
09-09-2023 (looks like some hours are missing)
10-09-2023 (i/o issue)
11-09-2023 (looks like some hours are missing)
12-09-2023 (looks like some hours are missing)
15-09-2023 (looks like some hours are missing)
16-09-2023 ((i/o issue)
17-09-2023 (no data)

Is the intention to fix the above or is the data now lost?

Thanks all.

Richard - Founder of   www.AircraftTrafficSurvey.com   , a Not-For-Profit organisation serving our UK communities.

Proof: 

select count(*) as DCounter, '16-09-2023' as vDate  FROM state_vectors_data4 where  hour between 1694822400.0 and 1694908800.0;

 

FYI: the latest info regarding the completeness of the state_vectors_data4 Impala table.   

It would be good to have an update shortly as there is lots of interest in this valuable dataset especially from communities trying to understand and contribute to the UK Airspace redesign/modernisation (ACOG).  Please let me know should you require any technical assistance.

Date (dd/mm/yy)     State Vector count     % populated
26/09/2023                       0                                  0
25/09/2023                       0                                  0
24/09/2023             86,078,853                          12
23/09/2023             89,860,718                          12
22/09/2023           149,724,543                          21
21/09/2023           119,151,531                          16
20/09/2023           227,448,306                          31
19/09/2023           320,499,859                          44
18/09/2023            84,881,960                           12
17/09/2023          341,325,985                           47
16/09/2023                I/O Error                            0
15/09/2023                I/O Error                            0
14/09/2023          768,424,985                         106
13/09/2023               I/O Error                             0
12/09/2023           77,675,841                            11
11/09/2023         181,679,038                            25
10/09/2023                I/O Error                           0
09/09/2023         472,339,981                          65
08/09/2023               I/O Error                           0
07/09/2023               I/O Error                           0
06/09/2023               I/O Error                           0
05/09/2023               I/O Error                           0
04/09/2023               I/O Error                           0
03/09/2023               I/O Error                           0
02/09/2023               I/O Error                           0
01/09/2023               I/O Error                           0
31/08/2023               I/O Error                           0
30/08/2023               724,751,987                    100
29/08/2023               724,232,690                    100
28/08/2023               711,929,996                    100
27/08/2023               713,033,821                    100

Thanks all.

Richard - Founder of     www.AircraftTrafficSurvey.com  a Not-For-Profit organisation serving our UK communities.
 

Please could I request another update as the state_vectors_data4 Impala table does not contain any data for Oct 23?

Is there an alternative dataset that provides the latest 30 to 60 days history in a similar format as the state_vectors_data4 table/view?  I'm trying to prevent duplication of processing.

Thanks all.

Richard - Founder of        www.AircraftTrafficSurvey.com   a Not-For-Profit organisation serving our UK communities.