Historic data: Impala: state_vectors_data4 table (10-Aug-23)

Hi All,

I would like to report an issue with the Impala state_vectors_data4 table.  I.e. Disk I/O error: Failed to open HDFS file hdfs://nameservice1/user/opensky/tables_v4/state_vectors/hour=1690815600/part-r-00169-49b3294c-eb8a-4971-8da6-cb45e86d6cdd.snappy.parquet

Currently it's not possible to can't obtain any data for any date from the above table.

Thanks all.

Richard - Founder of  www.AircraftTrafficSurvey.com  , a Not-For-Profit organisation serving our UK communities.

Proof: 

select count(*), FROM_unixtime(hour, 'yyyy/MM/dd') as rth_date, 'WINCHESTER-WEST' as area_of_record
FROM state_vectors_data4 where lat between 51.06152 and 51.09048
and lon between -1.366071 and -1.320889
and hour between ((unix_timestamp( from_timestamp(now(), 'yyyy-MM-dd'))) - (3600*24*10)) and (extract(epoch from now()) - (3600*24*0))
group by rth_date order by rth_date desc;

 

Same issue here, my usual request results in the following error:
       

Same issue here, my usual request results in the following error:
       

It still does not work this morning. The timestamp in the missing file name's path corresponds to 7th of Aug. 16:00, I assume data is missing from this point on and may be added later.

I had had an issue with missing data towards end of July, and that seems now fixed.

I'm now all data for all dates apart from the 7th Aug 2023.

For the 7th Aug 2023 I'm getting the following error;

File does not exist: /user/opensky/tables_v4/state_vectors/hour=1691380800/part-r-00169-ae56a1a6-62ab-4235-bfda-3bc115d5c3a9.snappy.parquet

Proof:

select count(*), FROM_unixtime(hour, 'yyyy/MM/dd') as rth_date, 'WINCHESTER-WEST' as area_of_record
FROM state_vectors_data4 where lat between 51.06152 and 51.09048
and lon between -1.366071 and -1.320889
and hour between ((unix_timestamp( from_timestamp(now(), 'yyyy-MM-dd'))) - (3600*24*10)) and (extract(epoch from now()) - (3600*24*10))
group by rth_date order by rth_date desc;

Thanks.

Richard.

Richard - Founder of    www.AircraftTrafficSurvey.com   , a Not-For-Profit organisation serving our UK communities.