Missing historic data - 01/Oct/2022 onwards.

The Impala state_vectors_data4 table appears to have a problem  (Disk I/O error: Failed to open HDFS file hdfs) from the 1st Oct. 2022 onwards.

Proof:
 
select count(*), FROM_unixtime(hour, 'yyyy/MM/dd') as rth_date, 'WINCHESTER-WEST' as area_of_record
FROM state_vectors_data4 where lat between 51.06152 and 51.09048
and lon between -1.366071 and -1.320889
and hour between (extract(epoch from now()) - (3600*24*20)) and (extract(epoch from now()) - (3600*24*1))
group by rth_date order by rth_date desc;

Result:  WARNINGS: Disk I/O error: Failed to open HDFS file hdfs://nameservice1/user/opensky/tables_v4/state_vectors/hour=1664654400/part-r-00169-d4feef3e-9220-4003-90bb-27152080c4f0.snappy.parquet
Error(2): No such file or directory
Root cause: RemoteException: File does not exist: /user/opensky/tables_v4/state_vectors/hour=1664654400/part-r-00169-d4feef3e-9220-4003-90bb-27152080c4f0.snappy.parquet

However, the following query does work:

select count(*), FROM_unixtime(hour, 'yyyy/MM/dd') as rth_date, 'WINCHESTER-WEST' as area_of_record
FROM state_vectors_data4 where lat between 51.06152 and 51.09048
and lon between -1.366071 and -1.320889
and hour between (extract(epoch from now()) - (3600*24*20)) and (extract(epoch from now()) - (3600*24*4))
group by rth_date order by rth_date desc;


Thanks all.

Richard Herson - Founder of www.AircraftTrafficSurvey.com, a Not-For-Profit organisation serving UK communities.
 

Problem now resolved. Thanks.

Richard Herson - Founder of www.AircraftTrafficSurvey.com, a Not-For-Profit organisation serving UK communities.