Missing historic data - 01/Oct/2022 onwards.

3 months 3 weeks ago #2134 by Herson
The Impala state_vectors_data4 table appears to have a problem  (Disk I/O error: Failed to open HDFS file hdfs) from the 1st Oct. 2022 onwards.

Proof:
 
select count(*), FROM_unixtime(hour, 'yyyy/MM/dd') as rth_date, 'WINCHESTER-WEST' as area_of_record
FROM state_vectors_data4 where lat between 51.06152 and 51.09048
and lon between -1.366071 and -1.320889
and hour between (extract(epoch from now()) - (3600*24*20)) and (extract(epoch from now()) - (3600*24*1))
group by rth_date order by rth_date desc;

Result:  WARNINGS: Disk I/O error: Failed to open HDFS file hdfs://nameservice1/user/opensky/tables_v4/state_vectors/hour=1664654400/part-r-00169-d4feef3e-9220-4003-90bb-27152080c4f0.snappy.parquet
Error(2): No such file or directory
Root cause: RemoteException: File does not exist: /user/opensky/tables_v4/state_vectors/hour=1664654400/part-r-00169-d4feef3e-9220-4003-90bb-27152080c4f0.snappy.parquet

However, the following query does work:

select count(*), FROM_unixtime(hour, 'yyyy/MM/dd') as rth_date, 'WINCHESTER-WEST' as area_of_record
FROM state_vectors_data4 where lat between 51.06152 and 51.09048
and lon between -1.366071 and -1.320889
and hour between (extract(epoch from now()) - (3600*24*20)) and (extract(epoch from now()) - (3600*24*4))
group by rth_date order by rth_date desc;


Thanks all.

Richard Herson - Founder of www.AircraftTrafficSurvey.com, a Not-For-Profit organisation serving UK communities.
 

Please Log in or Create an account to join the conversation.

3 months 3 weeks ago #2135 by Herson
Problem now resolved. Thanks.

Richard Herson - Founder of www.AircraftTrafficSurvey.com, a Not-For-Profit organisation serving UK communities.

Please Log in or Create an account to join the conversation.

Powered by Kunena Forum
This website uses cookies to offer you the best experience of our services. By using this website you agree to our privacy policy!