Issues with access the table state_vectors_data4 from the Impala shell.

Good morning all at Support,

Over the past 2 days I've been experiencing problems when trying to extract a small amount of data from the state_vectors_data4 table using the Impala shell.  

This morning a tried the following query which worked fine.
select *, 'OXTED' as area_of_record  FROM state_vectors_data4 where hour between 1635811200.0 and 1638489600.0 limit 10;

Then I tried the query:
 select count(*), 'OXTED' as area_of_record  FROM state_vectors_data4 where hour between 1635811200.0 and 1638489600.0 limit 10;

But this time I got the error message;

[hadoop-29:21000] > select count(*), 'OXTED' as area_of_record  FROM state_vectors_data4 where hour between 1635811200.0 and 1638489600.0 limit 10;
ERROR: Disk I/O error: Failed to open HDFS file hdfs://nameservice1/user/opensky/tables_v4/state_vectors/hour=1637575200/part-r-00169-107c871c-f984-4aed-8055-e3b59690f5f0.snappy.parquet
Error(2): No such file or directory
Root cause: RemoteException: File does not exist: /user/opensky/tables_v4/state_vectors/hour=1637575200/part-r-00169-107c871c-f984-4aed-8055-

Is the possibly a sever disk space or memory issue?

Thanks very much.

Richard.

I think I've identified the problem there are two corrupt hourly partitions in the state_vectors_data4 table i.e between Mon Nov 22 2021 10:00:00 GMT+0000 and Mon Nov 22 2021 11:00:00 GMT+0000 which is causing Impala queries to fail e.g.


select count(*), 'OXTED' as area_of_record FROM state_vectors_data4 where hour between 1637575200 and 1637575200 and lat between 51.24162 and 51.27058 and lon between -0.016063 and 0.029119 ;