Issues with access the table state_vectors_data4 from the Impala shell.

1 year 2 months ago #1693 by Herson
Good morning all at Support,

Over the past 2 days I've been experiencing problems when trying to extract a small amount of data from the state_vectors_data4 table using the Impala shell.  

This morning a tried the following query which worked fine.
select *, 'OXTED' as area_of_record  FROM state_vectors_data4 where hour between 1635811200.0 and 1638489600.0 limit 10;

Then I tried the query:
 select count(*), 'OXTED' as area_of_record  FROM state_vectors_data4 where hour between 1635811200.0 and 1638489600.0 limit 10;

But this time I got the error message;


[hadoop-29:21000] > select count(*), 'OXTED' as area_of_record  FROM state_vectors_data4 where hour between 1635811200.0 and 1638489600.0 limit 10;
ERROR: Disk I/O error: Failed to open HDFS file hdfs://nameservice1/user/opensky/tables_v4/state_vectors/hour=1637575200/part-r-00169-107c871c-f984-4aed-8055-e3b59690f5f0.snappy.parquet
Error(2): No such file or directory
Root cause: RemoteException: File does not exist: /user/opensky/tables_v4/state_vectors/hour=1637575200/part-r-00169-107c871c-f984-4aed-8055-

Is the possibly a sever disk space or memory issue?

Thanks very much.

Richard.

Please Log in or Create an account to join the conversation.

1 year 2 months ago - 1 year 2 months ago #1695 by Herson
I think I've identified the problem there are two corrupt hourly partitions in the state_vectors_data4 table i.e between Mon Nov 22 2021 10:00:00 GMT+0000 and Mon Nov 22 2021 11:00:00 GMT+0000 which is causing Impala queries to fail e.g.


select count(*), 'OXTED' as area_of_record FROM state_vectors_data4 where hour between 1637575200 and 1637575200 and lat between 51.24162 and 51.27058 and lon between -0.016063 and 0.029119 ;

Please Log in or Create an account to join the conversation.

1 year 2 months ago #1696 by allan.tart
Our dev.ops guys had a look and now the query should work as intended.
Thanks Richard for noticing and notifying.

BR
Allan
The following user(s) said Thank You: strohmeier

Please Log in or Create an account to join the conversation.

Powered by Kunena Forum
This website uses cookies to offer you the best experience of our services. By using this website you agree to our privacy policy!