access Impala from Python?

6 years 8 months ago - 6 years 7 months ago #256 by AllenDowney
Hi All. I would like to be able to look up an icao24 and get all StateVectors between two timestamps. The example on this page:

opensky-network.org/impala-guide

Shows how to do that using the Impala shell. I can run this query from the Impala shell and it works:

[hadoop-1:21000] > SELECT COUNT(*) FROM state_vectors_data4 WHERE icao24='a0d724' AND time>=1480760100 AND time<=1480764600 AND hour>=1480759200 AND hour<=1480762800;

Now I'd like to do the same thing from Python. I'm using impyla, which is here:

github.com/cloudera/impyla

And I'm trying this code, based on one of their examples:

from impala.dbapi import connect
conn = connect(host='data.opensky-network.org', port=2230,
user=user, password=password)
print(conn)
cur = conn.cursor()
print(cur)
cur.execute('SHOW TABLES')
res = cur.fetchall()

It gets to the first print statement, and prints:

<impala.hiveserver2.HiveServer2Connection object at 0x7f0b416da090>

So that's good so far. Then it hangs when running conn.cursor().

Does anyone have experience with this?

Alternatively, it looks like I can use the OpenSkyApi. I can run this example, and it works:

from opensky_api import OpenSkyApi
api = OpenSkyApi(username=user, password=password)
s = api.get_states(icao24='a65c95')
print(s)

But it doesn't look like that API allows me to get historical data from a time interval. Or does it?

Any help would be appreciated!

Allen

Please Log in or Create an account to join the conversation.

6 years 6 months ago #309 by fuchs
Replied by fuchs on topic access Impala from Python?
Hi Allen,

for security reasons Impala access is not supported through impyla or any other Impala client library for the moment. If you want to store results I recommend using the method described in Section 4 of the Impala Guide .

For the moment this is the only way to access the historical data set as a whole. The API supports retrieving historical data, but only for your own receivers. This is quite odd, I know, but we might open up the API for researchers in the future!

Hope that helps!

Markus
The following user(s) said Thank You: AllenDowney

Please Log in or Create an account to join the conversation.

6 years 1 month ago #431 by schaefer
Replied by schaefer on topic access Impala from Python?
ONERA's Xavier Olive wrote a Python client for our Impala shell, just in case you are still interested:
github.com/xoolive/opensky_data

Please Log in or Create an account to join the conversation.

Powered by Kunena Forum
This website uses cookies to offer you the best experience of our services. By using this website you agree to our privacy policy!