Inconsistent results and hanging connections via pyopensky

1 year 2 months ago #1671 by dkovar
Good evening,

I was reliably using pyopensky to download large volumes of data over many weeks. I now have three problems:

1) A week or so ago my connections would report:
"Server connection successful!"
and then hang completely, not producing records or errors, or terminating.

2) When it does work, I often see:* Obtaining details of the query...
* OpenSky Impala: 41 of records found.
* Fetching records...
* No record found.So, it finds records and then when it tries to fetch them no records are produced

3) If I make exactly the same query a few minutes apart, it reports a different number of records found:

"2021-09-01 00:00:00 2021-09-01 00:30:00 [25.958, 26.498, -98.522, -97.982]
* Obtaining details of the query...
* OpenSky Impala: 43 of records found.
* Fetching records...
* No record found."

"* Server connection successful!
2021-09-01 00:00:00 2021-09-01 00:30:00 [25.958, 26.498, -98.522, -97.982]
* Obtaining details of the query...
* OpenSky Impala: 44781 of records found.
* Fetching records..."

Same time slice, same bounding box, different number of records found.

----

We are dead in the water. Either connections hang or when they go through the results are inconsistent.

Is anyone else seeing this?

Thank you.

-David
 

Please Log in or Create an account to join the conversation.

1 year 2 months ago #1672 by junzis
David, could you please share the code you wrote for downloading the data? I can check if I can reproduce the same error.

Btw, which version of pyopensky are you using?

Please Log in or Create an account to join the conversation.

1 year 2 months ago #1673 by dkovar
Good morning,

And thank you.

My code is here: github.com/URSA-Inc/adsb
pyopensky 1.4 pypi_0 pypi

I'm calling it as follows:

./opensky-pull.py --location BrooklynMD --bbox 40 --latitude 39.2335 --longitude -76.6318 --dest './output' --start 2021-08-01 --end 2021-08-02 --slice .5

That produces:

* Server connection successful!
Query starting with:
Time slice start: 2021-08-01 00:00:00 Time slice end: 2021-08-01 00:30:00
Bounding box: [39.0535, 39.4135, -76.8118, -76.4518]
* Obtaining details of the query...
* OpenSky Impala: 8545 of records found.
* Fetching records...

And it has been hung on "Fetching records..." for twenty minutes., one of the three issues I've been experiencing.

If I query a location in NYC:

./opensky-pull.py --location NY1 --bbox 20 --latitude 40.7160 --longitude -73.9907 --dest './output' --start 2021-08-01 --end 2021-08-02 --slice .5
* Server connection successful!
Query starting with:
Time slice start: 2021-08-01 00:00:00 Time slice end: 2021-08-01 00:30:00
Bounding box: [40.626, 40.806, -74.0807, -73.9007]
* Obtaining details of the query...

And it hangs there.

A couple of weeks ago I could reliably pull data from Boston using a one hour time slice and 40km bounding box. Now, essentially nothing.

I've tried running this on several different machines, rebuilt the conda environment, and rewrote the code to no avail.

I hope I'm missing something simple.

Thanks very much.

-David

Please Log in or Create an account to join the conversation.

1 year 2 months ago #1674 by dkovar
Greetings,

And I am now seeing this occasionally:

* Server connection successful!
Query starting with:
Time slice start: 2021-08-01 00:00:00 Time slice end: 2021-08-01 00:30:00
Bounding box: [26.048, 26.408, -98.432, -98.072]
* Obtaining details of the query...
ERROR:paramiko.transport:Socket exception: Operation timed out (60)

Seems to imply that the server is overloaded.

-David

Please Log in or Create an account to join the conversation.

1 year 2 months ago #1677 by junzis
I think there is an error in your code line 87. The bound should have the following format:
- bound (list): Filter latitude and longitude bound with format of [lat1, lon1, lat2, lon2], default to None

I believe you swapped the values.

Please Log in or Create an account to join the conversation.

1 year 2 months ago #1678 by dkovar
Greetings,

Thank you very, very much. It is amazing how important a second set of eyes can be.

I owe you a beer or a coffee or something.

-David

Please Log in or Create an account to join the conversation.

1 year 2 months ago #1680 by junzis
You are welcome, David!
Oh, certainly, perhaps at the next OpenSky symposium. :D

Please Log in or Create an account to join the conversation.

1 year 2 months ago #1682 by dkovar
Greetings,

I'll definitely try to be there!

By the by, I'm now getting this error:

* Obtaining details of the query...ERROR:paramiko.transport:Socket exception: Connection reset by peer (54)Traceback (most recent call last):  File "/Users/kovar/work/adsb/./opensky-pull.py", line 124, in <module>    main()  File "/Users/kovar/work/adsb/./opensky-pull.py", line 120, in main    pull_data(args.location, float(args.latitude), float(args.longitude),  File "/Users/kovar/work/adsb/./opensky-pull.py", line 66, in pull_data    df = opensky_request(opensky, datestr, datestr_increment, bbox)  File "/Users/kovar/work/adsb/./opensky-pull.py", line 83, in opensky_request    df = opensky.query(  File "/Users/kovar/miniforge3/envs/a50-dev/lib/python3.9/site-packages/pyopensky/impala_wrapper.py", line 187, in query    output = self.shell("-q " + count_cmd)  File "/Users/kovar/miniforge3/envs/a50-dev/lib/python3.9/site-packages/pyopensky/ssh_client.py", line 148, in shell    raise Exception(Exception: Command returned non-zero exit code: -1

Can pyOpenSky catch and retry this?

Thank you.

-David
 

Please Log in or Create an account to join the conversation.

Powered by Kunena Forum
This website uses cookies to offer you the best experience of our services. By using this website you agree to our privacy policy!