Inconsistent results and hanging connections via pyopensky

Good evening,

I was reliably using pyopensky to download large volumes of data over many weeks. I now have three problems:

1) A week or so ago my connections would report:
"Server connection successful!"
and then hang completely, not producing records or errors, or terminating.

2) When it does work, I often see:* Obtaining details of the query...
* OpenSky Impala: 41 of records found.
* Fetching records...
* No record found.So, it finds records and then when it tries to fetch them no records are produced

3) If I make exactly the same query a few minutes apart, it reports a different number of records found:

"2021-09-01 00:00:00 2021-09-01 00:30:00 [25.958, 26.498, -98.522, -97.982]
* Obtaining details of the query...
* OpenSky Impala: 43 of records found.
* Fetching records...
* No record found."

"* Server connection successful!
2021-09-01 00:00:00 2021-09-01 00:30:00 [25.958, 26.498, -98.522, -97.982]
* Obtaining details of the query...
* OpenSky Impala: 44781 of records found.
* Fetching records..."

Same time slice, same bounding box, different number of records found.

----

We are dead in the water. Either connections hang or when they go through the results are inconsistent.

Is anyone else seeing this?

Thank you.

-David
 

David, could you please share the code you wrote for downloading the data? I can check if I can reproduce the same error.

Btw, which version of pyopensky are you using?

Good morning,

And thank you.

My code is here: github.com/URSA-Inc/adsb
pyopensky 1.4 pypi_0 pypi

I'm calling it as follows:

./opensky-pull.py --location BrooklynMD --bbox 40 --latitude 39.2335 --longitude -76.6318 --dest './output' --start 2021-08-01 --end 2021-08-02 --slice .5

That produces:

* Server connection successful!
Query starting with:
Time slice start: 2021-08-01 00:00:00 Time slice end: 2021-08-01 00:30:00
Bounding box: [39.0535, 39.4135, -76.8118, -76.4518]
* Obtaining details of the query...
* OpenSky Impala: 8545 of records found.
* Fetching records...

And it has been hung on "Fetching records..." for twenty minutes., one of the three issues I've been experiencing.

If I query a location in NYC:

./opensky-pull.py --location NY1 --bbox 20 --latitude 40.7160 --longitude -73.9907 --dest './output' --start 2021-08-01 --end 2021-08-02 --slice .5
* Server connection successful!
Query starting with:
Time slice start: 2021-08-01 00:00:00 Time slice end: 2021-08-01 00:30:00
Bounding box: [40.626, 40.806, -74.0807, -73.9007]
* Obtaining details of the query...

And it hangs there.

A couple of weeks ago I could reliably pull data from Boston using a one hour time slice and 40km bounding box. Now, essentially nothing.

I've tried running this on several different machines, rebuilt the conda environment, and rewrote the code to no avail.

I hope I'm missing something simple.

Thanks very much.

-David

Greetings,

And I am now seeing this occasionally:

* Server connection successful!
Query starting with:
Time slice start: 2021-08-01 00:00:00 Time slice end: 2021-08-01 00:30:00
Bounding box: [26.048, 26.408, -98.432, -98.072]
* Obtaining details of the query...
ERROR:paramiko.transport:Socket exception: Operation timed out (60)

Seems to imply that the server is overloaded.

-David

I think there is an error in your code line 87. The bound should have the following format:
- bound (list): Filter latitude and longitude bound with format of [lat1, lon1, lat2, lon2], default to None

I believe you swapped the values.

Greetings,

Thank you very, very much. It is amazing how important a second set of eyes can be.

I owe you a beer or a coffee or something.

-David

You are welcome, David!
Oh, certainly, perhaps at the next OpenSky symposium.

Greetings,

I'll definitely try to be there!

By the by, I'm now getting this error:

* Obtaining details of the query...ERROR:paramiko.transport:Socket exception: Connection reset by peer (54)Traceback (most recent call last):  File "/Users/kovar/work/adsb/./opensky-pull.py", line 124, in <module>    main()  File "/Users/kovar/work/adsb/./opensky-pull.py", line 120, in main    pull_data(args.location, float(args.latitude), float(args.longitude),  File "/Users/kovar/work/adsb/./opensky-pull.py", line 66, in pull_data    df = opensky_request(opensky, datestr, datestr_increment, bbox)  File "/Users/kovar/work/adsb/./opensky-pull.py", line 83, in opensky_request    df = opensky.query(  File "/Users/kovar/miniforge3/envs/a50-dev/lib/python3.9/site-packages/pyopensky/impala_wrapper.py", line 187, in query    output = self.shell("-q " + count_cmd)  File "/Users/kovar/miniforge3/envs/a50-dev/lib/python3.9/site-packages/pyopensky/ssh_client.py", line 148, in shell    raise Exception(Exception: Command returned non-zero exit code: -1

Can pyOpenSky catch and retry this?

Thank you.

-David