Indications of overloaded servers

1 week 5 days ago #1679 by dkovar
Greetings,

These are essentially identical queries against the OpenSky servers via pyopensky all done sequentially. The bounding box (40x40km) and time slice (1 hour)
are identical for each query but the hour is advanced by one.Query took: 6.213090 seconds.
Query took: 43.898065 seconds.
Query took: 89.453178 seconds.
Query took: 151.286648 seconds.
Query took: 578.597576 seconds.
Query took: 283.345230 seconds.
Query took: 35.102263 seconds.
Query took: 4.700390 seconds.
Query took: 233.348282 seconds.
Query took: 253.570446 seconds.
Query took: 10.560211 seconds.
Query took: 257.654125 seconds.

That is a lot of variation in response times and some really long response times. The query time is not related to the number of rows returned in any significant way.
 * Obtaining details of the query...
* OpenSky Impala: 0 of records found.
* No record found.
Query took: 283.345230 seconds

And:

* Obtaining details of the query...
* OpenSky Impala: 0 of records found.
* No record found.
Query took: 4.700390 seconds.-David

Please Log in or Create an account to join the conversation.

1 week 5 days ago #1681 by dkovar
Greetings,

And my three connections all terminated with:

ERROR:paramiko.transport:Socket exception: Connection reset by peer (54)

Another indication of server load.

-David

Please Log in or Create an account to join the conversation.

1 week 4 days ago #1684 by strohmeier
I've referred this to the engineers but with a lot of connections you might just be blocked by the firewall.

Please Log in or Create an account to join the conversation.

1 week 4 days ago #1685 by dkovar
Greetings,

I was advised by some people at OpenSky via email to open a number of connections to accomplish our goals.

I worried there might infrastructure on your side that might detect my activity as abusive and shut me down that they were unaware of.

Thank you very much for looking into this.

-David

Please Log in or Create an account to join the conversation.

1 week 1 day ago #1688 by dkovar
Greetings,

Any insights on this? I'm still getting the error and I cannot figure out how to programatically handle it.

Thank you.

-David

Please Log in or Create an account to join the conversation.

1 week 13 hours ago #1690 by dkovar
I've been running queries against OpenSky for several days now and logged all the query times:

count 8464.000000
mean 56.145642
std 140.582437
min 2.295887
25% 11.767489
50% 22.007387
75% 53.583520
max 2175.544041

The number of rows found has little impact on the time the query takes. There are long runs when queries take under 10 seconds but there are clearly a lot of times that they take more than that, and closer to a full minute.

-David

Please Log in or Create an account to join the conversation.

Powered by Kunena Forum
This website uses cookies to offer you the best experience of our services. By using this website you agree to our privacy policy!