states returned: 17 or 18 fields?

1 month 1 week ago - 1 month 1 week ago #1698 by jobbruggen
Hi everyone, when accessing the OpenSky interface with the following Python code:

url_data='https://'+user_name+':'+password+'@opensky-network.org/api/states/all?'+'lamin='+str(lat_min)+'&lomin='+str(lon_min)+'&lamax='+str(lat_max)+'&lomax='+str(lon_max)
col_name=['icao24','callsign','origin_country','time_position','last_contact','long','lat','baro_altitude','on_ground','velocity',       
'true_track','vertical_rate','sensors','geo_altitude','squawk','spi','position_source']
 response=requests.get(url_data).json()

I get changing number of fields returned. Sometimes 17 fields, sometimes 18 fields, where there seems to be an 18th field added to the end of the dict.
Any clues? Can I specify my request such that I get only the 17 fields returned?


Many thanks,
Job
 

Please Log in or Create an account to join the conversation.

1 month 1 day ago #1702 by jobbruggen
Does anyone have a clue why this is happening?

Please Log in or Create an account to join the conversation.

1 month 1 day ago #1703 by jluebbe
You could try to use the Python-API instead:
openskynetwork.github.io/opensky-api/python.html
There is also an example showing how to use a bounding box. And all fields are named which may simplify the use for you.

Please Log in or Create an account to join the conversation.

1 month 1 day ago - 1 month 1 day ago #1704 by jobbruggen
Yes I am aware of that of course.
However the update rate maxes out at 1 update per every five seconds. Just a little bit too slow for me.
And also, sometimes it returns 18 fields instead of 17 fields and i do not understand why...

Edit: let me check first if that IS the api that I was using

Please Log in or Create an account to join the conversation.

1 month 1 day ago #1705 by jluebbe
could you provide a data sample each (17 and 18 fields)?

Please Log in or Create an account to join the conversation.

1 month 1 day ago #1706 by jobbruggen
Here was the code I wrote for it:

url_data='https://'+user_name+':'+password+'@opensky-network.org/api/states/all?'+'lamin='+str(lat_min)+'&lomin='+str(lon_min)+'&lamax='+str(lat_max)+'&lomax='+str(lon_max)
col_name=['icao24','callsign','origin_country','time_position','last_contact','long','lat','baro_altitude','on_ground','velocity',
'true_track','vertical_rate','sensors','geo_altitude','squawk','spi','position_source']

#REQUEST INTERVAL
if user_name !='' and password !='':
sleep_time=5
else:
sleep_time=10


#GET DATA AND STORE INTO CSV
global cpoa_cols
cpoa_cols =
global cpoa_list


while col_name !='':
# create an empty dataframe to later store the results from the query

combination_count = 0
cpoa_list = []
with open(csv_data,'w') as csv_file:
csv_writer=csv.writer(csv_file,delimiter=';', quoting=csv.QUOTE_NONNUMERIC)
csv_writer.writerow(col_name)
response=requests.get(url_data).json()
try:
print(len(response),'data items returned from OpenSky')
states = response # states is a list of lists!
except Exception:
print('Hier gaat iets mis...')
pass

# for unknown reasons sometimes we get 18 fields returned, in stead of the expected 17 fields.
# in that case, just delete the last item in the list.
# assume that ALL response rows have 18 items
#print(len(states[0]))
if len(states[0]) == 18:
for i in range(0,len(states)):
states.pop()
#for index in range(0, len(states)):
# print(len(states[index]), end='')

df = pd.DataFrame(states, columns = col_name)

Please Log in or Create an account to join the conversation.

1 month 1 day ago #1707 by jobbruggen
And the error message:

AssertionError Traceback (most recent call last)
C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\internals\construction.py in _list_to_arrays(data, columns, coerce_float, dtype)
563 try:
--> 564 columns = _validate_or_indexify_columns(content, columns)
565 result = _convert_object_array(content, dtype=dtype, coerce_float=coerce_float)

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\internals\construction.py in _validate_or_indexify_columns(content, columns)
687 # caller's responsibility to check for this...
--> 688 raise AssertionError(
689 f"{len(columns)} columns passed, passed data had "

AssertionError: 17 columns passed, passed data had 18 columns

The above exception was the direct cause of the following exception:

ValueError Traceback (most recent call last)
<ipython-input-7-f5cfc221ecaa> in <module>
58 # print(len(states[index]), end='')
59
---> 60 df = pd.DataFrame(states, columns = col_name)
61 df = df.str.contains('|'.join(mil_callsigns))
62 #print(df.sum(),'military callsigns were marked based on callsign.')

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\frame.py in __init__(self, data, index, columns, dtype, copy)
507 if is_named_tuple(data[0]) and columns is None:
508 columns = data[0]._fields
--> 509 arrays, columns = to_arrays(data, columns, dtype=dtype)
510 columns = ensure_index(columns)
511

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\internals\construction.py in to_arrays(data, columns, coerce_float, dtype)
522 return [], [] # columns if columns is not None else []
523 if isinstance(data[0], (list, tuple)):
--> 524 return _list_to_arrays(data, columns, coerce_float=coerce_float, dtype=dtype)
525 elif isinstance(data[0], abc.Mapping):
526 return _list_of_dict_to_arrays(

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\internals\construction.py in _list_to_arrays(data, columns, coerce_float, dtype)
565 result = _convert_object_array(content, dtype=dtype, coerce_float=coerce_float)
566 except AssertionError as e:
--> 567 raise ValueError(e) from e
568 return result, columns
569

ValueError: 17 columns passed, passed data had 18 columns

Please Log in or Create an account to join the conversation.

1 month 1 day ago #1708 by jluebbe
Sorry, it's hard to follow your code when all the indentation formatting is gone. I assume, you need a pandas dataframe and want to filter by callsign?
Maybe it could help to look, how I consume the API data (version on github is work in progress):
github.com/jaluebbe/FlightRoutes/
Here I am consuming all states. If the callsigns do not fit a commercial airline pattern or any fields required for me are incomplete, I discard that state. Finally, I assemble a dict (callsign as key) of dicts (each holding all data from the state) which is stored in a redis database. You could easily create a dataframe instead: df = pd.DataFrame(positions.values()) .

Please Log in or Create an account to join the conversation.

1 month 1 day ago #1709 by jobbruggen
Thank you.
Indeed without the indenting it is hard to follow. Sorry.
In the meantime I switched to an API of another popular flight tracker. This seems to work stable.
And after that, I now read out the output of my own Raspberry Pi, which gives me a once per second update rate.

Please Log in or Create an account to join the conversation.

Powered by Kunena Forum
This website uses cookies to offer you the best experience of our services. By using this website you agree to our privacy policy!