states returned: 17 or 18 fields?

Hi everyone, when accessing the OpenSky interface with the following Python code:

url_data='https://'+user_name+':'+password+'@opensky-network.org/api/states/all?'+'lamin='+str(lat_min)+'&lomin='+str(lon_min)+'&lamax='+str(lat_max)+'&lomax='+str(lon_max)
col_name=['icao24','callsign','origin_country','time_position','last_contact','long','lat','baro_altitude','on_ground','velocity',       
'true_track','vertical_rate','sensors','geo_altitude','squawk','spi','position_source']
 response=requests.get(url_data).json()

I get changing number of fields returned. Sometimes 17 fields, sometimes 18 fields, where there seems to be an 18th field added to the end of the dict.
Any clues? Can I specify my request such that I get only the 17 fields returned?

Many thanks,
Job
 

Does anyone have a clue why this is happening?

You could try to use the Python-API instead:
openskynetwork.github.io/opensky-api/python.html
There is also an example showing how to use a bounding box. And all fields are named which may simplify the use for you.

Yes I am aware of that of course.
However the update rate maxes out at 1 update per every five seconds. Just a little bit too slow for me.
And also, sometimes it returns 18 fields instead of 17 fields and i do not understand why...

Edit: let me check first if that IS the api that I was using

could you provide a data sample each (17 and 18 fields)?

Here was the code I wrote for it:

url_data='https://'+user_name+':'+password+'@opensky-network.org/api/states/all?'+'lamin='+str(lat_min)+'&lomin='+str(lon_min)+'&lamax='+str(lat_max)+'&lomax='+str(lon_max)
col_name=['icao24','callsign','origin_country','time_position','last_contact','long','lat','baro_altitude','on_ground','velocity',
'true_track','vertical_rate','sensors','geo_altitude','squawk','spi','position_source']

#REQUEST INTERVAL
if user_name !='' and password !='':
sleep_time=5
else:
sleep_time=10


#GET DATA AND STORE INTO CSV
global cpoa_cols
cpoa_cols =
global cpoa_list


while col_name !='':
# create an empty dataframe to later store the results from the query

combination_count = 0
cpoa_list = []
with open(csv_data,'w') as csv_file:
csv_writer=csv.writer(csv_file,delimiter=';', quoting=csv.QUOTE_NONNUMERIC)
csv_writer.writerow(col_name)
response=requests.get(url_data).json()
try:
print(len(response),'data items returned from OpenSky')
states = response # states is a list of lists!
except Exception:
print('Hier gaat iets mis...')
pass

# for unknown reasons sometimes we get 18 fields returned, in stead of the expected 17 fields.
# in that case, just delete the last item in the list.
# assume that ALL response rows have 18 items
#print(len(states[0]))
if len(states[0]) == 18:
for i in range(0,len(states)):
states.pop()
#for index in range(0, len(states)):
# print(len(states[index]), end='')

df = pd.DataFrame(states, columns = col_name)

And the error message:

---

AssertionError Traceback (most recent call last)
C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\internals\construction.py in _list_to_arrays(data, columns, coerce_float, dtype)
563 try:
--> 564 columns = _validate_or_indexify_columns(content, columns)
565 result = _convert_object_array(content, dtype=dtype, coerce_float=coerce_float)

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\internals\construction.py in _validate_or_indexify_columns(content, columns)
687 # caller's responsibility to check for this...
--> 688 raise AssertionError(
689 f"{len(columns)} columns passed, passed data had "

AssertionError: 17 columns passed, passed data had 18 columns

The above exception was the direct cause of the following exception:

ValueError Traceback (most recent call last)
<ipython-input-7-f5cfc221ecaa> in <module>
58 # print(len(states[index]), end='')
59
---> 60 df = pd.DataFrame(states, columns = col_name)
61 df = df.str.contains('|'.join(mil_callsigns))
62 #print(df.sum(),'military callsigns were marked based on callsign.')

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\frame.py in __init__(self, data, index, columns, dtype, copy)
507 if is_named_tuple(data[0]) and columns is None:
508 columns = data[0]._fields
--> 509 arrays, columns = to_arrays(data, columns, dtype=dtype)
510 columns = ensure_index(columns)
511

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\internals\construction.py in to_arrays(data, columns, coerce_float, dtype)
522 return [], [] # columns if columns is not None else []
523 if isinstance(data[0], (list, tuple)):
--> 524 return _list_to_arrays(data, columns, coerce_float=coerce_float, dtype=dtype)
525 elif isinstance(data[0], abc.Mapping):
526 return _list_of_dict_to_arrays(

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\internals\construction.py in _list_to_arrays(data, columns, coerce_float, dtype)
565 result = _convert_object_array(content, dtype=dtype, coerce_float=coerce_float)
566 except AssertionError as e:
--> 567 raise ValueError(e) from e
568 return result, columns
569

ValueError: 17 columns passed, passed data had 18 columns

Sorry, it's hard to follow your code when all the indentation formatting is gone. I assume, you need a pandas dataframe and want to filter by callsign?
Maybe it could help to look, how I consume the API data (version on github is work in progress):
github.com/jaluebbe/FlightRoutes/
Here I am consuming all states. If the callsigns do not fit a commercial airline pattern or any fields required for me are incomplete, I discard that state. Finally, I assemble a dict (callsign as key) of dicts (each holding all data from the state) which is stored in a redis database. You could easily create a dataframe instead: df = pd.DataFrame(positions.values()) .

Thank you.
Indeed without the indenting it is hard to follow. Sorry.
In the meantime I switched to an API of another popular flight tracker. This seems to work stable.
And after that, I now read out the output of my own Raspberry Pi, which gives me a once per second update rate.

Trying to clean up the code I found this was still an error message. Opensky sometimes returns 18 field and my code crashes.
I now assume this is a random error of Opensky and will have to work around it somehow..

I'm seeing a similar issue.
Here's an example of the 18 field reply:
'4b1814', 'SWR109G ', 'Switzerland', 1655023061, 1655023061, 9.0316, 49.8678, 10668, False, 219.95, 187.39, 0.33, None, 11064.24, '1123', False, 0, 0

To my experience, 18 fields appear when I add a parameter "extended=true" to the request.

Thank you.
Indeed without the indenting it is hard to follow. Sorry.