The importance of oversight in an automated environment

In the last year or so, nothing has been hotter than the topic of machine intelligence. It seems every business is touting its revolutionary new product that is either machine intelligence-powered or enabled. The vast majority of these pitches are the wishful thinking of an on-trend marketing department. But access to these useful tools has never been more widespread, and these tools are becoming increasingly useful by the day. They are being used more and more in place of search engines and for researching topics by some users. What does that mean in the aviation world?

Dot Com Redux

In many ways, it’s not that different than the early days of the internet. Anyone who was around for those heady times will remember all the hope and genuine enthusiasm generated by access to so much information. Near-seamless communication across the globe had arrived, and it felt like the entirety of human knowledge was at your fingertips. It wasn’t long after that that people learned not everything on the internet was true. Whether it was an honest mistake, satire, or active deception, we quickly learned to adopt the proverb of "trust, but verify." 

When acting as pilot-in-command (PIC), we are responsible for all aspects of the flight, including the briefing, planning, and background research. The same applies to our aeronautical research and education; we are the “learner-in-command.” This doesn’t mean a pilot must know everything possible about a topic, but that they must make an effort to ensure what they have learned is correct. Pilots relying on machine intelligence sources should be wary. Many models are built by scraping data from various sources, often the entire internet. The old computer science mantra of GIGO (garbage in, garbage out) is hard to completely eliminate, but automated reasoning can be helpful in distilling a large amount of information into more digestible chunks. It can also allow the user to ask specific questions that fit their exact circumstances. Additionally, some models have ongoing capabilities that allow pilots to task them with assignments and build capabilities over time.

Ghosts in the Machine

There are subtleties about machine intelligence in general and each model in particular that create challenges to their use. One of the biggest challenges is referred to as hallucinations. In the machine intelligence world, a hallucination is when a model will, with absolute confidence, lie to you. It may not be a lie per se, but it will be a horrifically wrong answer stated with absolute conviction and, at times, provided with completely made-up sources. For example, a model may say Class D airspace is uncontrolled, which is not true. If the inquirer is familiar with the subject, this will be rather obvious, but if they aren’t, it can lead to a huge misunderstanding. Another concern is that models tend to be overly sycophantic, trying to ingratiate themselves with the user. This means that they might not be as direct in answering questions as we would like.

Additionally, machine intelligence is not a monolith. There are many models from various providers, each with their own strengths and weaknesses. For example, some models were notoriously bad at math, even basic math. They could provide excellent summaries of complex topics but couldn’t keep up with a junk-quality desk calculator from the 1980s in math. It seems counterintuitive that a model could be so impressive in one aspect and lackluster in another, but what a model is built for matters. So, selecting the right model for the right task is important. And remember, these models are constantly changing. That can lead to improvements in some areas and reduced performance in others.

Does this mean automated reasoning is useless? No, it can be a very useful tool. But like any tool, it requires training to get the most out of it. For example, you could task machine intelligence to monitor weather conditions and forecasts along a certain route in the days before a flight to help inform your briefing. Also, like any tool, it can be dangerous to use it without familiarity. Try experimenting with different tools and ask around to see if other pilots have suggestions. Following our “trust, but verify” approach, pilots should look for reliable sources to verify any machine intelligence statements. This can help evaluate how good a model is at a task. Check out the FAA's library of handbooks to get you started.

Editor’s note: Machine intelligence model recommendations were not provided to avoid endorsing any specific models and because recommendations may not be as valid by the time this article reaches publication.
 

Image