Augmented Reality and Aviation

Whatever techniques are used to enhance the accuracy and efficiency of flight, the pilot in command must always remember that he/she should entirely depend on skill, knowledge, and command.

by Dr. Ruwantissa Abeyratne
writing from The Bahamas

Our physical world is three dimensional. Even though we are far advanced in the digital age, most of our work is now being done on two-dimensional vision and application. Whether we look at a computer screen or smart phone we do not have the true picture until we translate the two-dimensional information we receive into a three-dimensional practicality which is the real world. This process of translation imposes a load on our mental capacity requiring time to decipher practical reality. This demand on our brain is called “cognitive load”. Augmented Reality (AR) greatly diminishes the demand on our cognitive load by converting the data obtained by two-dimensional methods images and animations that instantly gives us a picture of the real world. Michael Porter and James Heppelmann in their article Why Every Organization Needs an Augmented Reality Strategy published in the Harvard Business Review say: “Today, most AR applications are delivered through mobile devices, but increasingly delivery will shift to hands-free wearables such as head-mounted displays or smart glasses”.

The difference between AR and VR (Virtual Reality) is that, while AR takes digital information and transforms it to a 3D physical reality, obviating the burden of the cognitive load and is used in various commercial enterprises, VR takes physical reality into an environment that is computer generated and driven, thus making it an ideal application for entertainment purposes. The difference between the two is intrinsic in that AR gives us physical reality while VR gives us virtual reality. This is what makes AR valuable for the aviation industry. Porter and Heppelman go on to say: “At Boeing, AR training has had a dramatic impact on the productivity and quality of complex aircraft manufacturing procedures. In one Boeing study, AR was used to guide trainees through the 50 steps required to assemble an aircraft wing section involving 30 parts. With the help of AR, trainees completed the work in 35% less time than trainees using traditional 2-D drawings and documentation. And the number of trainees with little or no experience who could perform the operation correctly the first time increased by 90%”

The authors of System for synthetic vision and augmented reality in future flight decks (June 2000 Proceedings of SPIE - The International Society for Optical Engineering) say: “Rockwell Science Center is investigating novel human-computer interface techniques for enhancing the situational awareness in future flight decks. One aspect is to provide intuitive displays which provide the vital information and the spatial awareness by augmenting the real world with an overlay of relevant information registered to the real world. Such Augmented Reality (AR) techniques can be employed during bad weather scenarios to permit flying in Visual Flight Rules (VFR) in conditions which would normally require Instrumental Flight Rules (IFR). These systems could easily be implemented on heads-up displays (HUD)”. The new vision of the flight deck includes AR in weather information, surrounding air traffic and information on terrain.

It follows that AR could be of assistance to air traffic controllers as well in giving them a 3D vision of valuable data that could not be digested by human vision and transported through the cognitive load. This could certainly be true in the training of air traffic controllers as well.

But what if AR were to go wrong? Applying the analogy on the ground, Eric E. Sabelman and Roger Lam in their article The Real-Life Dangers of Augmented Reality say: “With augmented-reality gear barely on the market, rigorous studies of its effects on vision and mobility have yet to be done. But in reviewing the existing research on the way people perceive and interact with the world around them, we found a number of reasons to be concerned. Augmented reality can cause you to misjudge the speed of oncoming cars, underestimate your reaction time, and unintentionally ignore the hazards of navigating in the real world. And the worst thing about it: Until something bad happens, you won’t know you’re at greater risk of harm”. However, the authors hasten to add that this analogy may not be true for pilots, as unlike on the ground where the sphere of vision would be littered with people, cars and other objects and debris, the pilot gets a minimalist view of the sky which would enable the pilot to take decisions much quicker than a driver on the ground. However, this does not offer the airman a guarantee of 100% safeguard.

This brings one to the question: should a pilot depend entirely on an AR system installed in his/her aircraft? The definition of airmanship is “the consistent use of good judgment and well-developed skills to accomplish flight objectives. This consistency is founded on a cornerstone of uncompromising flight discipline and is developed through systematic skill acquisition and proficiency”. Would AR take these characteristics which are skill, knowledge and good judgment away from a pilot or in the least affect them? Christopher L. Parker, writing in the Magazine of Aircraft Owners and Pilots Association (AOPA) says in his article The Art of Airmanship a good airman(pilot) should have good command of the aircraft; apply his/her knowledge; show leadership; and show discipline. Of course, one can argue that AR could be used without adversely affecting any of these qualities, but the question is asked by many whether modern technology would make the pilot more complacent.

The Convention on International Civil Aviation provides that the pilot-in-command is responsible for the operation and safety of the aeroplane and for the safety of all persons on board, during flight time. Commercial airline pilots operate in a highly complex environment, particularly in single pilot operations. The difficulties faced by pilots in the work environment are compounded by the fact that often inadequate information aggravates the problem. Pilots rely heavily on their visual and auditory senses while flying, and it is of paramount importance that accurate information be available to the pilot at all times. Most importantly, pilots have usually the predilection to complete their given schedule no matter what, such as competing a flight as planned, meeting schedules, impressing their employees and pleasing the people they carry. Therefore, negligent issues concerning the professional conduct of a pilot are quintessential elements for a highly esoteric legal debate.

Perhaps the most significant case involving the negligence of a pilot is the 1976 decision in Taylor v. Alidair Limited which related to an unfair dismissal action in which a pilot appealed against his dismissal following a heavy landing leading to an uncorrected bounce and the collapse of the nosewheel assembly of the Viscount aircraft he was flying. The aircraft made a heavy touchdown, bouncing ten feet into the air and touching down heavily again on all three wheels on the runway. There was considerable damage to the engines and air-frame. Bristow J. held that: “there are activities in which the degree of professional skill which must be required is so high and the potential consequences of the smallest departure from that high standard are so serious that one failure to perform in accordance with those standards is enough to justify dismissal. The passenger-carrying airline pilot, the scientist operating the nuclear reactor, the chemist in charge of research into possible effect of, for example, thalidomide, the driver of the Manchester to London express, the driver of an articulated lorry full of sulphuric acid, are all in the situation in which one failure to maintain the proper standard of professional skill can bring about a major disaster”.

The position of the pilot in command of an aircraft therefore seems to be governed by the application of three presumptions. They are the special skill and expertise the pilot is presumed to possess; the enhanced duty of care expected of the pilot in view of such special skill and expertise; and the magnitude of the damage that may be caused in the eventuality of a breach of the duty by the care by the pilot. Whatever techniques are used to enhance the accuracy and efficiency of flight, the pilot in command must always remember that he/she should entirely depend on skill, knowledge, and command.