What ‘Real People’ Need to Know About Artificial Intelligence

AI is too important to be left to engineers and data scientists. You don’t have to become a certified expert in these disciplines. But if possible, it’s worth developing a basic understanding about how AI and machine learning works. At a minimum, it is essential to know what questions to ask if working with AI.

Here are five questions a non-technical person should ask about an AI system during its design and development.

1. What data will be used by the AI?

2. In what ways will the data be prepared?

3. How will the AI be evaluated?

4. How will humans interact with the AI?

5. What safety and ethical protections will be implemented?

If you don’t get good answers to these questions, it’s worth raising your hand and digging deeper.

What data will be used by the AI?

Data is the fuel of most AI systems. The data is used to train the AI and infer relationships or find patterns.

Typically the more data the better, but quality of data matters even more. It is worth asking if the team has validated the quality of the data in some way. Typically this is done through some analysis of the data or in some cases training a simple prototype of the AI on the data. The objective of validation is to show that there is some reason to believe that an AI can work. Sometimes this requires a big leap of faith, but often one can get a quick indication early on if the data (and algorithms) hold promise.

Make sure the team has the rights to the data. Most AI applications will require fresh data on an ongoing basis. For many startups, this data access/ ownership question is a knockout criteria. If you don’t have legitimate access to data (e.g., not screen scraping data from websites without permission), then there is no reason to invest in building an AI. It is worth considering whether there is a choice of data sources. Why this source and not another? Is the choice a sustainable source of data going forward at scale?

In some cases, it’s worth looking at the actual data elements being used. Why was this data element included and other elements excluded? For example, if you’re a bank building an AI to make credit decisions, you might want to think about income and assets of applicants, but possibly not eye color or horoscope sign. It’s good to ask what domain experts were engaged when making these decisions, and what is the theory for why each of these data elements might be relevant to the outcome and in what way. For example, knowing whether data elements have a nonlinear relationship (e.g., exponential) with the output typically requires domain expertise or experimentation.

It is worth considering the potential relationships among data elements. Are input variables independent of one another or in some way related (i.e., covariance)? In some cases, new elements may be derived from the data and input to the AI. For example, data elements of body weight and height may be combined to create your Body Mass Index (BMI) for determining your health. Such calculated elements usually come from domain expertise, and should be considered.

In what ways will the data be prepared?

In almost every case, the raw data must be prepared in ways to make it usable for AI. This can be everything from cleaning up images using filters to ignoring garbage data. Though these efforts are legitimate, they may have real implications on the performance of the AI.

It is important that people outside computer technology disciplines are involved in the data cleanup process. This includes business leaders, legal advisors and domain experts (e.g., doctors if building a healthcare AI). How does the model deal with errors in the data (e.g., clear outliers or anomalies)? It may be worth looking at examples of excluded data to make sure that they do not represent real and valid information. Likewise, it’s worth asking how the AI model deals with missing data. In some cases, missing data may have meaning, i.e., it was omitted intentionally.

One of the most important considerations is how to deal with bias. Bias is a distortion of a statistical result due to an unconsidered factor. In other words, models like AI, are simplifications of the world, and thus must ignore many factors that its designers consider not relevant to the outcome. Bias may be in the data inherently, or may be amplified by the learning algorithms. It is essential to understand how bias in the source data is being addressed. One method is to sample the data randomly and rerun the AI learning process repeatedly to see if you get different results. Also, it is not uncommon to normalize the data to make sure it reflects the real world better.

In many cases (e.g., supervised learning), the data must be labeled to be useful for training (e.g., each instance must be tagged by experts in the field). The cost of labeling can be prohibitive for some applications, and may not allow the AI to scale beyond a prototype. For this reason it is essential that the labeling plan should include input from domain experts and business analysts.

How will the AI be evaluated?

Defining success clearly and early is a good way to ensure it for any venture, but especially true for artificial intelligence.

The most common measure of an AI is accuracy. This is defined as percent of decisions or actions by the AI that were correct. Before embarking on the creation of an AI, the minimum acceptable accuracy should be set. Often this is based on the current best alternatives, including human capabilities. Determining accuracy of an AI usually means saving some set of data not used for training, called test data. It is important to understand how the test data was selected. Was it taken as a random sample and was it normalized for the real world if necessary?

Being better than humans most of the time may be good enough, but sometimes the cost of failure, especially when performance is needed the most under changing conditions, may not be acceptable. Humans are able to drive a car with a certain degree of accuracy, but when conditions merit (e.g., bad weather) they are able to heighten their attention and focus. Understanding how AI will deal with corner cases like these is essential.

No AI will be perfect, but not all errors are equal in effect. In general, there are two types of errors: false positives and false negatives. In some cases, it may be better to tune the AI to avoid false negatives even at the expense of being less accurate. For example, if the AI is diagnosing a life-threatening disease and the corrective action is not dangerous, then it would be better to err on the side of a finding for the disease (rather than a false negative, where the person had the disease but it was missed by the AI). These tradeoffs usually require a thorough understanding of the domain (e.g., medical diagnosis and treatment in this example) and economic costs of either error type.

The design, configuration and choice of AI model is a process of experimentation against these success criteria. In many cases, there are alternative approaches (or model types) to implementing the AI. Selecting the model type may have many implications to how the AI will perform. Some models are computationally more lightweight, while others may work better with images. In more complicated situations, multiple models of various types may be used and which may interact with each other. It is worth discussing the reasons behind the model choices and design decision comprising the AI.

Finally, one way to evaluate an AI model is how well it simplifies a real-world problem without missing essential factors. It is worth considering the complexity of the model compared to the source data or the problem (i.e., the model should be simpler). If the model perfectly reflects every instance in the training data, then it probably is not general enough to work with real-world data. If different train.ing data results in a very different result, then the model is said to be overfitting with original training data. If the AI is too general, so that it misses many situations in the data, then it is said to be underfitting.

How will humans interact with the AI?

As discussed previously, AI today is narrow in that it can do some things exceptionally well, but is quite useless otherwise.

For the foreseeable future, AIs will need to work alongside people to get things done. So getting the interface between human and machine right will be critical. It is important to think about how an AI will be used by humans. This human-machine interface is often not given much attention, even though it may greatly impact the ultimate success of the AI.

Where possible, it is best to design ways to augment humans with AI, rather than replacing them. Making the objective of the AI to completely replace humans is a recipe for disaster in most cases today. Will there be a human in the loop on the process with an AI, feeding it additional information or making adjustments to its recommendations? It’s worth asking what aspects of the work will be replaced by AI. Hopefully these are the more dull, repetitive and less fulfilling parts of the job anyway. In any case, an AI should have specific goals for productivity gains. Designed properly, AI should improve productivity, creativity and fulfillment of its human coworkers.

Likewise, being able to explain the reasons behind an AI decision or action will be vital to building trust and acceptance of the technology. It is worth asking, will the decisions or actions of the AI be explainable? Is there a way to audit the decision process? Some approaches to AI are inherently better at explaining their results. For example, decision trees or random forests are better at this than neural networks.

What safety and ethical protections will be implemented?

The last set of questions are around the safe and ethical use of AI.

Since the technology is so powerful yet sometimes unpredictable, AI needs to have some strong guardrails. Once again domain expertise is necessary to understand the risks of the AI being wrong. If the AI may be operating in situations that could harm people or property, there should be hard limits on its decisions. For example, a self-driving car may be limited in geography and speed, regardless of whatever the AI decides to do. Such logic rules are bolted over the AI model.

Finally, it is important to consider the ethical use of AI. One obvious area is the ethical use of data. Stewardship of the massive amount of data typically used for AI raises its own set of responsibilities. Is any of the data sensitive (e.g., healthcare) or has any special legal implications for users or owners of the data? AI ethics is a much larger topic and will be discussed in later articles in this series.

I give you these tools not just to be a smarty pants or to make your tech colleagues uncomfortable as they struggle to answer the questions (though both may feel good!). These are meaningful issues that should be addressed broadly from many disciplines and perspectives. Raising these issues should not interfere with the process of innovation and experimentation, and should be tailored to the situation of course. There may be good answers to these questions. In that case, you’ve done your job to validate the system for its likelihood for business success, humanity in approach and fitness to the problem.

Note: this article is based on a lecture given by me at UC Berkeley, Haas School of Business, Global Access Program, “What Real People Need To Know About Artificial Intelligence” February 15, 2019. Article date is approximately when the material was written and presented, but not the online publication date