THE AI CONUNDRUM

Book supplement · 2023-06-30

TGA - Training, Governance, Accountability and 25Q Checklist

An archived post from the original book website, presented in its historical context.

Training, Governance and Accountability (TGA)

Training

Training can be engaging and empowering when people get their hands on AI tools and can see the potential for themselves. At the same time, training should reveal the surprising foibles of AI—as well as the more serious faults. In terms of training, we suggest reviewing the fundamentals of how artificial intelligence works by covering pattern-fitting, gradient descent, and compression—and why AI can lack precision and produce biased outputs. Practical case examples relevant to the business or government entity should bring the strengths and weaknesses of artificial intelligence to the fore. Case studies of where AI has made mistakes can help drive home the point AI is fallible. There are excellent hands-on examples of AI bias in publicly available datasets, and step-by-step guides from resources such as the AI Fairness 360 program, developed by IBM and discussed in Chapter 7, as well as resources from Hugging Face, Google, Amazon, and others. There are also specific open-source AI bias detection resources available, and the more technical team members should be trained on these systems. Safeguards need to be built into the system, and staff need to know how to use them. Business decision makers also need to learn the protocols for model cards and service cards. 

In addition to training employees on AI, most organizations need to map out, in a very granular way, the work that is done currently and which tasks AI can replace. Redesigning the entire workflow for AI is often needed. This is the same transformation that software development went through thirty years ago when most computer programs were monolithic structures that required complex and costly rewriting when something needed to change. Many businesses faced a painful realization in the run-up to the year 2000 because their computer systems stored only the last two digits of the four-digit year. Going through these software systems to find and fix the date fields gave many an appreciation for a concept known as service-oriented architecture (SOA).

The big idea behind SOA is to break down an application into smaller, independent services that communicate with each other. This allows for more flexible development, testing, and deployment, because changes to one service do not require changes to others. SOA, which began to gain traction in the decade following the year 2000, was transformational for businesses because it enables faster innovation and scaling, improved fault isolation, and the ability to use different technologies for different services based on the specific needs of each service. A microservices architecture takes this same approach and extends it to the loosely coupled services which can be developed, deployed, and maintained independently. Each of these services is responsible for discrete tasks and can communicate with other services through simple APIs to solve a larger complex business problem. If an organization maps out the tasks and hand-offs from one person or team to another, it can then determine where an AI can fill in the task. The graphic from Chapter 8 (Figure 8.1) illustrating the change from translating a podcast to another language before and after the application of AI is a simplified version of how to break down the different elements of a workflow into tasks and hand-offs so that AI can be applied where appropriate. 

At the same time, if one transcription service is best in class today but a different vendor has a better synthetic voice, it would be ideal to select a best-of-breed solution for each rather than lock in with a single vendor that is good in one area and mediocre in another. A microservice approach provides flexibility for using different solutions. In addition, given the pace at which new solutions are developed on an ongoing basis, the microservice approach makes it easy to swap out one service for another when something better comes along.

Training the organization to see the work that is done as a collection of tasks (services) and hand-offs (similar to application interfaces) will go a long way to preparing the organization for a world where AI is used more extensively to create value.

Privacy and sensitivity of data should also be considered. Few think much about the fact that typing in a search to the Google search engine results in Google saving the search, effectively making it discoverable by a court and visible to Google. When using a large language model (LLM), a person may want to paste large blocks of text or data to get the LLM’s response, but one should be mindful that if model is a public rather than private model, the block of text or data essentially becomes public domain content that others may be able to surface in the future. Entering private data into a public AI system could violate privacy laws, or expose sensitive corporate secrets. In addition, when it comes to AI systems, certain architectures of AI can be reverse engineered to reveal the underlying trying dataset. 

Governance

A governance framework converts relevant ethical principles to implementable practices in an AI deployment process. In 2022, the government in Singapore developed, in conjunction with ten companies including Amazon, DBS Bank, Google, Meta, Microsoft, Singapore Airlines, Singtel Group, and others, a model framework for AI governance. Here’s an excerpt of Singapore’s model framework:

Guiding Principles:

  1. Decisions made by AI should be EXPLAINABLE, TRANSPARENT, and FAIR AI systems should be HUMAN-CENTRIC. 

From principles to practice, here are the key points in the governance framework:

  1. Internal Governance Structures and Measures

  1. Determining the Level of Human Involvement in AI-augmented Decision-making

  1. Operations Management

  1. Stakeholder Interaction and Communication

Links to Singapore’s model framework, which includes a range of resources for evaluating risk, and other governance considerations, are included in the AI Checklist in the Appendix.

Accountability

Accountability is the active ingredient in an AI governance framework—it is the mechanism that converts the failure to adhere to the values and procedures into consequences. In terms of accountability for removing bias, an organization can be clear on the bias it is trying to remove from artificial intelligence applications by explicitly listing any characteristics that should be safeguarded from bias—race, age, sex, and disability are protected classes for hiring, for example. It would be extremely problematic for an AI to entrench bias in a range of applications including law enforcement, lending, healthcare, education, and many other categories. 

There are two philosophies regarding fairness. One is that the AI model should be tuned to equalize—such as in hiring, where the model should not favor one group more than another. The second philosophy is that the AI should be tuned to achieve the best results possible, and if the results are uneven, then a countermeasure should be applied outside the data and model (for example, partitioning the hiring into two separate models). Those adhering to the second philosophy point out that equalizing the model itself may result in more overall harm as it may require de-tuning prediction or classification to make it worse for some groups—as the system seeks a least-common-denominator. But, the second approach of accepting worse results for some to get the best results overall can only be fair if the governance framework is followed to offset the weaknesses for the groups where AI doesn’t perform as well. 

The governance framework should specify the operating procedures for transparency when AI is used, the responsibilities for a human in the loop, and how the system will be periodically tested for bias. It should also define the consequences for individuals or vendors that do not follow governance procedures. When someone fails to follow AI governance, is it a fireable offense? Are there clawback provisions in an AI vendor contract for failure to adhere to the governance framework so that there is a financial penalty for the vendor’s failure? In other words, what is the accountability in the AI governance system? 

RACI is a framework in which people in an organization are classified in relation to decisions they make: those responsible for the decision, those who are accountable, those who require consultation before a decision is made, and those who are informed after a decision is made. For day-to-day operations of AI, a RACI should be established to ensure fair and equitable use of AI. Questions like these should be posed: 

We are fans of the Japanese quote from quality management: “A defect is a treasure.” It expresses the sentiment that we can learn from mistakes and improve the system. Does the accountable person have the authority to dig in to understand the root cause of the mistakes? Does the accountable person have the authority to change the AI system or the governance procedures to address root causes? What authority does the responsible party have for ensuring procedures are followed by others? 

Another aspect of accountability is the responsibility to the person whom an AI decision affected. For example, in the law enforcement scenario, what is the responsibility to the person being arrested? Have protocols been put in place to ensure that the AI is correct? What is the recompense for a wrongful arrest, especially when the rules governing the use of AI are not followed? Governance systems work best when paired with accountability. 

Organizations should consider working with peers to set-up independent review councils to co-fund independent checks of leading vendors. Or, if the organization has the skill set and financial wherewithal, perform the independent checks of vendors directly periodically and update training accordingly.


A Checklist for Applying AI

To aid in the assessment of determining when (and when not) to apply AI and how to apply countermeasures to offset weaknesses, we have created a worksheet of 25 questions. If you are involved in approving AI, use this list as a starting point and add your additional questions. Considering the limitations of today’s AI, we would all be well served by asking more questions and contemplating more countermeasures to offset AI’s weaknesses 

1. What is the current practice for the identified process that will be replaced by AI? (That is, what is the next best alternative to AI that you are currently using?) 

2. What are the benefits or needs that will be gratified by using AI?

3. What other alternatives, besides AI, exist to improve the current approach, and how do they compare to AI in terms of cost, time, ease of use, and effectiveness? 

4. Is the data that AI would use labeled automatically, or will it require building the dataset?

5. Is the application of AI considered very risky, risky, or not risky based on precision, input control, and rationale? (See Chapter 6 for full details on this framework.)


6. What are the benefits of the current approach that will be lost if AI is applied?

7. What is the truth set for accessing precision of current practice vs. AI? 

8. How will you measure precision? Will you use a false accept / false reject method, or some other approach?

9. Are the AI inputs from an open or closed system? 

10. Over time, how much is AI expected to improve? How much will the next best alternative improve?

11. Who will perform the risk assessment? 

12. Will people be in the workflow when AI is adopted?

13. What is the risk of hollowing out a large number of jobs? 

14. What is your back-up plan if it turns out that AI isn’t fully up to the task just yet? 

15. Is AI expected to access value and trade?

16. Can your application of AI influence attitudes or behaviors over the longer-term (e.g., personalization AI such as Facebook, YouTube, Amazon)?

17. Can changing the role of AI (the objective function) allow AI to add value without creating the same level of polarization as when AI is optimizing for engagement? 

18. In social media and discussion boards, is it possible to include a credibility or reputation score to improve the quality of content and sharing? 

19. How is training data collected, and what is the risk of bias? 

20. Could adding in a randomized control test to data collection reduce the risk of bias?

21. To address bias in the AI, can the model be partitioned to avoid AI bias for protected classes or otherwise add value without perpetuating bias?

22. Is fairness testing (such as AI Fairness 360) applied on a regular basis to check for bias? 

23. Does your organization have a governance framework that defines your guiding principles and standard operating procedures? 

24. Does your organization have a training module that teaches the team to recognize the hidden weaknesses of AI (lack of precision, risk in open environment, difficult to observe rationale for decisions) and how to offset these weaknesses in your specific applications? If not, do you have a plan to introduce training? 

25. Accountability: Is there true authority among those responsible for living up to the governance framework?