Engaging a third-party AI solution does not absolve an organization from its responsibilities towards consumers and employees impacted by it. Angela Juneau from Pashman Stein Walder Hayden outlines essential questions that organizations should consider prior to making a purchase — ranging from the training process of the model to the ownership of its outputs — enabling legal, compliance, and business teams to assess risks before implementation rather than after.
Organizations are integrating AI into nearly every facet of their operations: screening job candidates, creating content, analyzing data, enhancing internal investigations, addressing customer service requests, and generally expediting business processes.
However, companies that implement AI without a clear understanding of how a model was created, its functionality, and its data management practices may expose themselves to risks that only become evident when issues arise. As regulators, consumers, employees, and business partners increasingly scrutinize AI applications, organizations should approach AI procurement with the same level of diligence they would apply to any significant technology investment.
The following inquiries can assist organizations in assessing third-party AI prior to deployment and in establishing a framework for evaluating AI-related risks.
Data lineage
Where did the training data for the AI originate? Was the model trained on data that was legally obtained and appropriately licensed? Did the developer take measures to minimize the use of personal information, copyrighted materials, or other sensitive data that could pose legal or compliance risks?
Data quality
Can the quality of the data behind the model be trusted? The effectiveness of an AI system is contingent upon the quality of the data used for training. What actions has the vendor taken to ensure data accuracy, mitigate bias, and enhance data quality?
Intended use
Is the AI appropriate for your specific use case? An AI tool that excels in one scenario may not perform well in another. Has the model been evaluated for the particular business purpose it will serve?
Reliability
How accurate and dependable are the outputs? What evaluations have been conducted to assess accuracy, consistency, and error rates? How will your organization validate AI-generated content, recommendations, or conclusions before placing reliance on them?
Output ownership
Who holds the rights to the outputs and associated intellectual property? Carefully examine the vendor’s terms. Does your organization possess ownership of the content produced by the AI, or does the provider retain certain rights? Are there limitations on how outputs can be utilized or disseminated?
Human oversight
What level of human oversight is in place? Who is accountable for reviewing AI-generated outputs or AI-assisted decisions? If the AI impacts employment, customer, financial, or operational choices, what processes are available for review, escalation, or correction?
Privacy and security
How does the AI manage sensitive information? What data is collected, stored, retained, or shared? If employees, applicants, customers, or proprietary business information are involved, what measures are in place to safeguard that information?
Transparency
How transparent is the system? Are users aware when they are engaging with AI? Can the vendor clarify how the system arrives at conclusions, makes recommendations, or generates outputs to the extent necessary for business and regulatory compliance?
Governance and monitoring
How is the AI monitored and governed over time? AI performance can fluctuate as data, business conditions, and user behavior change. How does the vendor oversee performance, address model drift, respond to incidents, and implement updates?
Business continuity
What occurs if the system fails or becomes inaccessible? Does your organization have a contingency plan in place if the tool becomes unavailable, produces detrimental outputs, encounters a security breach, or ceases to function?
Source: Original article