The individuals supporting an AI service can acquire access, authority, or an expanded role without the necessity of a new model release. Vendor oversight must also account for these alterations.
On September 22, Reuters reported that Meta was experimenting with a ‘human concierge’ for its Muse AI agent. Contractors were managing some of the phone calls initiated through the agent, which led to privacy concerns raised by employees. A vice president at Meta acknowledged internally that initiating contractor-placed calls without adequate disclosure was ‘a miss’ and indicated that the feature had been temporarily retracted. This feature was being trialed internally with Meta staff, not enterprise clients. However, it prompts a critical question for organizations purchasing AI-driven services: What can shift even when the vendor and model version remain unchanged?
For a chief compliance officer, audit executive, or third-party risk manager, the pressing question is whether the current service still aligns with the facts that were significant at the time of approval. Most staffing changes at an AI vendor will not alter that assessment.
For several years, I oversaw the comprehensive experience of a global network comprising over 10,000 external mathematics experts who constituted the human component of an AI-enhanced learning product. When that network transitioned to a new provider, it was essential to ensure that data protection, access controls, payment security, and operational continuity were maintained throughout the transition. The inquiries revolved around who could access which information, where responsibilities had shifted, and how work and payments would proceed. While an expert network differs from an enterprise AI vendor, that experience taught me to monitor the role rather than the personnel.
A notice regarding model changes does not capture every alteration in service.
Having a contractual right to be informed when an AI vendor modifies its model is beneficial. In a recent CCI article discussing bank AI governance, Elaine F. Duffus and Aoife May included it among the vendor terms worth considering. However, a model release is merely one avenue through which an approved service can evolve.
Consider a scenario where, at the time of service approval, an external reviewer examined redacted outputs. Six months later, individuals in that role are addressing customer cases using complete records. Although the vendor and model version remain the same, access has broadened, and authority may have shifted. The organization might require different privacy, security, or customer protection measures.
This does not imply the need to track every individual who joins or departs from the vendor’s workforce. Staff turnover is a standard occurrence. Initial vendor due diligence determines whether a service is suitable for use. After approval, the focus should be on recognizing when the facts that informed that decision are no longer accurate.
A Gartner survey of third-party relationship managers revealed that when they noticed a change in the scope of a relationship, only 52% consistently communicated it to compliance. Relationship managers are often the first within an organization to detect changes. They may not be aware of which changes are significant to the control function.
Thus, the instruction ‘Inform us if anything changes’ is a weak monitoring directive. Provide the business owner with a more specific trigger.
Document four key facts about the role in the approval record.
NIST’s AI Risk Management Framework Playbook, under MANAGE 3.1, emphasizes the need for regular monitoring of third-party AI risks. It also highlights that documenting the technologies, personnel, and other resources utilized can aid in managing those risks. However, it does not specify which personnel facts should be retained.
For an external human role that significantly supports an AI service, I would recommend recording four key facts at the time of approval:
- Function and authority: Does the role sample outputs, resolve exceptions, moderate content, complete tasks that the AI cannot, or communicate with customers? Can the individual provide advice only, or do they have the authority to approve, send, pay, or take other actions? The term ‘human oversight’ is too vague to maintain that distinction.
- Access: Document what the role can view. Redacted output and a complete customer file present different risks, even if both are accessed through the same model.
- Delivery: Does the vendor provide…
Source: Original article