Who is NIST?
The National Institute of Standards and Technology (NIST) is a renowned scientific agency under the U.S. Department of Commerce with approximately 3,400 staff. Established in 1901, it is one of the nation's oldest physical science labs and has been a key driver of technological innovation and industrial competitiveness. It uses a multidisciplinary and consensus driven approach to create standards for measurements, ensuring accuracy and consistency across industries. NIST's contributions span diverse areas, including cybersecurity, manufacturing, healthcare, and renewable energy. Its history, multidisciplinary approach, and emphasis on collaboration make NIST a globally respected institution shaping scientific progress and standards.
What is the RMF?
In the United States, the legislature has delegated the AI standard setting process to NIST. It was specifically prompted by the National Artificial Intelligence Initiative Act, which was part of the 2020 national defense authorization. The AI RMF takes cues from previous frameworks on managing risks and governance in information technology, like the Cybersecurity Framework introduced in 2014 and the Privacy Framework unveiled in 2020. Similar to its predecessors, the AI RMF was crafted through an inclusive and iterative process, incorporating public feedback through two draft releases, numerous workshops, and other public involvement initiatives. Much like the earlier frameworks, the AI RMF aims to be a flexible and dynamic document that is voluntary, respects individual rights, applies to organizations across various sectors, and can be customized to fit any organization's size and nature. Additionally, the AI RMF, like its precursors, employs a framework that organizes implementation into "core functions," subcategories, and implementation profiles, which has proven effective in the past.
What are the key components of NIST RMF?
Firstly, it provides a conceptual roadmap for identifying risk and highlights key attributes of trustworthy AI, including safety, security, fairness, and transparency. Secondly, it presents organizational processes aimed at assessing and mitigating risk by aligning socio-technical aspects with different stages of the AI system lifecycle. These processes encompass test, evaluation, verification, and validation (TEVV). The framework delineates core functions such as governance, mapping, measurement, and management, further subdivided into specific categories. Notably distinct from its predecessor, the AI RMF does not encompass implementation tiers and profiles, opting for a broader approach to guide implementation strategies.
How does Themis AI help with the AI RMF?
Our uncertainty estimation technology helps with the ‘measurement’ function of the RMF. In the RMF playbook, one question within measurement is ‘What testing, if any, has the entity conducted on the AI system to identify errors and limitations (i.e.adversarial or stress testing)?’. Traditional tools and techniques to identify these errors and limitations, such as sensitivity analyses or adversarial examples searches, are usually an one-off, point-in-time analysis of the data or models. Our wrapper technology ensures that the AI model continues to produce uncertainty estimates with minimal effort. This easy implementation approach reduces the compliance cost that engineers will face in satisfying the testing component of the RMF.
To conclude, the AI Risk Management Framework is yet to be completed, and still serves as a positive step forward in bridging the gap between the policy and technical aspects of AI regulation. One criticism is that it leaves the technical implementation of its principles up to engineers, and this will naturally take time for the industry to fill in these knowledge gaps. Our team at Themis AI is proud to provide a timely solution for AI engineers to fulfill some important requirements that the RMF encourages.
