In the world of regulation, there exists much debate over the extent to which agencies can accurately measure their performance. In the most simplistic terms, one would think that if a regulator is doing its job, the regulated area would see a marginal number of complaints and incidents, and there would be little to record or observe. So how can regulators use external data as well as insights about their own internal processes to generate more tangible performance metrics and outline pathways for improvement?

As it turns out, this has been attempted quite a few times, by regulators and researchers all around the world who have sought to develop ways for agencies to analyze and evaluate their performance. In this piece, we offer a look at some performance measurement frameworks that have emerged over the course of the 21^st century, beginning with external assessment tools – frameworks used by oversight bodies to evaluate regulators and hold them accountable for their performance.

External assessment tools

U.K. Legal Services Board

The U.K.’s Legal Services Board (LSB) utilizes a performance framework that they describe as  “proportionate, risk-based, and evidence-based.” The overall goal of the framework is to make sure legal services regulators in the U.K. are held to account for their performance. Its current iteration, which went into effect at the beginning of 2023, attempts to help determine whether regulators meet three basic performance standards:

1. Well-led – Whether agencies have the leadership required to effectively meet regulatory objectives.
2. Effective approach to regulation – The extent to which regulators act on behalf of the public to identify opportunities and address risks.
3. Operational delivery – The extent to which regulatory activity is effective and pertinent to the public interest.

When using the framework, regulators are expected to provide evidence of how their own programs are designed to meet regulatory objectives and deliver their desired outcomes as well as evidence that these organizations meet the standards befitting an agency that is effective and well-led.

U.K. Professional Standards Authority for Health and Social Care: Standards of Good Regulation

The Professional Standards Authority (PSA) reviews the work of 10 health and care regulators on a 12-month rolling basis. The most recent iteration of its performance measurement framework assesses regulatory performance based on 18 standards which are divided into the following categories of performance:

• General standards – i.e., the regulator provides accurate information about the way it operates, is clear about its purposes, and reports on its performance regularly.
• Guidance and standards – i.e., the regulator maintains up-to-date standards for registrants and provides guidance to help them apply these standards.
• Education and training – i.e., the regulator maintains up-to-date standards for education and training and ensures educational providers deliver satisfactory services.
• Registration – i.e., the regulator operates an efficient registration system and maintains an accurate, up-to-date register.
• Fitness to practice – i.e., the regulator has an accessible, thorough, and effective complaint process that deals with cases quickly and prioritizes public protection.

These standards intend to prioritize the core role of health and care service regulators in protecting patients and service users, preventing harm, promoting professional standards, and maintaining public confidence in the health and care professions. According to the PSA, the standards follow the principles of good regulation – that regulators should be proportionate, consistent, targeted, transparent, accountable, and agile in their work.

Self-assessment tools

Listening/Learning/Leading: A Framework for Regulatory Excellence

In mid-2014, the Alberta Energy Regulator (AER) worked together with the Penn Program on Regulation (PPR) at the University of Pennsylvania in an attempt to answer the question of how regulatory excellence may be defined. Its research comprised four major dialogue sessions involving more than 150 participants with knowledge of energy regulation in the province and in general, as well as interviews with more than 60 other stakeholders.

In its research, the PPR identified three basic ways of thinking about regulatory excellence. These domains are listed below, along with a few examples of each:

1. Organizational traits – i.e., culture, resources, human capital, and autonomy.
2. Actions – i.e., public engagement, decision-making, rulemaking, incident response, and evaluation.
3. Outcomes – i.e., industry behavior and response, perpetual outcomes, and substantial outcomes.

More specifically, the PPR identified nine tenets of regulatory excellence, which they describe as “the most significant, comprehensive but still compact aspirational statements of regulatory excellence” and are intended to align with three core attributes:

1. Utmost integrity
   1. Fidelity to law.
   2. Respect for democracy.
   3. Commitment to public interest.
2. Empathetic engagement
   1. Even-handedness (treating all affected interests fairly).
   2. Listening.
   3. Responsiveness.
3. Stellar competence
   1. Analytical capability.
   2. Instrumental capacity (i.e., a sufficiently funded and highly trained staff).
   3. High performance.

The PPR report discusses performance measurement in mostly qualitative terms. It argues there is no one-size-fits-all solution to gauging regulatory excellence – that instead, regulators can focus either narrowly or broadly on the regulator’s own actions or trends within the regulated area at large. According to the report, regulators benefit most from the right combination of broad measurements, which can help alert regulators to general patterns, and narrow measurements, which can inform staff about specific conditions.

OECD Framework for Regulatory Policy Evaluation

The Organization for Economic Cooperation and Development (OECD) in 2014 published a framework intended to assist countries in evaluating the design and implementation of regulatory policy, analyzing how policy decisions help to achieve strategic large-scale objectives. Bearing some overlap with the PPR framework for measuring regulatory excellence, the OECD model seeks to collect data on:

• Input – What resources are committed to a policy goal?
• Process – Are the requirements for good regulatory practices in place (i.e., requirements for objective setting, consultation, evidence-based analysis, simplification, risk analysis)?
• Output – Have the good practices been implemented?
• Intermediate outcomes – Have good practices helped to get quality regulation?
• Regulatory quality – Have strategic objectives for regulatory policy in general been achieved?
• Regulatory output – Have strategic objectives for regulatory policy in a specific sector been achieved?

According to the OECD, the Framework for Regulatory Policy Evaluation is structured to help countries evaluate each step of the regulatory process, from the design and implementation of policy to the achievement of strategic regulatory objectives. Its full application requires comprehensive data and information from the countries that use it, which the report acknowledges is not systematically collected by most countries quite yet.

Ontario’s College Performance Measurement Framework (CPMF)

The Ontario Ministry of Health developed the College Performance Measurement Framework in close collaboration with the province’s health regulatory colleges to help strengthen accountability and oversight of these organizations and help them to improve their performance overall.

Though reports must be submitted to the Ministry of Health, the CPMF relies substantially on self-assessment from participating organizations. Similar in structure to the PSA’s model, it identifies seven domains of performance which comprise 14 standards altogether. The CPMF’s domains are:

• Governance – i.e., the expertise and competence of council and statutory committee members.
• Resources – i.e., responsible stewardship of a college’s financial and human resources.
• System partner – i.e., active engagement between the college and other regulatory bodies to align oversight of the regulated professions.
• Information management – i.e., the security and protection of information collected by the college.
• Regulatory policies – i.e., the alignment of regulatory policies, standards of practice, and practice guidelines, with changing public expectations and their reflection of current best practices.
• Suitability to practice – i.e., the establishment and maintenance of processes and procedures that assess competency, safety, and ethics in regulated professionals.
• Measurement, reporting, and improvement – i.e., the accurate, up-to-date monitoring, reporting, and improvement of regulatory performance.

Australian Regulator Performance Framework

Australia’s Regulator Performance Framework, used by agencies throughout the country to self-report on their performance, provides six key performance indicators (KPIs):

1. “Regulators do not unnecessarily impede the efficient operation of regulated entities.”
2. “Communication with regulated entities is clear, targeted, and effective.”
3. “Actions undertaken by regulators are proportionate to the regulatory risk being managed.”
4. “Compliance and monitoring approaches are streamlined and coordinated.”
5. “Regulators are open and transparent in their dealings with regulated entities.”
6. “Regulators actively contribute to the continuous improvement of regulatory frameworks.”

The framework also outlines a process for annual self-assessment, targeted external review every three years (for a sample of specific regulators), and annual external review (for another specific set of major regulators). It includes several measures of good performance, which can be used by regulators to evaluate their performance according to the indicators.

Performance Measurement Framework for the Canadian Transportation Agency

The need for a performance measurement instrument for the Canadian Transport Agency (CTA) was evident around the turn of the century when the Treasury Board released its Management Framework for the Government of Canada. In response to these calls, the CTA created its own Performance Measurement Framework, which intends to enhance the management and reporting of the CTA’s programs and enable managers to make more effective choices.

The CTA’s framework is guided by five core principles:

1. Outcomes and results must be clearly defined.
2. The performance measurement system should be simple and cost-effective.
3. The system should be positive, not punitive.
4. Performance indicators should be simple, valid, reliable, affordable, and relevant.
5. Performance indicators should be reviewed and improved on an ongoing basis.

The framework encourages organizations to select their own performance indicators – it recommends three for each performance measure – according to the following criteria:

1. Validity – does the indicator enable precision in measuring results?
2. Relevance – is the indicator relevant to the activity, product, or process being measured?
3. Reliability – is it a consistent measure over time?
4. Simplicity – is the information available and can it be feasibly collected and analyzed?
5. Affordability – can the organization afford to collect and analyze the information?

Using a results chain which asks why a program or initiative is being carried out, what the agency expects to see as a result of its outputs, who needs to be engaged, and how the agency’s activities can help it achieve its desired strategic outcomes, the framework then instructs agencies to create a performance measurement plan that specifies outcomes, performance indicators, data sources, collection methods, frequency of data collection, and personnel responsible for data collection.

Lastly, the framework encourages CTA officials to effectively communicate their results, tracking them in their annual Performance Reports and conveying them internally to senior management and the executive committee. According to the CTA, this enables the agency to develop best practices and record lessons learned which can strengthen management practices and inform future programs, processes, and initiatives.

Ascend Institute Regulatory Transformation Framework and AIRE Index

The Ascend Institute Regulatory Transformation Framework offers a holistic regulatory improvement approach built around a simple self-assessment tool, the Ascend Institute Regulatory Effectiveness (AIRE) Index. The AIRE Index attempts to capture the basic and essential elements of regulatory activity. It covers five regulatory functions and seven requirements of organizational management.

Our own team of regulatory experts has endeavored to create a modern and holistic tool to measure regulatory effectiveness. The five regulatory functions it covers are listed below:

• Statutory framework
  • This includes legislation/statute, regulations, bylaws, and rules – the formal requirements that the regulator should follow.
• Licensing and registration
  • This could include education, training, prior experience, entry-level competence assessment, background checks, and other requirements.
• Setting standards
  • This involves the creation, update, and publishing of risk-based standards for conduct and practice by the regulator.
• Ensuring compliance
  • This domain concerns how the regulator ensures that its regulated entities comply with the standards it has put forth.
• Investigations and discipline
  • This involves procedure for receiving regulatory complaints as well as investigating and determining fair and appropriate outcomes.

The AIRE Index’s organizational management areas are as follows:

• Strategic planning – how the regulator looks toward the future and plans for potential and actual changes, as well as its ability to set the direction of travel.
• Operational planning and delivery – the extent to which a regulator’s operational/business plan delivers on its strategic plan.
• Financial management – how the regulator manages its finances by setting a budget aligned with its strategic and operational plans.
• Risk management – strategic and operational risks to the organization, and how these risks are measured and monitored.
• Information governance – the regulator’s overall policy for ensuring that information is safe and secure and that all relevant data protection and privacy laws are complied with.
• Policy development and renewal – the regulator’s internal processes, accountabilities, and approvals for standards, procedures, and operational guidance.
• Learning and improvement – the extent to which the organization uses data to learn about its own performance and improve over time.

In total, the Index asks organizations to self-assess 23 components of five regulatory functions and 26 components of seven management functions. Regulators are encouraged to score themselves on each component to identify strengths and weaknesses and identify areas for improvement. These scores can also help regulators compare and benchmark themselves in an anonymized way against one another if they so choose to share their results with the Institute.

The framework also includes a checklist of questions covering all regulatory functions and organizational management areas that agencies can go through to check whether their self-assessment was reasonable and to inform their strategic and transformation priorities. Organizations can then consult the Ascend Institute’s soon-to-be-published resource library for advice and guidance in taking the steps toward transformation.

Using frameworks to improve the work of regulation

It is clear from even this limited set of examples that there are many ways to view the concept of regulatory performance and many ways to measure it. Though some overlaps do exist between them, not all performance measurement tools are equal, so where there is a choice, regulators can assess each framework according to its scope and choose the most appropriate one for their needs.