High-Impact Decisions: Report Card Grades Regulators

The Mercatus center at george Mason university has a new report card to grade the executive branch on high-priced decisions. it encourages regulatory impact analysisa tool presidents have tried to get agencies to apply for decades.

A few years ago at home, I found water on the bathroom floor. After cleaning up the water with some old rags, my family and I had to use trial-and-error to find the leaks source. We finally found a cracked cold water pipe. We also determined that the crack occurred because the toilet rocked back and forth, which put pressure on the pipe. Armed with this knowledge of the systemic problem, we replaced the broken piece of pipe, leveled the toilet, and fixed the leak permanently.

Regulatory agencies often go no further in analyzing the problems theyre trying to solve than saying,Look, theres water on the floor. Since this definition of the problem seems obvious, decision makers perceive no need to examine how the water got there. The solution seems similarly simple:Buy a mopan expenditure that may be unnecessary (since an old rag will work just as well) and doesnt really fix the root cause of the problem. This approach produces a costly perpetual mop mandate without stopping the leak.

According to the Office of Management and Budget, federal regulations produce tens of billions of dollars worth of benefits and costs each year. Since the 1970s, U.S. presidents have tried to get executive branch agencies to fix more cracked pipes and impose fewer mop mandates. Executive orders instruct agencies to conduct regulatory impact analyses and consider the results of that analysis when making decisions. Agencies are supposed to

1| define the outcome or outcomes they seek to achieve

2| understand the root causes of the problem that stand in the way of achieving the desired outcomes

3| develop diverse alternative ways to solve the problem

4| assess the pros and cons of each alternative.

Shorn of all the economics jargon, regulatory impact analysis is really Decision Making 101 applied to regulation.

For years two documents set the standards federal agencies must follow when analyzing proposed regulations: Executive Order 12866, adopted by President Clinton in 1993; and Circular A-4, a guidance document issued by OMB in 2003. In January 2011, President Obamas Executive Order 13563 reaffirmed the basic principles and processes that have guided executive branch regulatory analysis and review of proposed regulations for several decades.

How well do agencies actually perform the required regulatory analysis when they propose regulations? To what extent do they use it when making decisions?

Created by The Mercatus Center at George Mason University, the Regulatory Report Card seeks to answer those questions. The report card identifies how well the regulatory impact analysis for a proposed regulation implements the directives in Executive Order 12866 and OMB Circular A-4. The report card covers proposed regulations the federal government considers economically significantusually involving costs, benefits, federal spending, or other economic effects exceeding $100 million annually. These regulations address a wide variety of topics, including capital standards for banks, air pollution, Medicare reimbursement rates for hospitals, prison rape, education reform, and duck hunting. The first round of evaluations covered regulations proposed in 2008. (Evaluations and detailed scoring notes for each regulation, as well as links to the projects research papers, are available at www.mercatus.org/reportcard.)

Evaluation Criteria and Method

The regulatory report card consists of 12 criteria broken down into three categories: openness, analysis, and use. Figure 1 lists the questions under each category. For each question, the regulatory analysis receives a score from zero points (no relevant content) to five points (reasonably complete analysis with one or more best practices). Figure 2 shows the scoring guidelines.

-------------------------------------------------------------------------------------------

Figure 1| Regulatory Analysis Assessment Criteria

Openness

1. accessibility: how easily were the regulatory impact analysis, the proposed rule, and any supplementary materials found online?
2. Data documentation: how verifiable are the data used in the analysis?
3. Model documentation: how verifiable are the models and assumptions used in the analysis?
4. clarity: Was the agencys analysis comprehensible to an informed layperson?

Analysis

1. Outcomes: how well does the analysis identify the desired outcomes and demonstrate that the regulation will achieve them?
2. Systemic problem: how well does the analysis identify and demonstrate the existence of a market failure or other systemic problem the regulation is supposed to solve?
3. alternatives: how well does the analysis assess the effectiveness of alternative approaches?
4. benefit-cost analysis: how well does the analysis assess costs and benefits?

Use

1. use of analysis: Does the proposed rule or the regulatory impact analysis present evidence that the agency used the analysis?
2. net benefits: Did the agency maximize net benefits or explain why it chose another option?
3. Measures and goals: Does the proposed rule establish measures and goals that can be used to track the regulations results in the future?
4. retrospective data: Did the agency indicate what data it will use to assess the regulations performance in the future and establish provisions for doing so?

Source: Jerry Ellig and Patrick McLaughlin, The Quality and Use of Regulatory Analysis in 2008, Working Paper, Mercatus Center at George Mason University (June 22, 2010), p. 6, available at http://econpapers. repec.org/RAS/pdn2.htm#author-article.

-------------------------------------------------------------------------------------

--------------------------------------------------------------------------------------

Figure 2| What Do the Report Card Scores Mean?

Source: Jerry Ellig and Patrick McLaughlin, The Quality and Use of Regulatory Analysis in 2008, Working Paper, Mercatus Center at George Mason University (June 22, 2010), p. 7, available at http://econpapers.repec.org/RAS/pdn2.htm#author-article.

--------------------------------------------------------------------------------------

At least one senior economist and a graduate student with training in regulatory analysis evaluate each notice of proposed rulemaking and its accompanying analysis. The evaluators discuss scoring differences to achieve consensus and consult prior evaluations to maintain consistency. The entire evaluation team underwent a rigorous training process; the team leader also reviews all evaluations to ensure consistency.

Prior scholarly research on the quality and use of regulatory analysis has taken two approaches. In one approach, case studies have assessed the quality and use of analysis in depth for small numbers of very significant regulations, as in Reforming Regulatory Impact Analysis, a report published by Resources for the Future Press (2009). In the second, quantitative studies have employed objective checklists to evaluate a larger set of health, safety, and environmental regulations, as reported in the Has Economic Analysis Improved Regulatory Decisions? article in Journal of Economic Perspectives (Winter 2008). The report card takes a middle ground, offering a limited qualitative assessment for a wide variety of regulations.

Significant Opportunities for Improvement

There are clearly significant opportunities for improvement, as Figure 3 indicates. For each criterion, at least one regulatory analysis received the highest possible score of five in one of the two years. In most cases, however, average scores were substantially below five. With the exception of criterion 1, which assesses availability of the proposed regulation and accompanying analysis on the Internet, few regulations received the highest possible score. This indicates that many agency regulatory analyses are seriously incomplete. More widespread adoption of best practices would substantially improve the quality of agency regulatory analysis.

Some areas need more improvement than others. For example, the two lowest-scoring criteria in each year are criteria 11 and 12. These assess whether the agency provided for retrospective analysis by identifying goals, measures, and data that could be used to assess the regulations actual benefits and costs after it is implemented.

Executive Order 12866 and OMB Circular A-4 give agencies scant guidance on how to do this, but President Obamas Executive Order 13563 requires agencies to establish plans for retrospective analysis. The Government Performance and Results Modernization Act of 2010 requires the federal government to identify all programs, tax expenditures, and regulations that contribute to high-priority agency or government-wide goals and regularly monitor performance. Thus, retrospective analysis will likely grow in importance in the future. Logically, it makes sense for agencies to lay the groundwork for retrospective analysis in the work that attempts to project the regulations likely benefits and costs.

The next lowest score is on criterion 6, identification of the systemic problem the regulation is supposed to solve. This is a key weakness. A systemic problem is a widespread problem that can be traced to a defect in the rules of the game that govern behavioras opposed

to the faults of a few bad actors that can be dealt with on a case-by-case basis. If the agency cannot identify and demonstrate the existence of a systemic problem that a regulation might solve, how can it assess whether the regulation is likely to solve the problem or identify alternative solutions that might be more effective? This is like saying the problem is water on the bathroom floor, without searching for the cracked pipe or finding out why it cracked.

Examples of Best Practices

For each criterion, there are examples of best practices from which other agencies can learn. For instance, the sole regulation evaluated in 2008-09 that received five points for defining the systemic problem was proposed by the Department of Housing and Urban Development in 2008 under the Real Estate Settlement Procedures Act. The regulation would have revised the disclosures consumers receive about certain real estate settlement charges related to mortgages. HUD intended to reduce settlement costs by making the charges easier to understand and compare across different lenders.

-------------------------------------------------------------------------------

Figure 3| Report Card Scores for 2008 and 2009

Source: Jerry Ellig and John Morrall, Assessing the Quality of Regulatory Analysis: A New Evaluation and Data Set for Policy Research, Working Paper, Mercatus Center at George Mason University (Dec. 15, 2010), mercatus.org/publication/assessing-quality-regulatory-analysis.

-------------------------------------------------------------------------------

In defining the problem, HUDs regulatory analysis suggested that the complexity of real estate transactions and some borrowers lack of information allow mortgage providers to collect higher fees from less informed or less sophisticated borrowers. Charging different customers different prices is not necessarily evidence of a market failure, because it does not necessarily lead to economic inefficiency. Car dealers, airlines, and universities often charge different customers different prices based on the customers sophistication, knowledge, or perceived willingness to pay. The practice strikes many people as unfair. It is arguably inefficient if the transaction or the disclosures are so complex that a significant subset of customers does not understand them well enough to compare competing loan offers.

Whether the problem is inefficiency or inequity or both, HUDs analysis appropriately identified a systemic root cause. The analysis offered a coherent theory explaining how the information problem could allow mortgage providers to charge some customers higher fees than others. It even explained why this pricing practice might not produce a smoking gun of excessive profits for mortgage lenders or brokers: the firms may find they have to pay out most of the rewards to salespeople who are especially skilled at inducing less-informed customers to over-pay for loans.

In addition to a coherent theory, HUD offered empirical evidence. The analysis cited several studies by government entities and consulting firms that found consumers with less education, no financial counseling, or more complex shopping strategies tended to pay more for loans and settlement services. About the only faults the Mercatus Center evaluators could find with HUDs analysis of the systemic problem were that one study with results contradictory to HUDs was merely mentioned in a footnote rather than fully addressed, and the analysis did not completely assess uncertainties about the existence or size of the problem.

Theres also a good example of best practices for the retrospective analysis criteria. In 2008, the Department of Homeland Security proposed a rule requiring finger scans of aliens departing the United States via air and sea. The accompanying regulatory impact analysis explained how the projected benefits of the proposed regulation relate to the departments strategic goals and its proposed performance measures. The appendix also outlines more specific outcome performance measures based on the benefits projected for the rule. (Figure 4 presents some examples.) The Federal Register notice does not explicitly commit to evaluating the regulation in this way, but the regulatory impact analysis gives the impression that it will be done. The approach is sufficiently innovative that it deserves the high score it received.

Other Fascinating Findings

Along with providing insights into best practices, the report card project has generated other interesting revelations:

• Regulations that implement federal spending programs have much lower-quality analysis than prescriptive regulations that require or prohibit actions by private parties or other levels of government.
• There are few differences in the quality of regulatory analysis between 2008, the last year of the Bush administration, and 2009, the first year of the Obama administration.
• Regulations score best for accessibility via the Internet but the high grade doesnt reveal whether the posted analysis is any good.
• Better quality analysis and greater use of analysis go hand in hand. Regulations that score higher on the openness and analysis criteria also tend to score higher on the use criteria.

These findings are consistent with previous scholarship that has evaluated the quality and use of regulatory analysis via case studies or checklists. Unlike previous studies, the report card assesses all economically significant proposed regulations over several years. It demonstrates that the findings of prior studies of some regulations are actually typical for federal regulations. While this research project reveals some problems with agency regulatory analyses, it also points the way toward solutions by revealing important patterns and highlighting best practices.

-----------------------------------------------------------------------------

Figure 4| DHS Regulatory Analysis Links Goals

This is a portion of a figure in DhS regulatory analysis that links departmental strategic goals, program goals, and goals and measures for the proposed regulation

Source: U.S. Department of Homeland Security, Air-Sea Biometric Exit Project Regulatory Impact Analysis (April 17, 2008), p. 67.