Step 09 attachment 6 Reviewer Instruction Guide_8-6-13



[pic]

Division of Independent Review (DIR)

Reviewer Instruction Guide

Web-Teleconference and Face-to-Face

TABLE OF CONTENTS

1 INTRODUCTION 2

2 OVERVIEW 2

3 HRSA RESPONSIBILITIES 2

3.1 Review Administrator 2

3.2 Program Staff 3

3.3 Grants Management Specialist 3

3.4 DIR Contractor 3

4 REVIEWER RESPONSIBILITIES 4

4.1 Conflict of Interest 4

4.2 Confidentiality 5

5 CHAIRPERSON RESPONSIBILITIES 5

6 REVIEW PROCESS 5

6.1 Pre-Review 5

6.2 Review 7

6.3 Post-Review 9

ATTACHMENT A—GUIDELINES FOR WRITING STRENGTHS AND WEAKNESSES 11

ATTACHMENT B - WHAT DO I DO WITH THE BUDGET: A GUIDE FOR HRSA REVIEWERS…………………………………………………………………………………………..….13

ATTACHMENT C - HRSA SCORING RUBRIC………………………………………………………..16

1 INTRODUCTION

The Health Resources and Services Administration (HRSA) Division of Independent Review (DIR) is responsible for planning, directing, and managing the objective review of new, competing continuation, and program expansion supplement applications for HRSA discretionary grant and cooperative agreement programs. The primary purpose of an objective review is to ensure that each application receives a fair and objective assessment consistent with the published review criteria.

HRSA achieves this goal with the assistance of Reviewers who possess the expertise and/or skill-sets appropriate for the program under the Funding Opportunity Announcement. Reviewers are expected to provide a comprehensive evaluation of the merits of applications. HRSA program officials use those evaluations to make award decisions.

This document provides instructions for Reviewers, including a timeline for deliverables associated with the review process, and describes roles and responsibilities of HRSA and DIR Contractor staff involved in the review process.

2 OVERVIEW

DIR conducts objective reviews using the Application Review Module (ARM), a web-based application review tool. It is a user-friendly system designed specifically to meet the objective review requirements of HRSA Grant programs and provide a cost effective and efficient system to review grant applications. Reviewers will use this tool when evaluating and scoring assigned applications. For ARM specific procedures and instructions, Reviewers should refer to the separate ARM Reviewer Manual.

Reviewers begin the process by independently evaluating and scoring assigned applications. For Teleconferences, Reviewers then convene via telephone and the internet as an objective review committee consisting of one or more review panels to discuss the applications. For Face-to-Face review meetings, Reviewers then convene as an objective review committee consisting of multiple review panels at a designated central location, usually in the Washington, DC metro area, to discuss the applications.

Each application is assigned to three Reviewers. Prior to the meeting, Reviewers evaluate and score their assigned applications based on how thoroughly they determine each application meets the published review criteria in the program Funding Opportunity Announcement (FOA). Reviewers record their initial scores and note strengths and weaknesses by criterion within ARM. During the objective review panel meeting, the assigned Reviewers will be asked to present their evaluations and scores for each application. Following the presentations, the full panel will discuss each application to develop a final summary statement. Each Reviewer on the panel will enter and submit final criterion scores for each application into ARM. ARM calculates an overall score for each application based on the average of the individual Reviewer’s scores. See section 6, “Review Process,” provides detailed instructions for Reviewers in the conduct of the review.

3 HRSA RESPONSIBILITIES

3.1 Review Administrator

A Federal Review Administrator (RA) is assigned to each review. The RA manages the review process and provides assistance to Chairs and Reviewers to ensure compliance with applicable statutes, regulations, and policies and to maintain an objective review process.

Typically, the RA:

• Ensures the review is fair and objective;

• Provides oversight and direction during the objective review process;

• Manages Reviewer potential or actual conflicts of interest and ensures that Reviewers maintain the confidentiality of applications and review information throughout the review process;

• Convenes and facilitates the mandatory pre-review conference call;

• Oversees the objective review panel discussions, including the production of summary statements and scoring of applications; and

• Participates in the Reviewer evaluation and post-review debriefing session.

3.2 Program Staff

HRSA program staff members serve as a resource during the objective review process to ensure that program policies, procedures, and goals are carried out in accordance with prescribed requirements (i.e., legislative requirements, HRSA policy, and program-specific policy). Specifically, program staff:

• Participate in the pre-review conference call;

• Upon request, provide program expertise, particularly regarding application materials, application guidance, legislation, review criteria, and applicable funding factors (preferences, priorities, and/or special considerations); and

• Respond to any questions that require further explanation regarding programmatic issues.

3.3. Grants Management Specialist

Grants Management Specialists (GMS) are responsible for ensuring that applicants and applications comply with HRSA statutes, regulations, and policies that pertain to financial and business management matters. In collaboration with program staff, grants management staff screen applications to determine eligibility. A GMS is assigned to each review as a resource. Specifically they:

• Participate in the pre-review conference call;

• Respond to any questions that require further explanation regarding grants management issues, such as:

o Whether budget items are allowable;

o Applicant eligibility;

o Applicant compliance with required assurances; and

o Federal and HRSA grants policy, regulations, and procedures.

3.4 DIR Contractor

A contractor assists DIR RAs by providing technical, administrative, and logistical support services for the objective review. Specifically, the contractor assists with:

• Distributing review materials;

• Collecting and tracking of documents received from Reviewers;

• Consolidating individual Reviewer evaluations prior to the objective review panel meeting on each application; and

• Coordinating payment of honoraria to Reviewers.

Any Reviewer questions that are not addressed in the pre-review conference call should be addressed to the RA.

4 REVIEWER RESPONSIBILITIES

The primary responsibility of a Reviewer is to independently, fairly, and competently evaluate the merits of applications consistent with the program purpose and published review criteria. Each Reviewer is generally assigned to read and score up to eight grants or cooperative agreement applications. The number of applications assigned may vary depending on the program complexity, the number of eligible applications received, length of the applications and the time available for independent reviewing. All current and potential reviewers must attend a HRSA DIR Quarterly Reviewer Training session.

It is imperative that Reviewers follow the DIR Reviewer Instruction Guide and the FOA and complete assigned tasks in accordance with established deadlines. The potential integrity of the review process may be compromised if a Reviewer does not complete the assigned work or does not deliver the work on schedule. Reviewers must make all necessary arrangements to be available to complete review assignments, attend and participate in the entire objective review committee meeting. If for some reason a Reviewer believes that they are unable to fulfill these duties, or if unforeseen circumstances prevent the Reviewer from completing their duties, they must immediately contact their assigned RA and or the contractor representative. Honorarium will be paid only after all of the assigned responsibilities have been fulfilled.

4.1 Conflict of Interest

During the pre-review conference call, the RA will provide additional guidance regarding conflicts of interest. To achieve the goal of a fair and objective review, Reviewers must not carry any prejudices, biases or conflicts of interest into the review process. Application and panel assignments are made by DIR in consultation with Program staff to avoid biases, e.g., applications are assigned to Reviewers from a different geographic area than the applicant. The Reviewer is obligated to declare any and all potential or actual conflicts of interest, as soon as possible.

Upon receiving the review materials, all Reviewers are required to complete, sign, date and return the Conflict of Interest and Confidentiality Certification Form as directed. The Reviewer can attest to having no conflict of interest with the assigned applications or can include details regarding potential or actual conflicts of interest. In the latter situation, the DIR RA will reassign all applications to comply with HRSA policy prohibiting the participation of Reviewers where he or she has a potential or actual conflict of interest regarding any application to be reviewed by the objective review committee. No person who serves as an officer, director, trustee, partner, or employee of an applicant, its parent or subsidiary organization may serve as a Reviewer on the committee.

If the Reviewer believes that there exists a conflict of interest with any assigned application for review, or if it could be perceived that there is a conflict of interest, they must immediately inform the RA. It is never too late to disclose a conflict of interest; however, Reviewers should attempt to identify any conflict of interest as early as possible to avoid compromising the integrity of the review.

At the beginning of each application’s review, the Chairperson will ask the Reviewers to affirm that no potential or actual conflict exists. In the rare instance in which a Reviewer discovers that he or she has a conflict of interest or a reasonably perceived conflict, he or she will be recused for the duration of that application’s review. For Teleconferences, the Chairperson will be asked to note that the reviewer was recused for conflict of interest, and the reviewer will disconnect from the teleconference until notified via phone call to rejoin the review. For Face-to-Face review meetings, the recused Reviewer will be asked to note on the score sheet that he or she was recused for conflict of interest, sign a score sheet (without providing a score), and will leave the room until notified the discussion of that application has concluded.

4.2 Confidentiality

All review materials and the proceedings of objective review committee meetings are privileged information for use only by HRSA staff, DIR Contractors, and assigned Reviewers. At no time prior to, during, or after the review of applications should Reviewers discuss the applications, comments, recommendations, evaluations, scores, names of applicants, or names of other Reviewers with anyone not involved in the objective review process. Questions regarding applications to be reviewed should be directed to the RA. Direct contact or communication with applicants or associated persons is strictly prohibited.

Please note that all comments made by the Reviewer are treated as confidential information. In situations where an applicant (or other individual) requests review-related information via the Freedom of Information Act (FOIA), the Federal Government is required to release the names of all Reviewers who participated in the review activity; however, the names of the individual Reviewers will NOT be associated with specific application assignments.

5 CHAIRPERSON RESPONSIBILITIES

Each objective review panel has a Chairperson whose primary responsibility is to preside over the panel discussions and assure that each application is accorded a fair, ethical and equitable review. The Chairperson must also manage the schedule to assure timely yet thorough review of all applications. The Chairperson does not participate as a Reviewer and therefore does not contribute subject matter to the discussion, assist in the determination of an application’s merit, or score individual applications. The Chairperson does ensure that the contributions, views, and questions of all Reviewers are heard and given equal consideration, and manages the development of the final summary statement and the editing process. At the end of the review, the Chairperson certifies the results of the review.

6 REVIEW PROCESS

6.1 Pre-Review

6.1.1 Pre-Review Conference Call

Reviewers and Chairpersons are required to participate in a pre-review conference call with the RA, Grants Management staff, and the Program staff. The purpose of the conference call is to provide an overview of the review process and give Reviewers the opportunity to address with HRSA staff any questions or issues regarding review policies and procedures. If a Reviewer is unable to attend, an acceptable explanation must be discussed with the contractor and RA in advance of the conference call. Instructions on how to access the recorded conference call will be provided and the Reviewer must make time to listen to the recorded session prior to the objective review committee meeting and report to the contractor and RA that they have listened to the recorded session. Subjects addressed during the pre-review conference call include:

• Confirmation regarding access to review materials in ARM;

• General information regarding the objective review process;

• Presentation of Reviewer responsibilities including confidentiality and conflicts of interest;

• Guidance on developing and writing clear, concise, and accurate evaluations;

• Questions regarding honoraria ;

• Questions regarding program objectives or background information (Program);

• Questions regarding grants policy or application budget considerations (Grants Management); and,

• Questions regarding review logistics.

Immediately after the conference call, an ARM login and password as well as other information required to begin the evaluation process will be emailed to each Reviewer.

Any Reviewer questions that arise during or after the pre-review conference call should be addressed to the RA.

6.1.2 Review of Applications and Preparation and Submission of Preliminary Documents

Reviewers are required to read each assigned application in its entirety, evaluate and score each one based on the published review criteria, and enter their scores and comments in ARM. Please note that reviewers are required to evaluate and score only those applications assigned to them on the Reviewer Assignment Sheet. Reviewers must also read and familiarize themselves with the other applications assigned to their panel.

6.1.2.1 Completing the Evaluation

To complete the evaluation, Reviewers must enter the scores by criterion as well as the strengths and weaknesses for each assigned application in ARM (ARM will calculate the total score). For assistance in preparing these comments, see Attachment A— Guidelines for Writing Strengths and Weaknesses.

Reviewers must independently consider and separately score each criterion. The total score for an application (calculated by ARM) is the sum of all the criteria scores. All Reviewer criterion scores must be justified with written strengths or weaknesses. It is recommended that Reviewers read the entire application before entering their scores and comments in ARM. In addition to scores, strengths and weaknesses, Reviewers enter budget recommendations and, if applicable, funding factors. When Reviewers complete the evaluation in ARM, they must click on the “Submit to Chair” button. Please note that once this process is completed, Reviewers are no longer able to modify their scores or comments. In preparation for the objective review committee meeting, reviewers should print the merged evaluations for reference and presentation of their assigned applications. Saving the document for reference at the objective review is encouraged.

6.1.2.2 Consolidation of Reviewer Evaluations

The DIR Contractor will consolidate the reviewer evaluations for each application within ARM to create a draft summary statement. During the panel discussions, the draft summary statement will be displayed to the panel as they recommend edits and additional comments. The final summary statement will be sent to the applicant organization.

6.1.2.3 Descriptive Overview

This is presented by one of the assigned primary Reviewers for each application. The overview should focus on the “big picture” of who the applicant is, where it is located, what is being proposed, how it will be accomplished in accordance with the published FOA review criteria, and identify the most significant strength and weakness. Additional strengths and weaknesses will be presented and considered during the discussion of the application and formulation of the final summary statement. The overview will include:

• Applicant Information;

• Target population, service area, appropriateness of budget, etc.;

• Proposed project description;

• Major goals and objectives;

• Any outstanding strengths or weakness; and

• Any other pertinent information, such as Funding Factors, if applicable.

6.2 Review

Reviewers are required to read all applications assigned for review by their panel.

Teleconference Review

Once all Reviewers have joined the teleconference, the Chairperson will facilitate the meeting process:

➢ Introduction: Chairperson will introduce the first application to be reviewed

o Conflict of Interest: Chairperson will ask if there are any potential or actual conflicts of interest with any applications under consideration. If there are no conflicts of interest, the panel will proceed with the first application.

o In the rare instance that it is discovered a conflict of interest exists, the Chairperson will ask the Reviewer to recuse him or herself for the duration of the application’s review. The recused Reviewer will not score the application and disconnect from the teleconference until notified by telephone to rejoin the review. The Chairperson will note the recusal.

Face-to-Face Review

Upon arrival at the review location, each Reviewer will receive a review package with the following items:

• Meeting agenda;

• Schedule of applications for review by Reviewer assignment (i.e., primary, secondary, tertiary);

• Final scoring sheets for all applications;

• Evaluation forms;

• Honorarium and reimbursement form;

• Participant list; and

• Other logistical information (i.e., hotel and/or building information, transportation information, nearby restaurants, parking, contact numbers).

CDs of the application may be available in the meeting room for reference. To facilitate discussion and review, Reviewers are encouraged to bring laptops. Reviewers are required to read, evaluate and score all applications assigned to them on the Reviewer Assignment Sheet. Prior to the review meeting, Reviewers should read non-assigned applications in their panel. Questions and comments related to non-assigned applications should be brought for discussion on the day of the review.

Once all Reviewers have assembled in the meeting room, the Chairperson will facilitate the meeting process:

• Introduction: Chairperson will introduce the first application to be reviewed.

o Conflict of Interest: Chairperson will ask if there are any potential or actual conflicts of interest with the application under consideration. If there are no conflicts of interest, the panel will proceed.

o If a conflict of interest exists, the Chairperson will ask the Reviewer to recuse him or herself for the duration of the application’s review. The recused Reviewer will not score the application and will leave the panel room until summoned to rejoin the review. The Chairperson will note the recusal.

Both types of Review

• Overview and Open Discussion:

o First Reviewer will give a descriptive overview of the application as described under Section 6.1.2.3 above.

o Second and Third Reviewers give any additions, enhancements, or clarifications to the first Reviewer’s overview.

o Each assigned Reviewer announces their initial scores by criterion and includes the total score.

o Chairperson will invite discussion from all members of the review panel.

o Reviewers discuss the application including differences presented by the First, Second, and Third Reviewers.

o Full panel discussion and participation is expected.

o The overview and open discussion should not exceed 20 minutes.

• Display the Draft Summary Statement and Edit:

o The Chairperson will ask the Summary Statement Operator (SSO) to project the draft Summary Statement.

o The Chairperson will ask the First Reviewer to read aloud and lead the group editing of the projected summary statement. Differences of opinion regarding strengths and weaknesses noted by the First, Second, and Third Reviewers are to be discussed to derive group consensus. Contradictory statements are to be avoided.

o The SSO will make edits to the displayed summary statement based on the group discussion

o Reviewers will focus on content with emphasis on accuracy and consistency. Final editing of grammar, punctuation, etc., will occur prior to dissemination of the summary statement.

o Full panel discussion and participation are expected.

o The final summary statement will reflect a consensus of major strengths and weaknesses.

Teleconference Review

➢ Scoring Application2:

o Chairperson asks the SSO to return the evaluations to the reviewers in ARM.

o The reviewers will enter their scores by criterion in ARM. No strengths or weaknesses are to be added, and SSO instructions are to be followed explicitly.

o Each criterion must be scored individually from zero to the maximum allowable score for the specific criterion.

o Unless recused every Reviewer will score each application assigned to the panel.

o For assistance with scoring, please refer to Attachment C.

Face-to-Face Review

• Scoring Application:

o Chairperson asks each Reviewer to complete a hardcopy final score sheet to be submitted to the Chairperson.

o Each criterion must be scored individually from zero to the maximum allowable score for the specific criterion.

o Unless recused or absent, every Reviewer will score each application assigned to the panel.

o If recused the Reviewer will check the appropriate area on the score sheet and sign and date it.

o The final scores will be entered in ARM by the DIR Contractor.

o For assistance with scoring, please refer to Attachment C.

Both types of Review

• Budget Discussion:

o First, Second and Third Reviewers provide recommendations regarding the budget proposal.

o Considering the Reviewers’ recommendation, the panel discusses and determines a budget recommendation.

o The panel determines their final recommendation: “Recommend Budget as Requested” or “Recommend Budget as Reduced” with details concerning budget adjustments. If reduced a rationale must be provided.

• Funding Preference (if applicable):

o When appropriate, First, Second and Third Reviewers provide information on whether the applicant has requested a funding preference.

o The panel determines whether the funding preference has been requested and whether it has been met by the applicant.

o If preference was requested and not met, a rationale must be provided.

• Recommendations and Conditions: :

o A recommendation is information that Reviewers include in a summary statement to advise the program office or the applicant to consider (e.g., when carrying out the project, if approved, or in a future application for the project).

o A condition is advisory to Grants Management and may be suggested by the Reviewers as part of the summary statement for possible inclusion in the Notice of Grant Award (NGA), if an award is made. A suggested condition may address such things as a restriction on an item proposed in the application budget or the need for submission of additional information.

• Chairperson’s Certification:

o Chairperson completes the certification form in ARM for Teleconferences and hard copy for Face-to-Face review meetings.

Please note that, in general, the panel will not begin the review of an application after 4:30 p.m. It is imperative that each application receives equal consideration and that an application’s review not be rushed in the interest of time

6.3 Post-Review

6.3.1 Debriefing Session

Immediately following the completion of the last application’s review, the Chairperson will facilitate a debriefing session with all panel members. The debriefing will cover a wide variety of issues including review logistics, the review process, and evaluation criteria. Most often, Program, Grants Management, the RA and DIR Contractor staff will be on-hand to listen to feedback and provide clarification when necessary. Comments will be used to evaluate and improve the process of objective review.

6.3.2 Reviewer Evaluation

At the conclusion of the review process, all Reviewers will be asked to complete an evaluation form. Evaluations provide valuable feedback to HRSA in assessing the overall effectiveness and appropriateness of the objective review process.

6.3.3 Return or Destruction of Confidential Materials

All confidential review and review-related materials must be destroyed after the review. Once the process has concluded.. All electronic materials containing confidential information, such as computer files created by the Reviewer or e-mails, must be deleted by the Reviewer. At that time, Reviewers must certify (using the certification provided with the review materials) that they have appropriately returned or deleted all review-related documents and return this form to the DIR Contractor.

6.3.4 Reimbursement/Honorarium

To receive the honorarium payment, complete the provided honorarium forms and return them to the DIR Contractor. Travel reimbursement forms must also be completed and immediately returned to the DIR Contractor once travel has been completed. Unless otherwise specified, direct any questions regarding honorarium payment to the DIR Contractor.

ATTACHMENT A - Guidelines for Writing Strengths and Weaknesses

Reviewers should identify and clearly state strengths and weaknesses for each published review criterion. A strength is defined as a response that clearly meets or substantially exceeds requirements set forth in a review criterion. A weakness is defined as a response that falls short of meeting requirements set forth in a review criterion. Statements should be constructive and absent of bias and outside information. Please keep in mind that comments will be read by the applicant, and therefore should be appropriate and helpful to the applicant. The following are some guidelines to crafting statements of strengths and weaknesses.

• Provide a clear and succinct synopsis of the project in the “Overview” in your own words. Do not retype the introduction to the application.

• Use only the published review criteria for all assigned applications. Do not consider factors outside the scope of any review criterion.

• Make clear, definitive assertions. For example, instead of “The application appears to not have included objectives,” write “The application does not clearly define program objectives.”

• Use complete declarative sentences (present tense) with language that presents ideas fully. Incomplete sentences and fragments delay the work of producing a cogent and cohesive combined summary statement.

• Depersonalize criticisms. When describing a weakness, always refer to the application’s weaknesses, not the applicant. Use of the term “applicant organization” is also preferred over “applicant” when writing weaknesses. Avoid using phrases like “fails to, could have, appears,” and other negative statements. For instance, instead of stating “The applicant fails to demonstrate organizational expertise,” write “The application does not clearly demonstrate the organization’s expertise.” The difference is subtle but significant. Also, include an example, where possible. For instance, the statement “The application does not clearly demonstrate the organization’s expertise in the area of business management, construction management oversight and community outreach.

• Write in the third person (i.e., “The applicant organization demonstrates…”) rather than the first or second person, (i.e., “This reviewer feels…” or “In the opinion of this reviewer…”). The final summary statement will be the combined opinions of all reviewers. Use page numbers in lieu of lengthy passages of text when examples are too lengthy to include, but specific to the statement.

• When using acronyms, write out the entire name followed by the acronym in parenthesis.

• Remember that each application is to be evaluated on its own merit. The evaluation should reflect a clear, objective, explicit, and justified assessment of the application. The reviewer should write the strengths and weaknesses based on how well the applicant develops the proposed project and the extent to which the project is responsive to the published criteria.

• Avoid comparing one application’s content, process, or budget to any other assigned application. Each application should be reviewed independently, and be assessed and analyzed based on the facts presented within the application.

• Avoid scoring an application based on personal experiences, outside knowledge, and personal feelings of the application’s feasibility.

• Avoid complex formatting in the document, and use 12 point, Times New Roman Font. Reviewers are asked to use only basic formatting (i.e., bold, italic, underline, bullets, and tabs). Please refrain from using numbered bullets.

• Provide information that helps the applicant understand why a particular statement or recommendation was made.

• Review and be sure that the summary statements are completely accurate concerning the presence or absence of information in the application.

• Do not be prescriptive and avoid using phrases like “it would have been strengthened by……”

Below, please see examples of undesirable and desirable ways of stating Strengths and Weaknesses.

EXAMPLE SUMMARY STATEMENT COMMENTS

Undesirable

Strength

The budget as presented is reasonable. (There must be support language justifying this finding.)

Weakness

The applicant fails to present any statistics/numbers of its own members, of youth development and physical activity program directors with whom it communicates, or of the current reach of its projects and products. It fails to specify amounts of technical assistance, workshop presentation, meetings, conference calls or other collaborative activities

(Use “the application” or “the applicant organization” instead of “the applicant.” Avoid using words such as “fails to present,” in case the point is indeed addressed somewhere in the application. Change to “does not adequately present.”)

Desirable

Strength

The application provides a comprehensive overview of the quality management program, detailing the extent of the program and the work accomplished thus far.

The applicant organization clearly documents a process for collecting and analyzing data for program assessment measures to assess the network’s progress in relation to proposed outputs and outcomes.

Weakness

The application does not provide statistics specific to the target area population. The application does not consistently provide citations of data sources, specifically its statement that “Over 30 new aids cases are reported each month.

ATTACHMENT B – What Do I Do With the Budget?

A Guide for HRSA Reviewers

Introduction

One of the underlying assumptions of grant review is that the applicant is most knowledgeable about their proposed project. This means that, in some ways, the “burden of proof” is higher on the Reviewer to justify his or her opinions. Reviewers usually feel comfortable accepting this burden with the project narrative, but don’t always feel the same ease with the review of the application’s budget. Other Reviewers think that they have little say on the budget of a grantee, so they shouldn’t bother looking at that part at all. Neither of these should be the case.

A Reviewer is chosen as the expert on the programmatic elements of the proposal, thus Reviewers should trust their experience and judgment with regard to direct costs, labor categories, hours proposed and other programmatic details when evaluating the budget. The Reviewers should not write off their opinions. While the Reviewers’ budget recommendations are advisory, this advice is thoroughly considered and taken very seriously.

This handout contains HRSA’s expectations of Reviewers when reviewing an application’s budget, some tips on how to look at a budget, definitions for common budget terms, and answers to Reviewers’ frequently asked questions when reviewing application budgets.

HRSA’s Expectations

• Reviewers are expected to review the entire application – not just the narrative scope to the exclusion of the proposed budget.

• Reviewers are not expected to be accountants and manipulate every detail of budgets they review.

• At least some of the Reviewers on any given panel are expected to be knowledgeable about applicants’ proposed activities and the costs of conducting those activities.

• Reviewers are not expected to make judgments about whether costs are “allowable” in accordance with the Department of Health and Human Services’ policy, but instead to make judgments about whether proposed budgets are reasonable and necessary for performing the proposed projects.

• Advice regarding budgets should be the result of panel consensus.

• Any advice should be thorough and precise. Expressions of vague discomfort that stop short of a quantitative recommendation are not enough – HRSA needs to know what the Reviewers were thinking in order to correctly act on their recommendations.

Please keep these expectations in mind as you go through the budget portion of each application.

Tips on how to review & consider the budget

When an application’s budget is reviewed, Reviewers should trust their experience and opinions. Reviewers are not required to be fully knowledgeable on the budget process. As they are reviewing the budget, Reviewers should ask themselves questions such as, “Is this cost reasonable? Is this necessary or reasonable to perform the proposed project?”

Some examples of the type of questions that should be asked and may be helpful in forming an educated opinion include:

• Does the total proposed budget appear consistent with the size and scope of the project?

o Consider the bottom line versus the scope of the project – are they compatible? Can the applicant organization accomplish the goals of the proposal with the total amount of money proposed?

• Do the proposed budget and budget justification relate logically to the narrative describing the project?

o Are the budget and the budget justification describing the same project as the project narrative? Do they make sense for the proposed project?

• Are the individual line items understandable in terms of what they will pay for in the proposed project?

o Do the line items logically link to the activities in the proposed project? Are they necessary or reasonable to support the project?

• Are there extraneous items of cost that do not appear necessary or reasonable in support of the proposed project?

o Are there any line items that do not logically link to activities outlined in the project narrative? Are there line items that do not appear necessary or reasonable to accomplish the proposed project?

• Do any individual line items appear inflated (or under-funded) compared to the overall scope or individual tasks proposed?

o Are the budget line items too big or not? Too much? Too little? Reviewers should rely on their experience in determining these matters.

Common Budget Terms

These are common terms used for discussing application budgets:

Allocable cost – a cost whose relative benefits make it assignable to one or more of the cost objectives.

Allowable cost – a cost that is reasonable, allocable, within accepted standards, or otherwise conforms to generally acceptable accounting principles, specific limitations or exclusions, or agreed-to terms. HRSA’s Division of Grants Management Office (DGMO) makes these determinations.

Direct cost – any cost that can be identified specifically with a particular final cost objective.

Indirect cost – any cost that is not directly identified with a single, final cost objective, but identified with two or more final cost objectives or an intermediate cost objective.

Indirect cost rate – The percentage or dollar factor that expresses the ratio of indirect expense incurred in a given period to direct labor cost, manufacturing cost, or another appropriate base for the same period.

Unallowable cost – any cost that, under the provisions of any pertinent law, regulation, or contract, cannot be included in prices, cost-reimbursements, or settlements under a government contract to which it is allocable.

Frequently Asked Questions

Do I have to read the budget?

Yes. HRSA’s review policy is that all parts of the application are objectively reviewed, including the budget.

Do our budget recommendations really matter?

Yes. As an objective Reviewer, you are considered the expert on the programmatic elements of the budget. HRSA’s DGMO seriously considers and investigates Reviewers’ budget recommendations.

What should I be looking at when I review an application’s budget?

The most important items to look at in a budget are the direct costs. Are they necessary and reasonable to accomplish the project described in the program narrative? Most of the costs described in a budget are direct costs. Direct costs include personnel, equipment, travel, materials, consultants, and possibly many others.

What don’t I have to worry about when I look at an application’s budget?

There are a few things you don’t have to comment on when looking at an application’s budget. For example, many organizations that perform government work either through a contract or a grant have a negotiated indirect rate with the government. There is no need for a Reviewer to comment on these indirect rates which may be characterized as overhead or General and Administrative (G&A) cost elements in the budget proposal. You also don’t have to comment on whether a cost is allowable or not – just think about whether the cost is necessary and reasonable to accomplish the project.

How should I write my budget recommendations?

Be as concrete and definitive as possible. A statement that “This budget is too high for the proposed project.” is too vague. Give a number that can be applied specifically to a particular line item or cost element (For example, “Decrease the travel budget by $1,000, to cover the cost of traveling two people instead of three people to the All-Grantee Meeting.”) Ideally, this budget reduction is an actual dollar amount from a particular line item. But if you can only be specific to a percentage, note where in the budget this reduction should take place. (For example, “Data analyst is more appropriate at 50% time on this project than 100% time.)

Attachment C

HRSA Scoring Rubric

The following scale may be used by reviewers as a guideline when assigning scores to each criterion. For these purposes, an element is an item pertinent to a review criterion as defined in the Funding Opportunity Announcement (FOA).

Suggested Scoring Guidelines

|Total Point Value for a | | | | | |

|Review Criterion |Outstanding |Very Good |Satisfactory |Marginal |Poor |

|5 |5 |4 |3 |2 |1-0 |

|10 |10 |9-8 |7 |6 |5-0 |

|15 |15-14 |13-12 |11 |10-9 |8-0 |

|20 |20-19 |18-16 |15-14 |13-12 |11-0 |

|25 |25-24 |23-20 |19-18 |17-15 |14-0 |

|30 |30-29 |28-24 |23-21 |20-18 |17-0 |

|35 |35-33 |32-28 |27-25 |24-21 |20-0 |

|40 |40-38 |37-32 |31-28 |27-24 |23-0 |

|45 |45-43 |42-36 |35-32 |31-27 |26-0 |

Approx.% |95-100% |80-94% |70-79% |60-69% |59-0% | |

Outstanding All elements are clearly addressed, well-conceived, thoroughly developed, and well supported. Documentation and required information are specific and comprehensive. Weaknesses identified will likely have no effect on the applicant’s proposed project.

Very Good Elements are clearly addressed with necessary detail and adequate support. Most documentation and required information are specific and sufficient. Weaknesses identified will likely have minor effect on the applicant’s proposed project.

Satisfactory Elements are addressed, although some do not contain necessary detail and/or support. Most documentation and required information are present and acceptable. Weaknesses identified will likely have moderate effect on the applicant’s proposed project.

Marginal Some elements are not addressed, and those addressed do not contain necessary detail and/or support. Some documentation and required information are missing or deficient. Weaknesses identified will likely have significant effect on the applicant’s proposed project.

Poor Few, if any, elements are addressed. Documentation and required information are deficient or omitted. Weaknesses identified will likely have substantial effect on the applicant’s proposed project.

2 Prior to scoring applications under some programs, there will be a call for a motion to recommend for approval/not recommend for approval of an application.

-----------------------

What Constitutes a Conflict?

An individual has a conflict of interest regarding an application if that person, or a close relative or professional associate of that person, actually has or appears to have:

1. Received or could receive a direct financial benefit of any amount deriving from an application under review.

2. Received or could receive a financial benefit from the applicant institution, offeror or principal investigator that in the aggregate exceeds $10,000 per year (for reviewers who are federal employees the amount is $15,000 per year); this amount includes honoraria, fees, stock or other financial benefit, and additionally includes the current value of the reviewer's already existing stock holdings, apart from any direct financial benefit deriving from an application or proposal under review.

3. Any other interest in the application or proposal that is likely to bias the individual's evaluation of that application or proposal.

If you have any questions, please ask your RA.

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download