Module 3: The Role of the Program Evaluator
As a Program Evaluator, you have many roles. Most important, you are the face of ABET and must follow ABET’s Code of Conduct and adhere to ABET’s Conflict of Interest Policy. As a Program Evaluator, you are devoting your time and expertise to ensuring quality education for students studying for entry into your profession.
A. Program Evaluator Roles
ABET Representative: As a program evaluator, you are the face of ABET. You represent ABET when conducting all accreditation-related activities, and the institution that you visit judges ABET by your actions.
Evaluator: You have been selected as a program evaluator because you have subject matter expertise in the discipline you will be evaluating. Every program evaluator on your team is assigned a program to evaluate, based on the expertise he/she has in that program area. Accepting a position on a visit team is a commitment to fulfill all the obligations required. The role of the program evaluator is not to be the police, to look for “gotchas,” but rather to be a partner with the institution in the improvement of the program.
Team Member: As a program evaluator, you also play the important role of team member. Your interaction with your colleagues, particularly during team meetings, is crucial to an efficient and successful visit. Being team-oriented is an important competency for all program evaluators.
A program evaluator’s responsibilities are to:
- Complete the pre-visit work, including inspecting the materials provided by the institution.
- Participate in a 2-4 day on-site review.
- Evaluate information obtained according to ABET Criteria.
- Prepare materials in a timely manner, including recommended accreditation action.
- Provide visit materials to the team chair at the conclusion of the visit.
- Complete performance appraisal forms. (This will be covered in Module 6.)
- Submit expenses to ABET using the on-line expense system within 10 days of your site visit.
- Assist the team chair, as requested, in review of the institution’s 30-day due process response and development of the Final Draft Statement.
Important Note: Avoid any real or perceived conflicts of interest with the institutions being evaluated. For more information, see ABET’s Code of Conduct, Conflict of Interest, and Confidentiality Policies in the ABET Board of Delegates Rules of Procedure.
B. Activities Before The Visit
You will have a significant number of tasks to complete prior to arriving on campus for the site visit. Your thorough preparation before arriving on site is essential to an efficient and successful site visit. In particular, your pre-visit analysis will enable you to identify additional information you will need during the visit. Additional information requests must be given to the program well in advance of the visit and should be coordinated with the team chair. Because this is a collaborative process, you need to meet your obligations in a timely manner to allow the institution to gather additional information in a reasonable time. If the program is already ABET-accredited, the program evaluator will receive a copy of the previous final accreditation statement for the program.
Here are some of the activities and deliverables you will need to complete prior to arriving on campus:
Make Travel Arrangements
You are responsible for making your own travel arrangements in consultation with the team chair and in alignment with the ABET Travel Policy. Travel arrangements should be made as soon as the team chair communicates visit details, including overall schedule, hotel accommodations, and transportation arrangements to and from the airport. All reasonable attempts should be made to minimize travel costs. All travel must follow the ABET Travel Policy to be reimbursed. A copy of the ABET Travel Policy and Procedures Manual can be found here.
Review the Self-Study Report
This is the primary document the institution prepares to demonstrate compliance with ABET Criteria. Each program uses the Self-Study Questionnaire template appropriate to the commission. These are found on the ABET website. While the majority of the program’s Self-Study Report will be in paragraph format, a program may use tables, graphs, charts, and figures to illustrate compliance. The Self-Study Questionnaire includes tables that a program is required to complete along with prescribed formats for certain types of information such as course descriptions and faculty curriculum vita. Here are samples of what those may look like:
- Course Description Sample 1
- Course Description Sample 2
- Course Description Sample 3
- Faculty CV Sample 1
- Faculty CV Sample 2
You must thoroughly review the Self-Study Report prior to arriving on site, and prior to pre-visit conferences if scheduled by the team chair.
As part of your review of the Self-Study Report, you will need to analyze transcripts. ABET recognizes transcripts as the official record of student coursework. While summary sheets or transcript checklists provided by the institution may be helpful for transcript analysis, the transcript itself should be the key document analyzed. Procedures for doing this include:
- Be sure that the transcripts identify the name (title) of the program in a way that clearly identifies the program as an ABET-accredited program according to the institution catalog and other documents and in a way that distinguishes it from any non-accredited programs with which it could be confused by a potential employer. Identify any problems in this regard to your team chair.
- Make sure that the courses that are counted toward the degree are consistent with the published requirements of the program. In cases where the transcript is for a graduate of an earlier curriculum, the institution must provide a copy of the appropriate curriculum. The institution also should provide justification for any variances, such as transfer credits or substitutions that are not clearly documented on the transcripts.
- Check to be sure that prerequisites are taken before each course that requires them and that the course sequence on the transcript does not vary unreasonably from the recommended sequence. If courses are taken out of sequence, check to see if there is an indication of difficulty for the students in terms of the course grades. (If there are difficulties for students, then there could be a problem with the mechanisms for advising and the enforcement of prerequisites. If there are no problems, it could indicate prerequisite requirements that are not needed.)
- Ensure that the number of transfer credits and the number of course substitutions are reasonable.
- Request clarification for any apparent problems in the transcripts. Do your transcript analysis and request clarifications soon enough to allow reasonable time for the institution to respond.
Draft Plan for Visit
Your draft plan includes the questions and areas that require additional information based on your review of the Self-Study Report. The draft plan helps you manage your time efficiently when you are on site. Be sure to:
- Convey your preferences and arrangements for interviews and meetings with faculty, students, and others to your team chair and then contact the program head if directed to do so by the team chair.
- Tell your team chair about visits needed to supporting areas. (Team chairs usually coordinate visits to supporting areas.)
In Module 5 you will be provided with a variety of forms to help you walk through the self-study report and conduct a preliminary review of a program’s compliance with the ABET criteria.
Applying the Criteria
- Review the current ABET Criteria, the Self-Study Report, and any other materials sent by the institution.
- Assess areas of apparent strengths and shortcomings.
- Make preliminary assessment of what ABET Criteria appear to be met or may not be met.
- Follow up with the team chair regarding additional information you will need when you arrive on site. Your team chair may direct you to communicate this information to the program representative.
- Always respond promptly to requests from your team chair.
- Coordinate your travel plans with your team chair.
C. The Visit
The site visit enables ABET to assess factors that cannot be adequately addressed in the Self-Study Report. These factors include the intellectual atmosphere, the morale of the faculty and students, the caliber of the staff and student body, the facilities, and the outcome of the education offered as evidenced by the character of the work performed.
Do not conduct interviews or share preliminary conclusions with the institution prior to the site visit.
Your objectives while on site are to:
- Make a qualitative assessment of factors that cannot be documented in the Self-Study Report.
- Conduct a detailed examination of the materials compiled by the institution (this includes course materials and assessment materials).
- Provide the program with a preliminary assessment of its strong points and shortcomings.
- Review facilities.
Important Note: Visiting teams should help the programs assess their strong points as well as their weak points. You are not there only as an auditor but also as a consultant to assist the program in its continuous improvement efforts.
D. Activities That You Will Conduct During The Visit
Examine Assessment Materials
Assessment Materials are usually located with the course display materials. The purpose is to corroborate the assessment plan for student outcomes that was provided in the Self-Study Report, including ensuring that appropriate documentation exists for the various phases of the assessment process.
Examine Course Materials
Evaluators will review samples of displayed course materials including course syllabi, textbooks, example assignments and exams, and examples of student work, typically ranging from excellent through poor (Accreditation Policy and Procedure Manual, Section I.E.5.b.(2)).
The display materials should be readily available throughout the visit. You should verify that:
- The course is up-to-date and appropriate for the objectives of the program.
- The course prerequisites are adequate.
- The learning activities are appropriate for the course outcomes.
- The graded work adequately assesses the course learning outcomes.
Note that it is unlikely that a problem with a single course would be enough evidence for a Weakness or a Deficiency. Rather, the course materials should be used to corroborate potential problems indicated elsewhere, such as suspected failure by some faculty members to remain current, or should suggest potential problems that need to be corroborated by additional evidence, such as an apparent lack of rigor sufficient to achieve the course learning outcomes.
Interview the Program Head
This is an opportunity to develop a better understanding of the program’s strengths and weaknesses and the vision of its leadership for the future. Any issues that arose during your pre-visit preparation should be discussed to ensure that you are aware of all appropriate facts and have planned appropriate activities for further investigation. If a single program is being evaluated, the team chair may decide to interview the program head.
It is preferable not to make requests that require substantial effort to satisfy during this meeting (or at any time during the visit). Such requests should be made well in advance if your visit preparation is adequate. However, do not hesitate to make requests for things that are really needed to ensure a fair and complete review. It is advisable to discuss such requests with your team chair if they are likely to require a substantial amount of work from the program’s staff.
You should also interview others with administrative duties (class scheduling, student advising, transfer course evaluation, etc.) relative to the program. This does not include administrators at the institutional level, whose visits are coordinated by the Team Chair. If the program head is not the department chair or the administrative head over the program, then you should also interview the administrative head. If more than one program under the administrative head is being reviewed, you may wish to coordinate your interview with the program evaluator(s) for the other program(s), perhaps even doing the interview simultaneously with the other PEV(s).
Interview Faculty Members
During the faculty interviews, you should try to understand each faculty member’s teaching philosophy; activities undertaken to maintain currency; level of understanding of, and participation in, the assessment process and preparation for the visit; and his/her view of the strengths and weaknesses of the program and its future direction.
Faculty members also can be a source of initial identification or corroboration of problems related to the quality and maintenance of facilities (labs, classrooms, library, etc.). Faculty attitudes regarding the work environment and their commitment to remaining at the institution are also important in assessing the adequacy of institutional support to maintain program stability during the potential period of accreditation.
There are many personal preferences regarding the best way to interview faculty members. Ideally, it is best to interview faculty members in their offices (if private) on a one-on-one basis. However, if the faculty is large or offices are not private, an alternative approach may be needed. Individual appointments should last for approximately 30 minutes. You should identify some of the faculty you are especially interested in interviewing if you will not have time to interview all of them. When selecting faculty to interview, consider those teaching courses for which you have questions, those with laboratory responsibilities, length of service, and rank. This should all be arranged before the visit and in consultation with your team chair.
Interviewing the students is often one of the most rewarding and enjoyable parts of the visit. The level of satisfaction with, and enthusiasm for, the program can be a good indicator of the program quality. In addition, students can provide important corroboration for such aspects as the lack of adequate quality in certain courses, problems with advising, and the adequacy of facilities (labs, classrooms, library, etc.).
As we know, however, the student perspective on the importance of certain courses and the characteristics of the program is often quite different from the perspectives of the faculty and administration. Although these perspectives can be valuable in getting a feel about the overall quality of a program, it is important to keep in mind that students are not always in a good position to judge the real value of an educational experience. Therefore, student complaints should not result in a conclusion that there is a problem unless the problem can be corroborated by evidence that is more concrete.
There are many individual preferences for selecting the students to be interviewed as a group. Ideally, the group will include a representative number of upper-level students and some other students as well. The student interviews can be done during a scheduled class (the instructor should not be present), but it is difficult to get a good representative sample from a single class. On the other hand, if the student interview is held during an open meeting with voluntary attendance, it can be difficult to get good representative attendance as well (or even good attendance at all). In any case, the group to be interviewed should not be selected by the program staff unless you ensure that those selected are a representative sample of the student body, especially the upper-level students. You should allow at least 30 minutes for meeting with the students.
Tour the Facilities
In touring the facilities, you want to ensure that they are sufficiently up-to-date and maintained to support the needs of the program. You also want to be aware of safety and access issues. The adequacy of the support staff is important, as is a mechanism for ensuring the continuing adequacy through the period of accreditation. The following are typical support facility visits that are conducted on an accreditation review visit. These visits may be done by the team chair or may be assigned to PEVs by the team chair.
If more than one program is being visited, these activities may be coordinated among the program evaluators and findings shared.
- Visit the library. When you visit the library, you are primarily verifying the evidence in the Self-Study Report: adequacy of the collection, availability of staff support, electronic access to information and materials, etc. The adequacy of the resources is important to observe, and this is difficult to determine from the Self-Study Report.
- Visit one or more supporting departments. Supporting departments (mathematics, physics, English, etc.) play an important role in the foundational preparation of students. In interviewing the department chair or other departmental representative, you should try to determine how well the students of the programs that the team is reviewing do in the supporting courses and whether there are any problems with the nature of the courses from the perspective of the supporting department. If course materials have been provided in a course materials display area, you should review these materials before going to visit the supporting department. If the materials have not been arranged in advance, then you cannot expect to see much more than course textbooks and syllabi for courses that are currently being offered.
- Visit the career support center, placement office, cooperative education office, etc. The intent of visits to offices such as these is to assess the level of support for students. You should inquire about the support that is available and general procedures for students to obtain the support. Sometimes you can assist the program, students, or faculty by identifying available support that is not being used effectively in the program. Understaffed offices, inadequate facilities, or outdated programs can indicate or corroborate problems that appear in assessment results, student or faculty interviews, etc.
Write an Exit Statement
The Exit Statement has a prescribed format for each commission. The Exit Statement must clearly identify each shortcoming relative to the ABET Criteria or Accreditation Policies and Procedures.
Complete a Program Audit Form (PAF)
The PAF lists the shortcomings for a program. It normally is completed after the Exit Statement is finished, and should use the same language as the Exit Statement in describing shortcomings. Usually the text in the PAF is copied from the Exit Statement and pasted into the PAF. The PAFs are given to the dean by the team chair at the conclusion of the Exit Meeting, and they serve as a written summary of the team’s findings for each of the programs that were evaluated for accreditation. Note that the PAF does not identify the recommended action on a program.
Debrief the Program Head
The debrief for the program head by the PEV is less formal than the exit meeting. There should be sufficient discussion to ensure that the program head understands the shortcomings that have been identified. Information that could be helpful in addressing the shortcomings can be provided by the PEV, but care must be taken to avoid any impression of prescribing a solution.
Participate in the Exit Meeting
The purpose of the exit meeting is to communicate the team’s findings relative to all programs. Following initial statements by the team chair, each PEV will read (verbatim) the Exit Statement for the program evaluated by that PEV. The exit meeting is a formal report by the team. The team chair serves as chair for the exit meeting, and only questions for clarification are permitted. If you are asked by your team chair to respond to a question, you should be very careful not to ramble, but focus on the observed facts that lead to the finding.
The team chair may modify this basic procedure as appropriate. For example, if the number of programs reviewed is large, general information about each program and the program strengths might be omitted from what is read at the exit meeting. Similarly, if there is a shortcoming that is common to all programs, then the team chair may state that shortcoming in the introductory remarks and each PEV would then omit that shortcoming from the reading of the PEV’s statement. Finally, if there is more than one PEV for a program, the team chair will determine what is to be read by each PEV.
E. A Typical Visit Agenda
During the initial team meeting, your responsibilities are to:
- Review visit plan.
- Provide initial Program Evaluator Visit Report and Program Evaluator Worksheet to the team chair. Note: Your team chair may request these prior to the visit.
- Discuss pre-visit assessment.
- Review criteria, policy, and procedure changes.
You will also visit the program to:
- Evaluate course materials (syllabi, texts, graded student work, etc.).
- Evaluate assessment materials (results and analysis from surveys, etc.).
- Conduct other visits (labs, etc.) as needed.
During the team meeting, your responsibilities are to:
- Identify all potential Deficiencies, Weaknesses, and Concerns.
- Work as a team to come to consensus and to ensure consistency about findings among programs.
- Team meets with president and/or dean and designated guests.
- Meet with program head.
- Meet with program faculty, students, and support staff.
- Attend optional luncheon for team with institutional officials and guests.
- Continue meetings and interviews.
- Visit facilities.
- Visit supporting departments.
During the team meeting, your responsibilities are to:
- Report on findings relative to previously identified potential Deficiencies, Weaknesses, and Concerns.
- Discuss possible accreditation action.
- Prepare draft of Program Audit Form (PAF) and exit statement.
- Provide draft of Program Audit Form (PAF) and exit statement to team chair.
- Complete interviews and facilities visits not completed on Monday.
- Brief program head on findings (informal debrief).
- Team meeting (working lunch).
Complete Program Audit Form (PAF) and exit statement:
- Address each criterion for which there are findings.
- Document facts, Deficiencies, Weaknesses, Concerns, and Observations.
- Your statement must correlate with PEV Report and Program Evaluator Worksheet.
Provide team chair with PEV Visit Report, Program Evaluator Worksheet, and Program Audit Form.
- Update to reflect visit findings.
- List persons interviewed.
- Identify recommended accreditation action.
- Add program exit statement.
Participate in exit meeting. Read your written exit statement, do not deviate, and do not ramble! Please Note: During exit meeting, it is very important not to mention a proposed recommended accreditation action!
As a Program Evaluator, your responsibilities include:
- Identifying potential shortcomings by criterion.
- Determining appropriate findings.
- Selecting the key term that applies overall for each finding.
- Providing evidence for each finding and the effect it has on the program in the exit statement and Program Audit Form (PAF).
- Discussing or recommending an accreditation action to the team only.
As a Program Evaluator, you should never:
- Recommend solutions to shortcomings.
- Be prescriptive in suggesting improvements.
- Share the recommended accreditation action with anyone outside the team.
Draft Statement Outline
The Team Chair will combine the exit statements of all programs evaluated into a draft statement. The draft statement will follow this outline:
- For each program
- Program Strengths
- Program Deficiencies
- Program Weaknesses
- Program Concerns
- Program Observations
After going through an editing process, the Draft Statement will be sent to the institution. The institution then has 30 days to respond. This response is called the Due Process Response.
Please Remember To:
- Explain all shortcomings.
- Be consistent with ABET Criteria.
- Provide evidence for each finding and the impact it has on the program as it relates to the ABET Criteria.
- Be consistent with all forms provided to the team chair.
F. Applying The Criteria: Consistency Counts
Accreditation actions must be consistent across all programs with similar shortcomings (Concern, Weakness, Deficiency) and across all institutions. As a PEV, it is essential that you compare your findings with those of the other programs being evaluated at the institution and ensure that different findings do not result from similar observations. The visit team should work together to resolve potential differences in findings for situations that are the same or very similar. Consistency is checked throughout the ABET Accreditation Process. The process is visually illustrated in this graphic.
G. After The Visit
Within ten (10) business days after the visit, you will need to:
- Submit your expenses using the ABET on-line expense system, including scanning or copying all receipts. The expense report will be sent to your team chair for approval by the system. Once approved by the team chair, the system will forward your expense report to ABET headquarters for payment. Information to submit expense reports is available.
- Complete the online appraisal form for the Team Chair and Program Evaluators on your visiting team.
ABET Expense Reimbursement Policy: All travelers should be aware that only persons who are authorized prior to visiting campuses or attending pertinent meetings and activities will be reimbursed for their allowable and reasonable expenses. If there is a question on reimbursements or allowable expenses, please seek guidance from your team chair or ABET liaison. It is ABET’s policy to reimburse reasonable expenses incurred by those individuals who are authorized to attend specific visits. All expenses should be submitted within ten (10) working days after returning from your trip. Failure to comply with the travel policy instructions will delay the reimbursement of your expenses. Expenses submitted more than sixty (60) days after the end of travel may not be reimbursed.