Module 3: The Role of the Program Evaluator
As a Program Evaluator, you have many roles. Most important, you are the face of ABET and must follow ABET’s Code of Conduct (Rules of Procedure, Section 4) and adhere to ABET’s Conflict of Interest Policy. As a Program Evaluator, you are devoting your time and expertise to ensuring quality education for students studying for entry into your profession.
A. Program Evaluator Roles
ABET Representative: As a Program Evaluator, you are the face of ABET. You represent ABET when conducting all accreditation-related activities, and the institution you visit judges ABET by your actions.
Evaluator: You have been selected as a Program Evaluator because you have subject matter expertise in the discipline you will be evaluating. Program Evaluators on your team bring an expertise in a program area and are assigned a program to evaluate based on their program area. Accepting a position on a visit team is a commitment to fulfill all the obligations required. The role of the Program Evaluator is not to be the police, or to look for “gotchas,” but rather to be a partner with the institution for the improvement of the program.
Team Member: As a Program Evaluator, you also play the important role of team member. Your interaction with your colleagues, particularly during team meetings, is crucial to an efficient and successful visit. Being team-oriented is an important competency for all Program Evaluators.
Your responsibilities, as a Program Evaluator, are to:
- Complete the pre-visit work, including inspecting the materials provided by the institution.
- Participate in a 2 to 4-day on-site review.
- Evaluate information obtained according to ABET Criteria.
- Identify potential shortcomings by criterion.
- Determine appropriate finding.
- Select the key term that applies overall for each finding.
- Provide evidence for each finding and the effect of the finding on the program, in the Exit Statement and the Program Audit Form (PAF).
- Discuss or recommend an accreditation action with the team only.
- Prepare materials in a timely manner, including recommended accreditation action.
- Provide visit materials to the Team Chair at the conclusion of the visit.
- Send copies of visit documents to your ABET member society as requested by the member society.
- Complete performance appraisal forms. (This will be covered in Module 6.)
- Submit expenses to ABET using the on-line expense system (Concur) within 10 days of your site visit.
- Assist the Team Chair, as requested, in review of the institution’s 30-day due process response and development of the Final Draft Statement.
B. Activities Before the Visit
You will have a significant number of tasks to complete prior to arriving on campus for the site visit. Your thorough preparation before arriving on site is essential to an efficient and successful site visit. In particular, your pre-visit analysis will enable you to identify additional information you will need during the visit. Additional information requests must be given to the program well in advance of the visit and should be coordinated with the Team Chair.
Because this is a collaborative process, you need to meet your obligations in a timely manner to provide the institution with a reasonable amount of time to gather additional information. If the program is already ABET-accredited, the you will receive a copy of the previous final accreditation statement for the program.
Here are some of the activities and deliverables you will need to complete prior to arriving on campus:
Make Travel Arrangements
You are responsible for making your own travel arrangements in consultation with the Team Chair and in accordance with the ABET Travel Policy. Travel arrangements should be made as soon as the Team Chair communicates visit details, including overall schedule, hotel accommodations, and transportation arrangements to and from the airport. All reasonable attempts should be made to minimize travel costs. All travel must follow the ABET Travel Policy to be reimbursed. The ABET Travel Policy may be found at the Program Evaluator Secure Website.
Review the Self-Study
This is the primary document the institution prepares to demonstrate compliance with ABET Criteria. Each program uses the Self-Study Questionnaire template appropriate to the commission. These are found on the ABET website. While the majority of the program’s Self-Study Report will be in paragraph format, a program may use tables, graphs, charts, and figures to illustrate compliance. The Self-Study Questionnaire includes tables a program is required to complete along with prescribed formats for information like course descriptions and faculty curriculum vita. Here are samples of what those may look like:
- Course Description Sample 1
- Course Description Sample 2
- Course Description Sample 3
- Faculty CV Sample 1
- Faculty CV Sample 2
You must thoroughly review the Self-Study prior to arriving on site and prior to pre-visit conference calls if scheduled by the Team Chair.
As part of your review of the Self-Study Report, you will need to analyze transcripts. ABET recognizes transcripts as the official record of student coursework. While summary sheets or transcript checklists provided by the institution may be helpful for transcript analysis, the transcript itself should be the key document analyzed. Procedures for doing this include:
- Be sure the transcripts identify the name (title) of the program in a way that clearly identifies the program as an ABET-accredited program according to the institution catalog and other documents, and in a way that distinguishes it from any non-accredited programs with which it could be confused by a potential employer. Identify any problems in this regard to your Team Chair.
- Make sure the courses counted toward the degree are consistent with the published requirements of the program. In cases where the transcript is for a graduate of an earlier curriculum, the institution must provide a copy of the appropriate curriculum. The institution also should provide justification for any variances, such as transfer credits or substitutions not clearly documented on the transcripts.
- Check to be sure prerequisites are taken before each course requiring them and the course sequence on the transcript does not vary unreasonably from the recommended sequence. If courses are taken out of sequence, check to see if there is an indication of difficulty for the students in terms of the course grades. (If there are difficulties for students, then there could be a problem with the mechanisms for advising and the enforcement of prerequisites. If there are no problems, it could indicate prerequisite requirements not needed.)
- Ensure the number of transfer credits and the number of course substitutions are reasonable.
- Request clarification for any apparent problems in the transcripts. Do your transcript analysis and request clarifications soon enough to allow reasonable time for the institution to respond.
Draft Plan for Visit
Your draft plan includes the questions and areas requiring additional information based on your review of the Self-Study. The draft plan helps you manage your time efficiently when you are on site. Be sure to:
- Convey your preferences and arrangements for interviews and meetings with faculty, students, and others to your Team Chair prior to contacting the program head.
- Tell your Team Chair about visits needed to supporting areas. (Team Chairs usually coordinate visits to supporting areas.)
Complete Pre-Visit Forms
You will be provided with a variety of forms to help you walk through the Self-Study report and conduct a preliminary review of a program’s compliance with the ABET criteria. Forms you will use in this training to help with the review of the Self-Study include the following:
Applied and Natural Science Evaluation Forms
- Program Evaluator Visit Report (R351) (contains Program Evaluator Worksheet)
Computing Evaluation Forms
Engineering Evaluation Forms
Engineering Technology Evaluation Forms
- Program Evaluator Visit Report (T351) (contains Program Evaluator Worksheet)
You will be required to complete as much of these forms as possible based upon the Self-Study, transcripts and additional information provided by the program prior to your arrival on site.
- Review the current ABET Criteria, the Self-Study, and any other materials sent by the institution.
- Assess areas of apparent strengths and shortcomings.
- Make preliminary assessment of what ABET Criteria appear to be met or may not be met.
- Follow up with the Team Chair regarding additional information you will need when you arrive on site. Your Team Chair may direct you to communicate this information to the program representative.
- Always respond promptly to requests from your Team Chair.
- Coordinate your travel plans with your Team Chair.
C. During the Visit
The site visit enables ABET to assess factors that cannot be adequately addressed in the Self-Study. These factors include the intellectual atmosphere, the morale of the faculty and students, the caliber of the staff and student body, the facilities, and the outcome of the education offered as evidenced by the character of the work performed.
Do not conduct interviews or share preliminary conclusions with the institution prior to the site visit.
Your objectives while on site are to:
- Make a qualitative assessment of factors that cannot be documented in the Self-Study.
- Conduct a detailed examination of the materials compiled by the institution (including course materials and assessment materials).
- Provide the program with a preliminary assessment of its strong points and shortcomings.
- Review facilities.
Important Note: Visiting teams should help the programs assess their strong points as well as their weak points. You are not there only as an auditor but also as a consultant to assist the program in its continuous improvement efforts.
D. Activities You Will Conduct During the Visit
Examine Assessment and Evaluation Materials
Assessment and Evaluation Materials are usually located with the course display materials. The purpose is to corroborate the assessment plan for student outcomes provided in the Self-Study, including ensuring appropriate documentation exists for the various phases of the assessment process.
Examine Course Materials
Evaluators will review samples of displayed course materials including course syllabi, textbooks, example assignments and exams, and examples of student work, typically ranging from excellent through poor (Accreditation Policy and Procedure Manual, Section I.E.8.a.).
The display materials should be readily available throughout the visit. You should verify:
- The course is up-to-date and appropriate for the objectives of the program.
- The course prerequisites are adequate.
- The learning activities are appropriate for the course outcomes.
- The graded work adequately assesses the course learning outcomes.
It is unlikely a problem with a single course would be enough evidence for a Weakness or a Deficiency. Rather, the course materials should be used to corroborate potential problems indicated elsewhere, such as suspected failure by some faculty members to remain current. Alternatively, the course material could suggest potential problems that need to be corroborated by additional evidence, such as an apparent lack of rigor sufficient to achieve the course learning outcomes.
Interview the Program Head
This is an opportunity to get a better feel for the program’s strengths, weaknesses, and the vision of its leadership for the future. Discuss any issues that arose during your pre-visit preparation to ensure you are aware of all appropriate facts and have planned appropriate activities for further investigation. If a single program is being evaluated, the Team Chair may decide to interview the program head.
It is preferable not to make requests that require substantial effort to satisfy on the part of the program, during this meeting (or at any time during the visit). Such requests should be made well in advance if your visit preparation is adequate. However, do not hesitate to make requests for things really needed to ensure a fair and complete review. It is advisable to discuss such requests with your Team Chair if they are likely to require a substantial amount of work from the program’s staff.
You should also interview others with administrative duties (e.g. class scheduling, student advising, transfer course evaluation, etc.) relative to the program. This does not include administrators at the institutional level, whose visits are coordinated by the Team Chair. If the program head is not the department chair or other administrative head over the program, then you also should interview the administrative head. If more than one program under the administrative head is being reviewed, you may wish to coordinate your interview with the Program Evaluator(s) for the other program(s), perhaps even doing the interview simultaneously with the other PEV(s).
Interview Faculty Members
During the faculty interviews, you should try to get a feel for the philosophy of each faculty member towards teaching; activities undertaken to maintain currency; level of understanding of, and participation in, the assessment process and preparation for the visit; and the view of the strengths and weaknesses of the program and its future direction.
Faculty members also can be a source of initial identification or corroboration for problems with such aspects as the quality and maintenance of facilities (labs, classrooms, library, etc.). Faculty attitudes regarding the work environment and their commitment to remaining at the institution are also important in assessing the adequacy of institutional support to maintain program stability during the potential period of accreditation.
There are many personal preferences regarding the best way to interview faculty members. Ideally, it is best to interview faculty members in their offices (if private) on a one-on-one basis. However, if the faculty is large or offices are not private, an alternative approach may be needed. Individual appointments should last for at least 30 minutes. You should identify some of the faculty you are especially interested in interviewing if you will not have time to interview all of them. When selecting faculty to interview, consider those teaching courses for which you have questions, those with laboratory responsibilities, length of service, and rank. This should all be arranged before the visit and in consultation with your Team Chair.
Interviewing the students is often one of the most rewarding and enjoyable parts of the visit. The level of satisfaction with, and enthusiasm for, the program can be a good indicator of the program quality. In addition, students can provide important corroboration for such aspects as the lack of adequate quality in certain courses, problems with advising, and the adequacy of facilities (labs, classrooms, library, etc.).
As we know, however, the student perspective on the importance of certain courses and the characteristics of the program is often quite different from the perspectives of the faculty and administration. Although these perspectives can be valuable in getting a feel about the overall quality of a program, it is important to keep in mind that students are not always in a good position to judge the real value of an educational experience. Therefore, student complaints should not result in a conclusion that there is a problem unless the problem can be corroborated by other evidence.
There are many individual preferences for selecting the students to be interviewed as a group. Ideally, the group will include a representative number of upper-level students and some other students as well. The student interviews can be done during a scheduled class (the instructor should not be present), but it is difficult to get a good representative sample from a single class. On the other hand, if the student interview is held during an open meeting with voluntary attendance, it can be difficult to get good representative attendance (or even good attendance at all). In any case, the group to be interviewed should not be selected by the program staff unless you ensure those selected are a representative sample of the student body, especially the upper-level students. You should allow at least 30 minutes for meeting with the students.
Tour the Facilities
In touring the facilities, you want to ensure they are sufficiently up-to-date and maintained to support the needs of the program. You also want to be aware of safety and access issues. The adequacy of the support staff is important, as is a mechanism for ensuring the continuing adequacy through the period of accreditation. The following are representative of typical support facility visits that are conducted on an accreditation review visit. The Team Chair may conduct these visits or may assign them to PEVs.
If more than one program is being visited, these activities may be coordinated among the Program Evaluators and findings shared.
- Visit the library. When you visit the library, you are primarily verifying the evidence in the Self-Study: adequacy of the collection, availability of staff support, electronic access to information and materials, etc. The adequacy of the resources is important to observe, and this is difficult to determine from the Self-Study.
- Visit one or more supporting departments. Supporting departments (e.g. mathematics, physics, English, etc.) play an important role in the foundational preparation of students. In interviewing the department chair or other departmental representative, you should try to get a feel for how well the students of the programs the team is reviewing do in the supporting courses and whether there are any problems with the nature of the courses from the perspective of the supporting department. If course materials have been provided in a course materials display area, you should review these materials before going to visit the supporting department. If the materials have not been arranged for in advance, then you cannot expect to see much more than course textbooks and syllabi for courses currently being offered.
- Visit the career support center, placement office, cooperative education office, etc. The intent of visits to offices such as these is to assess the level of support for students. You should inquire about the support available and general procedures for students to obtain the support. Sometimes you can assist the program, students, or faculty by identifying available support not being used effectively in the program. Understaffed offices, inadequate facilities, or outdated programs can indicate or corroborate problems appearing in assessment results, student or faculty interviews, etc.
Write an Exit Statement
The Exit Statement has a prescribed format for each commission. The Exit Statement must clearly identify each shortcoming relative to the ABET Criteria or Accreditation Policies and Procedures. NOTE: ETAC does not require the PEV to write an Exit Statement.
Complete a Program Audit Form (PAF)
The PAF lists the shortcomings for a program. It normally is completed after the Exit Statement is finished, and should use the same language as the Exit Statement in describing shortcomings. Usually the text in the PAF is copied from the Exit Statement and pasted into the PAF. The Team Chair gives the PAFs to the dean at the conclusion of the Exit Meeting, and they serve as a written summary of the team’s findings on the programs evaluated for accreditation. Note, the PAF does not identify the recommended action on a program.
Debrief the Program Head
The PEV’s debrief for the program head is less formal than the Exit Meeting. There should be sufficient discussion, to ensure the program head understands the shortcomings been identified. Information helpful in addressing the shortcomings can be provided by the PEV, but care must be taken to avoid any impression of prescribing a solution.
Participate in the Exit Meeting
The purpose of the Exit Meeting is to communicate the team’s findings relative to all programs. Following initial statements by the Team Chair, each PEV will read (verbatim) the Exit Statement for the program evaluated by that PEV. The Exit Meeting is a formal report by the team. The Team Chair conducts the meeting, and only questions for clarification are permitted. If your Team Chair asks you to respond to a question, you should be very careful not to ramble, but focus on the observed facts that lead to the finding.
The Team Chair may modify this basic procedure as appropriate. For example, if the number of programs reviewed is large, general information about each program and the program strengths might be omitted from what is read at the Exit Meeting. Similarly, if there is a shortcoming common to all programs, then the Team Chair may state that shortcoming in the introductory remarks and each PEV would then omit that shortcoming from the reading of the PEV’s statement. Finally, if there is more than one PEV for a program, the Team Chair will determine what is to be read by each PEV.
E. A Typical Visit Agenda
During the initial team meeting, your responsibilities are to:
- Review visit plan.
- Provide initial Program Evaluator Visit Report and Program Evaluator Worksheet to the Team Chair. Note: Your Team Chair may request these prior to the visit.
- Discuss pre-visit assessment.
- Review criteria, policy, and procedure changes.
You will also visit the program to:
- Evaluate course materials (syllabi, texts, graded student work, etc.)
- Evaluate assessment materials (results and analysis from surveys, etc.)
- Conduct other visits (labs, etc.), as needed.
During the team meeting, your responsibilities are to:
- Identify all potential Deficiencies, Weaknesses, and Concerns.
- Work as a team to come to consensus and to ensure consistency about findings among programs.
- Team meets with president and/or dean and designated guests.
- Meet with program head.
- Meet with program faculty, students, and support staff.
- Attend optional luncheon for team with institutional officials and guests.
- Continue meetings and interviews.
- Visit facilities.
- Visit supporting departments.
During the team meeting, your responsibilities are to:
- Report on findings relative to previously identified potential Deficiencies, Weaknesses, and Concerns.
- Discuss possible accreditation action.
- Prepare draft of Program Audit Form (PAF) and Exit Statement.
- Provide draft of Program Audit Form (PAF) and Exit Statement to the Team Chair.
- Complete interviews and facilities visits not completed on Monday.
- Brief program head on findings (informal debrief).
- Team meeting (working lunch)
Complete Program Audit Form (PAF) and Exit Statement:
- Address each criterion for which there are findings.
- Document facts, Deficiencies, Weaknesses, Concerns, and Observations.
- Your statement must correlate with PEV Report and Program Evaluator Worksheet.
Provide the Team Chair with PEV Visit Report, Program Evaluator Worksheet, and Program Audit Form.
- Update to reflect visit findings.
- List persons interviewed.
- Identify recommended accreditation action.
- Add program Exit Statement.
Please Remember To:
- Explain all shortcomings.
- Be consistent with ABET Criteria.
- Provide evidence for each finding and the impact it has on the program as it relates to the ABET Criteria.
- Be consistent with all forms provided to the Team Chair.
Participate in Exit Meeting:
- Read your written Exit Statement, do not deviate, and do not ramble.
- Please Note: During the Exit Meeting, it is very important not to mention a proposed recommended accreditation action!
F. After the Visit
Draft Statement Outline
The Team Chair will combine the Exit Statements of all programs evaluated into a draft statement. The draft statement will follow this outline:
- For each program:
- Program Strengths
- Program Deficiencies
- Program Weaknesses
- Program Concerns
- Program Observations
After going through an editing process, the Draft Statement will be sent to the institution. The institution then has 30 days to respond. This response is called the Due Process Response.
Within 10 business days after the visit, you will need to:
- Submit your expenses using the ABET on-line expense system, including scanning or copying all receipts. Once approved by ABET Headquarters, the system will forward your expense report for payment. Please see the Instructions for the Online Expense Report.
- Complete the online appraisal form for the Team Chair and Program Evaluators on your visiting team.
ABET Travel and Expense Policy: All travelers should be aware that only persons who are authorized prior to visiting campuses or attending pertinent meetings and activities will be reimbursed for their allowable and reasonable expenses. If there is a question on reimbursements or allowable expenses, please seek guidance from your Team Chair or staff liaison. It is ABET’s policy to reimburse reasonable expenses incurred by those individuals who are authorized to attend specific visits. You should submit all expenses within 10 working days after returning from your trip. Failure to comply with the travel policy instructions will delay the reimbursement of your expenses. Expenses submitted more than 60 days after the end of travel may not be reimbursed.