Module 1: Overview and Reminders
This module contains an overview and reminders for members of an ABET accreditation review team. The information in this module applies to all ABET commissions (except as noted) and addresses standard ABET review procedures and expectations. Those who recently completed Program Evaluator Candidate Training or Recertification Training, as well as those who are very experienced, will not need to go through all the material in depth, but details are provided for those who may have forgotten them and for use as a reference.
A. Team Responsibilities and Expectations
The purpose of an ABET review team is to:
- review one or more programs that an ABET accreditation commission will consider for an accreditation action.
- evaluate the extent to which programs submitted by an institution satisfy the relevant accreditation criteria, policies, and procedures ABET publishes.
The role of each review team member is to:
- interact with institutional personnel in a professional and collegial manner, showing appropriate respect for institutional personnel and students.
- be a benevolent auditor, working as a partner in a program’s continuous improvement process rather than as an investigator or detective.
- be helpful, as appropriate, in assisting programs to improve.
The selection and performance evaluation for ABET program evaluators are based on the PEV Competency Model. PEVs are expected to be:
- Technically current
- Effective communicators
- Interpersonally skilled
- Team oriented
Read the Program Evaluator Competency Model.
Code of Conduct
Team members represent ABET and an ABET commission. As such, each team member is bound by the ABET Board of Delegates Code of Conduct found in the Rules of Procedure. The code requires all persons involved with ABET, every volunteer and staff member, exhibit the highest standards of professionalism, honesty, and integrity in the pursuit of their duties. The services ABET provides require impartiality, fairness, and equity. All members of ABET evaluation teams are bound by the code and agree to abide by it as a condition of assignment to a review team.
Conflict of Interest
Team members also are bound by the ABET Conflict of Interest Policy, found in the Board of Delegates Rules of Procedure. Any real or perceived conflict of interest (COI), as defined in the policy, must be disclosed immediately. The following are specifically stated as definite conflicts in the ABET Accreditation Policy and Procedure Manual (APPM):
- Current or past employment as faculty, staff, or consultant by the institution or program
- Current or past discussion or negotiation of employment with the institution or program
- Attendance as a student at the institution
- Receipt of an honorary degree from the institution
- Involvement of a close family relative as a student or employee of the institution or program
- An unpaid official relationship with an institution, e.g., membership on the institution’s board of trustees or industry advisory board
Other associations or relationships with the program, institution, or personnel of the program or institution could represent a COI depending on the details. Any questionable circumstances should be discussed with the team chair or appropriate ABET official as soon as possible.
As representatives of ABET, each team member’s actions reflect on ABET. Team members should dress in a manner appropriate for a consultant to the institution. “Business casual” dress is usually appropriate for a review of course and assessment materials on “Day 0” of a visit. However, professional dress (i.e., jacket and tie for men) should be worn on the other two days of a visit.
During a visit, details of findings should not be discussed with faculty and students, and the team recommendations for accreditation actions should never be disclosed except in confidential ABET documents. The team’s deliberations are confidential. All information about the institution and its programs obtained by team members is confidential without time limit. Information should be discussed only with appropriate institutional representatives and within the review team. All notes and review materials should be retained until after the commission action in July, after which they should be destroyed (except possibly for public documents).
Size of Team
In most cases, an on-site review team for a general review will consist of a Team Chair and one Program Evaluator (PEV) for each program being reviewed by the team. If the team is large, then there may be co-chairs for the team.
The typical minimum team size is three for a general review, so if only one program is being reviewed and no team from a different commission is simultaneously reviewing programs at the same institution, there may be two PEVs for the single program.
In certain cases, the Commission Executive Committee may approve having a team chair and one PEV for a single program general review. The team size also may vary for interim evaluations.
Regardless of the number of PEVs, all reviews are a team effort, and the findings and accreditation recommendations from each review are team decisions.
Some teams may have one or more observers during the visit. Usually, observers are representatives of state boards, other accreditation organizations, or related organization, or they are future PEVs making an observation visit as the last part of PEV training.
Observers may participate in team discussions at the discretion of the team chair, but they do not normally participate actively in interviews with institutional personnel unless requested to do so by the Team Chair or PEV. Observers do not speak at the exit meeting, except possibly to express thanks to the institution.
E. Consistency Issues
Consistency in program reviews is a major concern and focus of ABET. It is essential that every effort be made to have consistency to be fair to all reviewed programs and to enhance the credibility of the review process.
Established interpretations of criteria and guidelines must be applied rather than the personal interpretations or beliefs of individual PEVs. Team Chairs may offer guidance on how a commission has interpreted and applied criteria consistently in the past.
Remember review findings on shortcomings and recommended actions are team decisions. A PEV whose interpretations of criteria are inconsistent with the other team members’ interpretations must either convince the team to modify its findings or conform to the majority opinion to achieve consistency.
Areas in which consistency issues can arise are:
- Depth and completeness of the review from program to program.
PEVs must be thorough in auditing the extent to which each criterion is satisfied, and the level of detailed investigation should be the same from program to program.
- Consistency across all programs in an institution.
It is essential that a team not have different findings for similar characteristics in different programs. It also is essential that teams from different commissions not have different findings for programs when program characteristics and relevant criteria are similar. PEVs must be attentive to the findings on other programs and consider whether there is consistency between findings for each of the other programs and the program that they are reviewing. The Team Chair usually coordinates consistency in the findings of different teams.
- Use of key terms.
The key terms, deficiency, weakness, concern, and observation, must be used correctly and only as defined and intended by ABET.
- Consistency on the interim recommendation.
For programs requiring an interim review, consistency on the interim recommendation (IR or IV) is essential. The two actions IR and IV do not indicate different levels of severity. The only issue is whether a report would be expected to document corrective actions sufficiently to determine whether each weakness has been resolved.
F. Accreditation Policy and Procedures
Questions sometime arise regarding the policies and procedures a PEV should check during a review. Most of the policy and procedure checks during a normal review are done by ABET staff or the team chair, and any potential issues relative to policies and procedures should be discussed with the Team Chair.
PEVs should be aware of the policy regarding identifying the accreditation status of a program. It is important that accredited programs be clearly identified and distinguished from any related non-accredited programs, and that the wording specified in the Accreditation Policy and Procedure Manual be used whenever the accredited status of a program is noted in public documents.
G. Writing Exit Statements
Careful use of language and careful construction of statements on findings are essential in writing an exit statement. It is important to remember that the exit statement, when it appears in a draft or final statement, will be read by:
- The institution.
Therefore, the exit statement must identify and describe shortcomings clearly and sufficiently enough for the institution to be able to determine appropriate corrective actions for the shortcomings. Additionally, institution personnel preparing the self-study for the next (future) general review will need to understand the basis for the shortcoming (which was identified as long as six years ago) to be able to describe relevant improvements appropriately.
- The commission.
Therefore, the exit statement must identify and describe shortcomings clearly and sufficiently enough for commissioners, who have not seen the self-study or other visit material, to make correct decisions relative to the findings.
- The next evaluation team.
Therefore, the exit statement must describe the findings accurately enough so an evaluator, who has no materials except the final statement from the previous review, can determine exactly what caused shortcomings in the previous review.
Terminology for Shortcomings
The terminology that applies to shortcomings (and only to shortcomings) is defined in the ABET Accreditation Policy and Procedure Manual. These three terms and their meanings are:
- Deficiency – A criterion, policy or procedure is not satisfied.
- Weakness –A criterion, policy, or procedure is satisfied to some meaningful extent, but the strength of compliance is not sufficient to ensure that the quality and accreditation of the program will not be compromised prior to the next general review.
- Concern – A criterion, policy, or procedure is satisfied, but an observed potential exists for non-satisfaction in the near future.
Each of these terms may apply only to a written ABET criterion, policy, or procedure. For example, it would not be appropriate in these two commissions to assign a deficiency if 10 of a program’s 11 student outcomes are assessed well but the eleventh is not assessed at all. Slightly different approaches may be taken in ANSAC, CAC, and ETAC.
For convenience the term “shortcoming” is used within ABET to mean a deficiency, weakness, or concern. However, a concern is not really a shortcoming in the literal sense because it requires no corrective action by the institution, whereas deficiencies and weaknesses do require corrective action.
Note that additional discussion of the differences, as well as example scenarios, are provided in Module 5 of the PEV Recertification Training.
Writing Shortcoming Statements
Each shortcoming listed in the exit statement must have the three components listed below.
1. Criterion citation. The applicable part of the criterion must be stated. It is preferable to do this verbatim, although only the relevant requirements should be excerpted and should not be placed within quotation marks. What is stated for this component forms the basis for comparison with the observed facts that make the reason for the shortcoming clear.
2. Observation. This describes the observed facts inconsistent with the stated criterion requirements, in the case of a deficiency or weakness, and the observation representing the potential for future non-compliance, in the case of a concern.
3. The negative effect on the program. This is a statement of the effect on the program of a deficiency, the lack of strength of compliance for a weakness, or the potential future non-compliance for a concern.
The following examples may not exactly match the criteria for every commission, since there are variations in criteria among the commissions. The examples are intended to demonstrate good and poor statements of findings. They do not necessarily illustrate possible findings for every commission.
Example of a good shortcoming statement citing a weakness or a contributing factor for a weakness.
The three component identifiers are inserted in brackets, preceding the corresponding text in the statement.
[Criterion citation] Criterion 4 requires there be a process that assesses the extent to which each student outcome is attained. [Observation] Although most of the student outcomes were assessed reliably based on actual student performance, the only mechanism used for assessment of student knowledge of contemporary issues (Student Outcome SO-10) is through administration of a survey of students, asking whether they have knowledge of contemporary issues. [The negative effect on the program] Reliance on self-evaluation does not provide sufficient evidence of actual student knowledge, to determine reliably the level of attainment of the required knowledge. Therefore, the strength of compliance with this criterion is not sufficient, to ensure continued compliance in the near future.
Remember this would be listed under the heading of “Program Weaknesses” or something similar, in which case it may not be necessary to restate specifically that the finding is a weakness. However, a specific commission may prefer the last sentence be recast something like, “Therefore, the strength of compliance with this criterion is not sufficient to ensure continued compliance in the near future, which contributes to a weakness in Criterion 4.”
Remember the language must be very clear to all three audiences for the statement of findings:
Poor or unclear statement of what was observed: There is inadequate assessment of outcome SO-10.
- How does the institution know why what they are doing is inadequate?
- How will the commission be able to decide whether the finding of inadequate assessment is appropriate?
- How will the next evaluation team know what specifically needed changing?
- General note: It often is best to avoid words such as “inadequate” and “insufficient,” except in stating a deficiency.
Good or clear statement of what was observed: The only mechanism used for assessment of student knowledge of contemporary issues (Student Outcome SO-10) is through administration of a survey of students, asking whether they have knowledge of contemporary issues.
Language used should correspond to the level of severity of the shortcoming:
- For a deficiency, the words have to clearly say the criterion is totally or largely unmet:
- “There is no evidence that. . . “
- “There is no assessment and evaluation process.”
- “Not all students are required to engage in a major design experience.”
- For a weakness, the words have to clearly say the criterion is met, but strength of compliance is lacking (and how it is lacking needs to be clear):
- “Although there is a process for the assessment and evaluation for each student outcome, assessment of outcomes 3(d), 3(f), 3(g), 3(h), and 3(i) appears to be done randomly and not according to any schedule. Systematic assessment and evaluation of these outcomes would strengthen compliance with this criterion.”
- “Although the quality of the leadership of the program appears to be adequate currently, there has been a different interim department head during each of the past six years, including the current year, and the current head does not intend to continue in an interim position next year. Stability in the department head position, by providing a good leader on a non-interim basis, would strengthen compliance with this criterion.”
- For a concern, the words have to clearly say the criterion is met and indicate exactly what leads to the concern about the potential for future noncompliance:
- “It is clear all of the student outcomes are being assessed, but evaluation of some of the assessment data appears to be inconsistent, possibly because the rubrics are not applied consistently by different instructors. Unless evaluation of the extent to which outcomes are attained is carried out on a consistent basis, future compliance with this criterion may be jeopardized.”
- “Although all of the transcripts examined provided evidence students meet all graduation requirements, the process by which graduation requirements are audited prior to graduation appears to be ad hoc. This leads to concern that future compliance with this criterion may be jeopardized.”
Some commissions provide a Style Guide that includes lists of commonly used phrases for each shortcoming level.
Not all commissions use strength statements. When a commission does not use strength statements in the draft statement, strengths may be stated as observations. The discussion of strength statements here applies both to specific strength statements and to observations of strengths.
As is the case with shortcoming statements, statements of strength each have three components:
This describes the observed facts that represent a program strength.
2. What makes it stand out above the norm.
This describes why the observed facts are unusual and represent a strength above what is normally expected.
3. What positive effect it has on the program.
Example of a good strength statement:
Undergraduate research is a very important component of the program. This year, about 20% of the students are involved in undergraduate research, and during the student interview, about 80% of the students stated their intention of going to graduate school after they graduate from the program. The opportunity for an undergraduate research experience significantly enhances the overall quality of the program.
Example of a poor strength statement:
The faculty members are very dedicated and hard working.
• At least two components are missing, and the other (what was observed) is stated without basis.
• This is not a program strength as stated. It is generally expected faculty members will be dedicated and hard working in every program.
• If the faculty members really are unusually dedicated to providing an outstanding educational experience, something like the following might be appropriate:
The dedication of the faculty to the quality of the students’ education is exceptionally high. Efforts to use a variety of pedagogies for different learning styles and the high level of faculty interaction with students result in a high level of motivation and effort among the students. The results contribute substantially to the overall quality and effectiveness of the program.
• Do not relate to findings relative to criteria.
• Are provided in the interest of general program improvement (not for corrective actions to address potential shortcomings).
• Must not appear prescriptive.
• Have no consequence relative to accreditation, if ignored by the institution.
Example of a good observation statement:
Offerings of upper-level elective courses have been limited, and there appears to be significant interest among students in electives that have not been offered recently. A richer selection of elective alternatives would enhance the students’ education.
An on-site review team (and each team member) is “The Face of ABET.”
The four big Cs go a long way in promoting a productive review:
I. Additional Information
I. Additional Information
A more complete document on visit procedures and the role and responsibilities of a PEV can be found at the general PEV Recertification Training website.
For visit related issues and questions, you may contact one of the following (in preferred order):
- Your Team Chair
- Your Commission Chair
- The ABET Director of Accreditation Operations, Jane Emmet (firstname.lastname@example.org)
- Your commission adjunct (ABET staff):
For training or IT-related questions, you may contact the Director of Accreditation Operations, Jane Emmet (email@example.com) and/or ABET Adjunct Training Director, Mike Leonard (firstname.lastname@example.org).