ABET

Pre-Visit Preparation Module 1: Overview and Reminders

Pre-visit Preparation

Module 1: Overview and Reminders

This module contains an overview and reminders for members of an ABET accreditation review team. The information in this module applies to all ABET commissions (except as noted) and addresses standard ABET review procedures and expectations. Those who recently completed Program Evaluator Candidate Training or Recertification Training, as well as those who are very experienced, will not need to go through all the material in depth, but details are provided for those who may have forgotten them and for use as a reference.

A. Team Responsibilities and Expectations

The purpose of an ABET review team is to:

  • review one or more programs that an ABET accreditation commission will consider for an accreditation action.
  • evaluate the extent to which programs submitted by an institution satisfy the relevant accreditation criteria, policies, and procedures ABET publishes.

The role of each review team member is to:

  • interact with institutional personnel in a professional and collegial manner, showing appropriate respect for institutional personnel and students.
  • be a benevolent auditor, working as a partner in a program’s continuous improvement process rather than as an investigator or detective.
  • be helpful, as appropriate, in assisting programs to improve.

The selection and performance evaluation for ABET program evaluators are based on the PEV Competency Model. PEVs are expected to be:

  • Technically current
  • Effective communicators
  • Interpersonally skilled
  • Team oriented
  • Professional
  • Organized

Read the Program Evaluator Competency Model.

Code of Conduct

Team members represent ABET and an ABET commission. As such, each team member is bound by the ABET Code of Conduct found in the Rules of Procedure. The code requires  all persons involved with ABET, every volunteer and staff member, exhibit the highest standards of professionalism, honesty, and integrity in the pursuit of their duties.  The services ABET provides require impartiality, fairness, and equity. All members of ABET evaluation teams are bound by the code and agree to abide by it as a condition of assignment to a review team.

Conflict of Interest

Team members also are bound by the ABET Conflict of Interest Policy, found in the Rules of Procedure. Any real or perceived conflict of interest (COI), as defined in the policy, must be disclosed immediately. The following are specifically stated as definite conflicts in the ABET Accreditation Policy and Procedure Manual (APPM):

  • Current or past employment as faculty, staff, or consultant by the institution or program
  • Current or past discussion or negotiation of employment with the institution or program
  • Attendance as a student at the institution
  • Receipt of an honorary degree from the institution
  • Involvement of a close family relative as a student or employee of the institution or program
  • An unpaid official relationship with an institution, e.g., membership on the institution’s board of trustees or industry advisory board

Other associations or relationships with the program, institution, or personnel of the program or institution could represent a COI depending on the details. Any questionable circumstances should be discussed with the team chair or appropriate ABET official as soon as possible.

Professional Dress

As representatives of ABET, each team member’s actions reflect on ABET. Team members should dress in a manner appropriate for a consultant to the institution. “Business casual” dress is usually appropriate for a review of course and assessment materials on “Day 0” of a visit. However, professional dress (i.e., jacket and tie for men) should be worn on the other two days of a visit.

Confidentiality

During a visit, details of findings should not be discussed with faculty and students, and the team recommendations for accreditation actions should never be disclosed except in confidential ABET documents. The team’s deliberations are confidential. All information about the institution and its programs obtained by team members is confidential without time limit. Information should be discussed only with appropriate institutional representatives and within the review team. All notes and review materials should be retained until after the commission action in July, after which they should be destroyed (except possibly for public documents).

Size of Team

In most cases, an on-site review team for a general review will consist of a Team Chair and one Program Evaluator (PEV) for each program being reviewed by the team. If the team is large, then there may be co-chairs for the team.

The typical minimum team size is three for a general review, so if only one program is being reviewed and no team from a different commission is simultaneously reviewing programs at the same institution, there may be two PEVs for the single program.

In certain cases, the Commission Executive Committee may approve having a team chair and one PEV for a single program general review. The team size also may vary for interim evaluations.

Regardless of the number of PEVs, all reviews are a team effort, and the findings and accreditation recommendations from each review are team decisions.

Observers

Some teams may have one or more observers during the visit. Usually, observers are representatives of state boards, other accreditation organizations, or related organization, or they are future PEVs making an observation visit as the last part of PEV training.

Observers may participate in team discussions at the discretion of the team chair, but they do not normally participate actively in interviews with institutional personnel unless requested to do so by the Team Chair or PEV. Observers do not speak at the exit meeting, except possibly to express thanks to the institution.

B. Review of Pre-Visit Procedures

Travel Arrangements

Review team members should make travel arrangements as soon as possible after receiving final confirmation of visit dates and times for the first team meeting and for the exit meeting. Reservations on commercial carriers should be made through the ABET travel agency, American Express Travel. (Travel booked through the ABET travel agent provides travel insurance.) The ABET travel policy changes every year and should be reviewed to ensure compliance. Log in to MyABET to access the current Travel Policy and Procedures Manual.

Important Travel Note

Due to concerns regarding potential emergencies which could occur during an ABET team’s on-site review, ABET does not support spouses, family members, or significant others accompanying team members on visits. Please bring any issues with this directive to the attention of your team chair. If the Team Chair cannot resolve the issue, the Director for Accreditation Operations must be made aware of the problem immediately so timely decisions/actions can be made or taken as necessary.

Forms and Criteria to Use for the Current Cycle

The forms and criteria used during the visit often change from year to year. It is important you download the documents for the current accreditation cycle from the ABET website. DO NOT USE FORMS OR CRITERIA FROM PREVIOUS CYCLES UNLESS YOU ARE DOING AN INTERIM REVIEW.

Guidelines for selecting the current forms and criteria:

  • Use the applicable set of criteria. The criteria document must have the applicable cycle printed on the cover page. For example, general reviews conducted during the fall of 2016 are in the 2016 – 17 cycle
  • For a general review, use the criteria in the current (at the time of the visit) cycle.
  • For an interim review, use the criteria in effect at the time of the last general review (unless the institution has opted to use the current criteria).
  • The correct forms to use should have the current cycle printed in the heading of the form. (Some variation may occur on an interim visit.) In some cases, there are different forms for general reviews and interim reviews.
  • The Team Chair should provide criteria for interim reviews.

If you have not already done so, you should download the current documents now and use these for your review. You will find all current accreditation documents here.

Review of Materials

In addition to making the needed transportation and hotel reservations, the pre-visit activities for each team member consist of reviewing the program’s self-study report and other materials and making plans for activities during the visit itself. Usually, the Team Chair will arrange one or more conference calls with the team prior to the visit to discuss various aspects of the visit and potential shortcomings in the programs that will be reviewed by the team.

 

The review of the materials available before the visit results in indications by each PEV of potential program shortcomings on the PEV worksheet. The PEV worksheet may be a separate document or may be part of the PEV visit report, depending on the commission.

 

Additionally, each PEV reviews the transcripts assigned by the Team Chair and records the transcript reviews on the PEV report or other form provided for that purpose. The worksheet will be submitted to the Team Chair prior to the visit, and the Team Chair may request the transcript reviews prior to the visit as well.

 

The primary purpose of the transcript analysis is for assessing Criterion 1 which requires there be procedures to ensure that graduates meet all program requirements, and students have access to advice regarding curriculum matters. The transcript analysis should compare each transcript to the program’s stated curricular requirements. Issues such as inappropriate transfer credit, inappropriate course substitutions, and prerequisites not completed as required are the most frequent indicators in transcripts of potential problems relative to Criterion 1.

 

The program’s curricular requirements as listed in the self-study and other relevant materials must themselves be analyzed for compliance with the requirements of Criterion 5 (plus any program criteria).

 

During the pre-visit review, it is especially important to identify additional information needed to complete the review. This additional information should be requested to be sent before the visit, or to be provided during the visit, as appropriate.

 

For the EAC, each PEV will arrange for the visit schedule and communicate additional information directly to the program head. However, the Team Chair should receive a copy of all such requests, and the Team Chair should be consulted for advice in any case where the PEV is uncertain of a request’s reasonableness.

 

For the ASAC, the CAC and the ETAC, the Team Chair makes all schedule arrangements and conducts all communication with the institution, so all PEV requests for additional information or scheduling are directed to the Team Chair in these three commissions.

Pre-Visit Deliverables and Document Summary

  • PEV completes the initial evaluation worksheet. (The worksheet is a separate document for CAC and EAC and part of the visit report for ASAC and ETAC.) The first column of the worksheet is completed based on a careful review of the self-study report and other materials available before the visit.
  • Team Chair uses the initial worksheet results to guide pre-visit conferences.
  • PEV completes the curriculum and transcript analysis parts of the visit report.
  • PEV submits the initial worksheet, and possibly the partially completed visit report, to the Team Chair prior to the visit.

C. Review of Visit Procedures

Typical Schedule for a General Review

Day 0 (usually a Sunday)

  • Team reviews course/assessment materials.
  • Team visits labs and other facilities.

Day 1 (usually a Monday)

  • Team meets with the dean.
  • PEVs meet with department/program heads.
  • Team Chair meets with the dean and institution officials.
  • PEVs interview faculty and students.
  • PEVs update worksheets to reflect the evaluation status of the program at the end of Day 1.
  • PEVs draft exit statements and PEV reports.

Day 2 (usually a Tuesday)

  • Team agrees on recommended findings and action for each program.
  • PEVs conduct informal debrief with department/program head.
  • Team Chair conducts informal debrief with dean.
  • PEVs give electronic copies of exit statement and PEV report, and evaluation worksheet if requested, plus a hard copy of the Program Audit Form (PAF) for the institution, to the team chair. An electronic copy of the PAF may also be required.
  • Exit meeting is held; PEVs read exit statements verbatim.
  • Team leaves campus immediately after the exit meeting.

Reminders

  • Maintain an open line of communication with the program head or other primary administrative contact.
  • Identify all shortcomings, especially potential deficiencies, as soon as possible.
  • Treat all institutional and team personnel with professional courtesy and respect.
  • Do not compare the program or institution to others, especially your own.
  • Discuss all issues with the program head (or other administrative contact) at the debriefing.

Additionally:

  • Department/program administrators should not be surprised by the findings during the debriefing.
  • Do not discuss the recommended accreditation action with anyone except team members.

D. Review of Post-Visit Procedures

Post Visit Deliverables and Document

  • PEV submits expense report (if reimbursement is requested) using the online system. Instructions for using the online system are available here(login required).
  • PEV completes online evaluations of Team Chair and other PEVs, using the online evaluations forms on the ABET website.
  • PEV submits documents relative to the visit to his/her society, if required. If you do not know the requirements of your society, you should check with your society’s PEV coordinator.
  • Team Chair uses exit statements to prepare and submit a Draft Statement of Findings for each program reviewed.
  • After editing, the Draft Statement is sent to the institution.

Deliverables and Documents Produced After Receipt of the Due Process Response

  • Team Chair consults with PEVs as needed to modify the findings based on the documented corrective actions taken after the visit. Notes and materials from the visit should be retained until after the commission’s action on the program in case re-evaluation of findings is needed in light of new information.
  • Team Chair prepares Final Statement that reflects actions reported in the due process response and the resulting status for each shortcoming listed in the Draft Statement.
  • The Final Statement is edited. (Additional updates submitted by the institution may be incorporated into the statement if received in time.)
  • The commission reads the Final Statement and discusses the findings as needed, then decides the accreditation action for each program.
  • ABET send the institution the Final Statement and a letter stating the accreditation action for each program.

E. Consistency Issues

Consistency in program reviews is a major concern and focus of ABET. It is essential that every effort be made to have consistency to be fair to all reviewed programs and to enhance the credibility of the review process.

 

Established interpretations of criteria and guidelines must be applied rather than the personal interpretations or beliefs of individual PEVs. Team Chairs may offer guidance on how a commission has interpreted and applied criteria consistently in the past.

 

Remember review findings on shortcomings and recommended actions are team decisions. A PEV whose interpretations of criteria are inconsistent with the other team members’ interpretations must either convince the team to modify its findings or conform to the majority opinion to achieve consistency.

 

Areas in which consistency issues can arise are:

  • Depth and completeness of the review from program to program.
    PEVs must be thorough in auditing the extent to which each criterion is satisfied, and the level of detailed investigation should be the same from program to program.
  • Consistency across all programs in an institution.
    It is essential that a team not have different findings for similar characteristics in different programs. It also is essential that teams from different commissions not have different findings for programs when program characteristics and relevant criteria are similar. PEVs must be attentive to the findings on other programs and consider whether there is consistency between findings for each of the other programs and the program that they are reviewing. The Team Chair usually coordinates consistency in the findings of different teams.
  • Use of key terms.
    The key terms, deficiency, weakness, concern, and observation, must be used correctly and only as defined and intended by ABET.
  • Consistency on the interim recommendation.
    For programs requiring an interim review, consistency on the interim recommendation (IR or IV) is essential. The two actions IR and IV do not indicate different levels of severity. The only issue is whether a report would be expected to document corrective actions sufficiently to determine whether each weakness has been resolved.

F. Accreditation Policy and Procedures

Questions sometime arise regarding the policies and procedures a PEV should check during a review. Most of the policy and procedure checks during a normal review are done by ABET staff or the team chair, and any potential issues relative to policies and procedures should be discussed with the Team Chair.

PEVs should be aware of the policy regarding identifying the accreditation status of a program. It is important accredited programs be clearly identified and distinguished from any related non-accredited programs and the wording specified in the Accreditation Policy and Procedure Manual be used whenever the accredited status of a program is noted in public documents.

 

G. Writing Exit Statements

Careful use of language and careful construction of statements on findings are essential in writing an exit statement. It is important to remember that the exit statement, when it appears in a draft or final statement, will be read by:

  • The institution.
    Therefore, the exit statement must identify and describe shortcomings clearly and sufficiently enough for the institution to be able to determine appropriate corrective actions for the shortcomings. Additionally, institution personnel preparing the self-study for the next (future) general review will need to understand the basis for the shortcoming (which was identified as long as six years ago) to be able to describe relevant improvements appropriately.
  • The commission.
    Therefore, the exit statement must identify and describe shortcomings clearly and sufficiently enough for commissioners, who have not seen the self-study or other visit material, to make correct decisions relative to the findings.
  • The next evaluation team.
    Therefore, the exit statement must describe the findings accurately enough so an evaluator, who has no materials except the final statement from the previous review, can determine exactly what caused shortcomings in the previous review.

Terminology for Shortcomings

The terminology that applies to shortcomings (and only to shortcomings) is defined in the ABET Accreditation Policy and Procedure Manual. These three terms and their meanings are:

  • Deficiency – A criterion, policy or procedure is not satisfied.
  • Weakness –A criterion, policy, or procedure is satisfied to some meaningful extent, but the strength of compliance is not sufficient to ensure that the quality and accreditation of the program will not be compromised prior to the next general review.
  • Concern – A criterion, policy, or procedure is satisfied, but an observed potential exists for non-satisfaction in the near future.

Each of these terms may apply only to a written ABET criterion, policy, or procedure. In the CAC and EAC, each criterion, policy, or procedure is considered as a whole.

For example, it would not be appropriate in these two commissions to assign a deficiency if 10 of a program’s student outcomes are assessed well but one is not assessed at all. Slightly different approaches may be taken in ASAC and ETAC.

For convenience the term “shortcoming” is used within ABET to mean a deficiency, weakness, or concern. However, a concern is not really a shortcoming in the literal sense because it requires no corrective action by the institution, whereas deficiencies and weaknesses do require corrective action.

Note that additional discussion of the differences, as well as example scenarios, are provided in Module 5 of the PEV Recertification Training.

Writing Shortcoming Statements
Each shortcoming listed in the exit statement must have the three components listed below. In cases where multiple factors apply to a single criterion, each factor contributing to the shortcoming must have the three components.

1. Criterion citation. The applicable part of the criterion must be stated. It is preferable to do this verbatim, although only the relevant requirements should be excerpted. What is stated for this component forms the basis for comparison with the observed facts that make the reason for the shortcoming clear.

2. Observation.This describes the observed facts inconsistent with the stated criterion requirements, in the case of a deficiency or weakness, and the observation representing the potential for future non-compliance, in the case of a concern.

3. The negative effect on the program. This is a statement of the effect on the program of a deficiency, the lack of strength of compliance for a weakness, or the potential future non-compliance for a concern.
The following examples may not exactly match the criteria for every commission, since there are variations in criteria among the commissions. The examples are intended to demonstrate good and poor statements of findings. They do not necessarily illustrate possible findings for every commission.
Example of a good shortcoming statement citing a weakness or a contributing factor for a weakness.

The three component identifiers are inserted in brackets, preceding the corresponding text in the statement.

[Criterion citation] Criterion 4 requires there be a process that assesses the extent to which each student outcome is attained. [Observation] Although most of the student outcomes were assessed reliably based on actual student performance, the only mechanism used for assessment of student knowledge of contemporary issues (Student Outcome SO-10) is through administration of a survey of students, asking whether they have knowledge of contemporary issues. [The negative effect on the program] Reliance on self-evaluation does not provide sufficient evidence of actual student knowledge, to determine reliably the level of attainment of the required knowledge. Therefore, the strength of compliance with this criterion is not sufficient, to ensure continued compliance in the near future.

Remember this would be listed under the heading of “Program Weaknesses” or something similar, in which case it may not be necessary to restate specifically that the finding is a weakness. However, a specific commission may prefer the last sentence be recast something like, “Therefore, the strength of compliance with this criterion is not sufficient to ensure continued compliance in the near future, which contributes to a weakness in Criterion 4.”

Remember the language must be very clear to all three audiences for the statement of findings:

Poor or unclear statement of what was observed: There is inadequate assessment of outcome SO-10.

  • How does the institution know why what they are doing is inadequate?
  • How will the commission be able to decide whether the finding of inadequate assessment is appropriate?
  • How will the next evaluation team know what specifically needed changing?
  • General note: It often is best to avoid words such as “inadequate” and “insufficient,” except in stating a deficiency.

Good or clear statement of what was observed: The only mechanism used for assessment of student knowledge of contemporary issues (Student Outcome SO-10) is through administration of a survey of students, asking whether they have knowledge of contemporary issues.
Language used should correspond to the level of severity of the shortcoming:

  • For a deficiency, the words have to clearly say the criterion is totally or largely unmet:
    • “There is no evidence that. . . “
    • “There is no assessment and evaluation process.”
    • “Not all students are required to engage in a major design experience.”
  • For a weakness, the words have to clearly say the criterion is met, but strength of compliance is lacking (and how it is lacking needs to be clear):
    • “Although there is a process for the assessment and evaluation for each student outcome, assessment of outcomes 3(d), 3(f), 3(g), 3(h), and 3(i) appears to be done randomly and not according to any schedule. Systematic assessment and evaluation of these outcomes would strengthen compliance with this criterion.”
    • “Although the quality of the leadership of the program appears to be adequate currently, there has been a different interim department head during each of the past six years, including the current year, and the current head does not intend to continue in an interim position next year. Stability in the department head position, by providing a good leader on a non-interim basis, would strengthen compliance with this criterion.”
  • For a concern, the words have to clearly say the criterion is met and indicate exactly what leads to the concern about the potential for future noncompliance:
    • “It is clear all of the student outcomes are being assessed, but evaluation of some of the assessment data appears to be inconsistent, possibly because the rubrics are not applied consistently by different instructors. Unless evaluation of the extent to which outcomes are attained is carried out on a consistent basis, future compliance with this criterion may be jeopardized.”
    • “Although all of the transcripts examined provided evidence students meet all graduation requirements, the process by which graduation requirements are audited prior to graduation appears to be ad hoc. This leads to concern that future compliance with this criterion may be jeopardized.”

Strength Statements
Not all commissions use strength statements. When a commission does not use strength statements in the draft statement, strengths may be stated as observations. The discussion of strength statements here applies both to specific strength statements and to observations of strengths.

As is the case with shortcoming statements, statements of strength each have three components:

1. Observation.
This describes the observed facts that represent a program strength.
2. What makes it stand out above the norm.
This describes why the observed facts are unusual and represent a strength above what is normally expected.
3. What positive effect it has on the program.
Example of a good strength statement:

Undergraduate research is a very important component of the program. This year, about 20% of the students are involved in undergraduate research, and during the student interview, about 80% of the students stated their intention of going to graduate school after they graduate from the program. The opportunity for an undergraduate research experience significantly enhances the overall quality of the program.

Example of a poor strength statement:

The faculty members are very dedicated and hard working.
• At least two components are missing, and the other (what was observed) is stated without basis.
• This is not a program strength as stated. It is generally expected faculty members will be dedicated and hard working in every program.
• If the faculty members really are unusually dedicated to providing an outstanding educational experience, something like the following might be appropriate:

The dedication of the faculty to the quality of the students’ education is exceptionally high. Efforts to use a variety of pedagogies for different learning styles and the high level of faculty interaction with students result in a high level of motivation and effort among the students. The results contribute substantially to the overall quality and effectiveness of the program.

Observation Statements
• Do not relate to findings relative to criteria.
• Are provided in the interest of general program improvement (not for corrective actions to address potential shortcomings).
• Must not appear prescriptive.
• Have no consequence relative to accreditation, if ignored by the institution.

Example of a good observation statement:
Offerings of upper-level elective courses have been limited, and there appears to be significant interest among students in electives that have not been offered recently. A richer selection of elective alternatives would enhance the students’ education.

H. Conclusion

H. Conclusion

Remember:

An on-site review team (and each team member) is “The Face of ABET.”

The four big Cs go a long way in promoting a productive review:

  • Courtesy
  • Consistency
  • Clarity
  • Confidentiality

I. Additional Information

I. Additional Information

A more complete document on visit procedures and the role and responsibilities of a PEV can be found at the general PEV Recertification Training website.

For visit related issues and questions, you may contact one of the following (in preferred order):

For training or IT-related questions, you may contact the Director of Accreditation Operations, Jane Emmet (jemmet@abet.org) and/or ABET Adjunct Training Director, Mike Leonard (mleonard@abet.org).