B. Criteria Review and Changes
A complete summary of changes in criteria and in policies and procedures since 2011 – 2012 can be found in Keep up with Accreditation Changes. Those who have not recently participated in a program review should review these changes.
Important: Be sure to download and use the forms in the current PEV Workbook for your visit this fall! Changes in the workbook are sometimes to the Workbooks. The Workbook file will have a date as part of the file name. Before your visit, check the date for the online version of the Workbook to see if any changes have been made since you downloaded your copy.
Please review the definitions in the first section of the 2016-2017 Criteria for Accrediting Engineering Technology Programs. As a PEV, it is imperative that you understand and can explain these definitions to program personnel if you find shortcomings related to those related criteria.
Please have a copy of the current Criteria available as you review the Self-Study in preparation for your visit. It can also be useful to have a copy of the Criteria available during the visit so that you can quote directly from the Criteria when responding to questions about accreditation.
As a PEV, you will be asked by your Team Chair to provide the Introduction section of the draft statement before the visit based on the information in the self-study. The Introduction section that you will be expected to write consists of a one-paragraph description of the program, including the target employers of the graduates and any unique aspects of the program content or delivery, and the program’s educational objectives.
ETAC will now include Strength findings in the Program Audit Form, now the T301! A program Strength is an exceptionally strong and effective practice or condition that stands above the norm and that has a positive effect on the program. Program strengths must have a positive effect on the program that is seldom seen. When writing a program strength statement, state: a) what was observed, b) what makes it stand above the norm, and c) the positive effect it has on the program.
T351 PEV Report: The Comment column in this form must be filled out to briefly explain the final quality rating agreed upon by the team.
The following advice is based on issues found in recent draft statements, and addresses criteria or policies where inconsistencies or misinterpretations have most often occurred.
Multiple Issues with a Criterion: The ETAC criteria document contains eight General Criteria: Criterion 1 through Criterion 8, and 24 discipline-specific or program criteria. Where possible, try to combine all issues for a given Criterion into a single finding. (e.g., avoiding multiple findings based on different sections of the same criterion). The “degree of compliance” (expressed as a Deficiency, Weakness, or Concern) should be based on how well the overall Criterion (not just a small piece of it) has been satisfied.
Criterion 1 – Students:
There continues to be findings related to students taking courses without stated prerequisites. This shortcoming is typically identified by the PEV’s transcript analysis. Thus, it is critical that the PEV is provided transcripts well ahead of the visit so a thorough review can be accomplished. Ideally, the program is asked to explain or provide documentation for each transcript shortcoming before the visit. This may lead to discovery of a shortcoming related to missing or incomplete documentation of how or why the student was allowed to take a course without its prerequisites. [Note that a lack of prerequisites or other issues related to the curricular structure would be addressed in Criterion 5.]
Criterion 2 – Program Educational Objectives:
The primary findings stemming from this criterion comes from programs not having a documented and systematically utilized process, involving stated program constituencies (as per the program’s self-study or other venues), for the periodic review of the program educational objectives. In many cases, programs list constituencies but do not have documentation of those constituencies being involved in the periodic review of the objectives. Note that the specific wording or nature of the educational objectives should not be a focus of the PEV. If the program educational objectives have been created via a documented process by stated program constituencies, a presumption of appropriateness of the educational objective is recommended. For instance, if program educational objectives seem to be very similar to student outcomes, ETAC’s position has been to write an Observation recommending that the program educational objectives be re-written to better align with ABET’s definition.
However, if a good process has not been followed/documented and the nature of the educational objective’s wording does not match the criterion’s definition, e.g., it reads like an outcome, the finding should address both issues.
Documentation means that written evidence, meeting minutes, etc., is available that shows the involvement of listed constituencies in the review. Systematic and periodic means such a review has occurred on a scheduled basis. Lacking any of these elements, a finding should be written.
Criterion 3 – Student Outcomes:
First, does the program have documented (which means published on their website) student outcomes that address or satisfy the required student outcomes? Do those program outcomes address or satisfy all the elements of the required Criterion 3 outcomes/learned capabilities? Second, is there a documented and effective process for the periodic review of these outcomes? For instance, if the stated student outcomes don’t address all required outcome elements in this criterion, the shortcoming is twofold – they don’t address the required outcomes and they don’t have an effective review process. Note that ETAC does not require that a program use the literal wording of the Criterion 3 student outcomes/learned capabilities.
However, regardless of how the program expresses its student outcomes, the program must demonstrate that its student outcomes address all elements of the Criterion 3 student outcomes. Such demonstration may be done via a matrix or other illustrative device that shows the correlation between the program’s student outcomes and the learned capabilities listed in this criterion.
Criterion 4 – Continuous Improvement:
This criterion is often a source of findings. While the statement of the criterion is concise, it has complexity that deserves careful thought and attention both before and during a visit.
The points below should help you navigate these issues.
- The intent of this criterion is to ensure the program (1) has processes in place to assess and evaluate how well it is achieving its student outcomes, and (2) has processes in place to use the results of that evaluation to improve the program. The process and the results of these processes must be both appropriate and documented.
- While there is no explicit mention of the manner in which assessment must be carried out, the definition of assessment (in the preface of the criteria document) indicates that “effective assessment uses relevant direct, indirect, quantitative and qualitative measures as appropriate to the outcome being measured.” So, if a program only uses a few surveys, one examination, or one class to accomplish all its assessment activities, it would be reasonable and supportable to write a finding focused on the lack of appropriate and effective assessment.
- If the program has been making changes for improvement but these changes are not related to student outcome assessment and evaluation process results, there is a Criterion 4 shortcoming.
- Criterion 4 does not require that a program achieve a particular level of attainment for its student outcomes. It is the program’s responsibility to define acceptable performance (although a ridiculously low level of “acceptable” performance could lead to a finding using the criterion’s requirement that the process be appropriate). The program (not the PEV, not ABET criteria) defines “satisfactory” level of performance for each of [a-i] or [a-k]. “Satisfactory” should be based on what is needed to satisfy the needs of organizations being served by graduates.
- Student outcomes and Criterion 3 characteristics refer to capabilities of students as a group, so assessment of attainment should be reported as such; the program does not have to show that every individual graduate has achieved an acceptable level of performance in student outcomes or in Criterion 3 learned capabilities (characteristics). However, the number of students involved in the assessments should be reported to allow judgment of the appropriateness of the process.
- A PEV should not try to judge whether student performance is satisfactory in meeting student outcomes or Criterion 3 student outcomes/learned capabilities. The PEV’s responsibility is to determine how well the program itself has demonstrated that student performance is satisfactory (e.g., is the process documented and effective?) according to the program’s own benchmarks. A PEV should never attempt to use displays of student work or displays of raw assessment data to determine whether student outcomes have been met. It is the program’s responsibility to develop assessment data (using primary evidence as much as possible) and to then evaluate those data to draw its own conclusions about student achievement. It is the PEV’s responsibility to determine whether the program’s process for demonstrating achievement is credible and reliable, meeting the criterion’s requirements that the process is appropriate and documented.
- A Criterion 4 finding should not say students have—or have not—attained an outcome; rather, a Criterion 4 finding should say the program has—or has not—demonstrated that graduates have attained the respective student outcomes via an appropriate and documented process.
- An appropriate assessment process should involve the use of “primary evidence” of student attainment of outcomes. Primary evidence is associated with direct measures of student performance and should be found in the display materials of student work. As examples, survey data are indirect and secondary evidence while assessment via rubrics or other data collection mechanisms based on student project work, exams, homework or laboratory work would be primary or direct evidence.
- A PEV should allocate sufficient time on Day “0”, typically Sunday, for a thorough review of display materials towards determining whether a program’s process for demonstrating student achievement of outcomes is credible and reliable. The PEV may have the program representative lead them through the display materials with the specific guidance of “show me how the program assesses and evaluates student attainment of outcomes.” It is recommended that at least two hours of focused time be allocated to this process. It is important to gain a thorough understanding of the program’s processes and documentation on Day 0 as this provides the PEV with follow-up questions for program faculty on Day “1” and deliberations with the team and the team chair.
- Any required Program Criteria outcomes must be included in the continuous improvement plan. The student attainment of such outcomes should be assessed as part of the continuous improvement plan and any shortcomings cited in a Criterion 4 finding. These Program Criteria requirements may be stated as additional student outcomes or may be incorporated as components of student outcomes that are already mapped to Criterion 3 learned capabilities.
Criterion 5 – Curriculum:
The primary areas of findings from this criterion come from either the mathematics portion or the advisory committee portion of the criterion. Baccalaureate degree programs are not specifically required to include courses in calculus or other mathematics above algebra and trigonometry but the program must include applications of integral and differential calculus in the solution of technical problems. PEVs should look for or have the program faculty provide evidence of the student’s use of the appropriate mathematics. The advisory committee language within the criterion deserves attention as it is comprehensive as to the role of the committee and its interaction with the program, more so than other ABET commissions. Look for clear documentation of an advisory board’s fulfillment of these tasks on a periodic basis—which means a reasonable and repeated basis. If the evidence provided is for only one review right before the ABET visit and nothing else, a shortcoming exists
Criterion 6 – Faculty:
This criterion has some requirements for individual faculty members and some for the faculty as a whole, so be careful in distinguishing between the two in a finding that may be written against the criterion. Common areas for recent findings are adequate resources for, or evidence of, continuing professional development and sufficient numbers of faculty. As always, make sure to provide clear evidence related to impact on the program in such findings.
Criterion 7 – Facilities:
This criterion no longer refers to “equipment characteristic of that being used in the industry and practice being served by the program.” However, it does include language that “modern tools, equipment, computing resources, and laboratories appropriate to the program” be available. Although explicit comment regarding safety has been removed from the criteria, the language in this criterion of “students must be provided appropriate guidance regarding the use of the tools, equipment, computing resources, and laboratories” could provide the basis for a finding if the safety issue is related to instruction within the laboratory. In more general cases, safety-related shortcomings should be cited under APPM I.E.5. Also, as noted above, the criterion now mentions that the library services and computing/information infrastructure must be adequate to support scholarly and professional activities of the students and faculty.
Criterion 8 – Institutional Support:
This criterion is often cited inappropriately in findings. Focus findings on the issue’s impact on the program. For example, do not write findings that require a program to hire additional personnel; write the finding based on what is not being accomplished, and let the institution decide whether to resolve it with additional personnel or by other means.
Most programs will have program criteria that must be satisfied. The program criteria are prepared by the professional societies that you, as a PEV, represent. Depending on the program’s degree name, it must satisfy the appropriate program criteria as well as the eight general criteria. Be sure to review these requirements and evaluate how well the program being visited satisfies the current version of the criterion. If you have questions before a visit on interpretation of a program criterion, contact your professional society mentor or your professional society’s ABET liaison for clarification.
Program Criterion Changes:
The applicable Program Criteria for the program being evaluated should have been determined by ABET HQ. If there is any question about this, consult your Team Chair. PEVs must examine the applicable Program Criteria and see if the program has satisfied those requirements in addition to the requirements in Criteria 1 through 8. Carefully check to see if all of the requirements of the Program Criteria are addressed by the Self-Study. Check to see if appropriate assessment, evaluation, and continuous improvement are accomplished for any outcomes required in the Program Criteria. Failure by the program to address Program Criteria should be discovered and communicated to your Team Chair well before the visit so that the program can be given an opportunity to remedy this oversight.
C. ABET Policy and Procedures (APPM) Issues
Name of the Program: The official name of the program is the name precisely as it appears on the Request for Evaluation (RFE); this is the exact name that should be used in all documents throughout the accreditation review. It is important that this program name also be shown on the student transcripts in the same way as on the RFE. Additions to the official name of the program, as shown on the RFE, on the student transcripts should trigger investigation related to program criteria and the appropriate number or type of program evaluators. If a PEV believes there may be an issue of this sort, contact the Team Chair immediately so that it can be investigated.
Modes and Locations of Instruction: Be aware of any online and off-campus or remote offerings of the program. If it is possible for a student to take a significant amount of technical courses at sites in different modalities, (e.g., face-to-face at places other than at the home campus or online including via the Internet) other than at the home campus, then:
- An online/hybrid/multiple-site program may require a greater time commitment in preparation and evaluation than is normal for a single site program delivered face-to-face.
- If a program, or portions of a program, is offered at multiple sites, the program must be able to demonstrate that the program is equivalent at all sites.
- The program should be prepared for the team to visit any site at which the program is offered and these sites must be reviewed as part of the accreditation review.
- The “weakest link” concept applies to the program evaluation and if an issue is found within one delivery modality or at a specific site, the finding and any resulting accreditation action will apply to the program in its entirety, regardless of delivery method or location.
- The program must demonstrate how it assures that development of student outcomes at the remote site is equivalent to that at the home primary site and campus being reviewed.
- The program should provide separate course/assessment materials for each delivery method/location and include graded student work ranging from excellent through poor for students for each delivery method. Assessment materials from remote sites should be disaggregated from other sites to allow for effective assessment and evaluation processes of all routes to the degree. Otherwise, lack of student attainment of outcomes at a remote site may be masked by assessment data from the home campus.
Here is a list of specific APPM requirements that may lead to findings:
- I.A.4. An institution may not use the same program name to identify both an accredited program and a non-accredited program.
- I.A.6. Each accredited program must be identified as “accredited by the Engineering Technology Accreditation Commission of ABET, http://www.abet.org”
- I.A.6.a. Accredited programs must publicly state their educational objectives and student outcomes. A shortcoming can be written if the information is extremely difficult to locate by the public.
- I. A.6.b. Accredited programs must publicly post annual enrollment and graduation data per program. A shortcoming can be written if the information is extremely difficult to locate by the public.
- I.C.4.b. Program name must be shown consistently on transcripts, all publications, and the RFE.
- I.C.4.c.(2). All program criteria for any implied program specialization must be satisfied.
- I.C.5. To be eligible for an initial accreditation review, a program must have at least one graduate within the academic year prior to the academic year of the on-site review.
- I.E. All paths to completion of the program must satisfy the appropriate criteria.
- I.E.5.b.(1) Facilities used by the program must be adequate and safe for intended purposes.
- I.E.b.(2) Displayed course materials should include syllabi, textbooks, example assignments and exams, and examples of student work. Note: Examples of student work may be shown via course specific materials or sufficient examples may be seen via inspection of the various materials provided as part of the program’s Criterion 4 display materials.