Printer-friendly version
 

 Module 5: Applying the Criteria

Up until this point, you have learned many of the fundamentals of the ABET Accreditation Process. Now, you will apply what you have learned in a series of exercises, in checks for understanding, and, ultimately, in the simulated preparation for the campus visit.   

Points of Learning

  • Understanding the Criteria

  • Applying the Criteria

  • Submitting Your Simulation Materials
Criteria Application Basics
  • ABET accredits educational programs leading to degrees.
  • ABET defines an educational program as an organized educational experience that consists of a cohesive set of courses or other educational modules sequenced so that a student gains reasonable depth in upper-level courses.
  • ABET does not dictate program names to an institution.

ABET does NOT accredit institutions, departments, or degrees.  

You can find a thorough explanation of the above in the Accreditation Policy and Procedure Manual Section II.B. and Section II.D. Read Section II.B. and Section II.D. now. 
Understanding the Criteria 
Using the Criteria to evaluate a program's compliance begins with understanding the Criteria. ABET Criteria are minimum standards that you will apply with judgment. General Criteria cover the following areas of an educational program:
  • Students
  • Program Educational Objectives
  • Program Outcomes
  • Continuous Improvement
  • Curriculum 
  • Faculty
  • Facilities
  • Support
Your Member Society may also have additional Criteria that cover minimum standards for the specific program discipline you will be evaluating. These are called "Program Criteria."

 

Read the Criteria now! The term "ABET Criteria" refers to both General and Program Criteria.
Common Issues Associated with Each Criterion
Over the years, ABET has identified common issues that may surface as you review a program's Self-Study Report for evidence of compliance. ABET provides a common set of questions that you should ask yourself as you read through the Self-Study Report.

The issues listed below for each Criterion area are not exhaustive. You may identify additional issues as you review the Self-Study Report.
You are evaluating the program based on the Criteria and the strength of the evidence provided by the program . . . NOT YOUR OWN PERSONAL PREFERENCES.  
Students
  • Problems with student advising (often cited with Faculty Criterion).
    • Ineffective and inconsistent advising.
    • Lack of understanding of curricular requirements, especially if many options are available.
  • Ineffective monitoring.
    • No documentation of course substitutions or missing prerequisites.
  • Problems with transfer process.
    • No documentation on acceptability of transfer credits.
Program Educational Objectives
  • Educational Objectives not published or readily accessible to the public.
  • Educational Objectives not related to institutional mission.
  • No evidence of constituency input in setting or periodic evaluation of objectives.
  • No process for evaluating the extent to which objectives are attained.
  • No data available on the attainment of Educational Objectives.
Questions You May Want to Ask About Program Educational Objectives

  • Objectives

    • How were the objectives determined?

    • Are they consistent with the mission statement?

    • Are they statements describing accomplishments after graduation?

    • Are they measurable?

    • How does the program accomplish its objectives?

    • What is the process to review and update objectives?

    • How does the program know when the objectives are achieved?

  • Constituencies

    • Who are the program's constituents?
 

Program Outcomes

  • No evidence demonstrating one or more outcomes.
  • Outcomes not assessed directly (student performance).
    • Anecdotal versus measured results.
    • Reliance on course grades as assessment of outcomes.
    • Over-reliance on indirect assessment (e.g. surveys)
  • No systematic assessment process.
    • No process or the process is not documented.
    • Plans developed but not implemented.
  • Little or no faculty support for the process.
Questions You May Want to Ask About Program Outcomes

  • Outcomes

    • What should the students know and be able to do upon graduation?

    • Are students familiar with the required outcomes?

    • How does the program define the outcomes to make them measurable?

  • Processes

    • What are the program's processes to achieve the outcomes?

    • How is the process documented?

    • What does the process do?

    • How does the program know the process does what they claim?

    • What is faculty/administration involvement?

  • Assessment

    • Are data being collected for each outcome?

    • What is measured? How often?

    • How does the program use the data collected?

  • Results

    • How does the program demonstrate that outcomes are achieved?

    • What is the program's evaluation of its quality?

    • Based on the program's evaluation of the assessment data, what are the plans for additional improvement?

    • Exactly which outcomes must each graduate attain?
  • A system must be in place to identify the degree to which students have achieved the required outcomes of the program.
  • The level of achievement may vary among students so long as the variation is consistent with the program objectives.
  • Look for processes in place that provide for
    • Definition of desired, measurable outcomes
    • Collection of data linked to the outcomes
    • Analysis of data and evaluation of results
    • Implementation of change
    • Repetition of cycle and review

Continuous Improvement

  • Results of evaluation of objectives not used to develop and improve the Program Outcomes.
  • No evidence that the assessment results are being applied to develop and improve the programs.
    • Assessment results not used.
    • Assessment and development cycle not complete.
Questions You May Want to Ask About Continuous Improvement

  • Constituencies

    • How does the program involve constituents in the continuous improvement process?

  • Processes

    • How does the program know the process does what it claims?

    • What is faculty/administration involvement?

  • Assessment

    • Are data being collected for each outcome?

    • What is measured? How often?

    • How does the program use the data collected?

    • Does the evaluation of data provide the information needed to make program improvements?

  • Evaluation

    • Are assessment results systematically evaluated to determine needed areas of improvement?

    • Are improvements made as a result of data collected and evaluated?

  • Results

    • What is the program's evaluation of its quality?

    • What evidence does the program have that improvement efforts are producing results?

    • Based on the program's evaluation of the assessment data, what are the plans for additional improvement?
KEEP IN MIND: You don't have to be an expert on assessment. The program must provide evidence that it has a working and effective system in place. The program must describe a clear relationship between Program Educational Objectives, Program Outcomes, and measurable indicators of success with required levels of achievement.  Note that objectives, outcomes, and continuous improvement are linked closely together. 
You are evaluating the program based on the Criteria and the strength of the evidence provided by the program . . . NOT YOUR OWN PERSONAL PREFERENCES.  
Curriculum

A program's curriculum provides the foundation for entry into the profession. Issues related to this Criterion could be

  • Curriculum fails to meet semester credit hour requirements (if specified by Criterion)
  • Quality of the major design or capstone-integrated experience (if required by the Criterion)
    • No culminating experience - analysis or research instead of design or capstone; several courses with elements of design
    • Multiple capstone courses with widely varying quality
    • Design or capstone experience not addressing multiple constraints
    • Variation in courses taught by different instructors
Faculty
  • Insufficient number
    • To support concentrations, electives, etc.
    • To provide student advising
  • Poor faculty morale affecting the program
    • Lack of professional development
    • Excessive workloads
    • Retention/turnover rate
    • Salaries (as they relate to retention and recruiting (cited with Institution Resources))
    • Excessive reliance on adjuncts
    • Heavy reliance on temporary faculty appointments, potentially jeopardizing program stability (cited with Institution Resources)
Facilities
  • Insufficient space
    • Overcrowded laboratories and classrooms
  • Laboratories
    • Unsafe conditions
    • Inoperable equipment
    • Lack of modern instrumentation
    • Lack of funds for upgrading (cited with Support)
  • Computing/information infrastructure
    • Lack of funds for upgrading (cited with Support)
    • Lack of software/hardware needed to support the curriculum
Support
  • Unstable leadership affecting programs
    • Dean/program head positions open or filled by interim appointments for an extended period
    • Frequent turnover of university administration and unit leadership
  • Inadequate operating budget affecting
    • Acquisition and maintenance of laboratories and computing equipment
    • Faculty salaries, promotions, and professional development affecting hiring and retention
  • Insufficient support staff
    • Teaching assistants
    • Technicians for instructional laboratories, machine shops, and laboratory services
    • Administrative/clerical
You are evaluating the program based on the Criteria and the strength of the evidence provided by the program . . . NOT YOUR OWN PERSONAL PREFERENCES.  
Your Draft Visit Plan should detail with whom you will visit on campus to resolve any issues with program compliance to Criteria that are not answered to your satisfaction in the Self-Study Report. 
The Decision-Making Process

Pre-Visit

By completing the Program Evaluator Worksheet and Program Evaluator Visit Report specific to your commission, you should be able to make a preliminary evaluation of the program based on your review of the program's Self-Study Report. You should make a list of those issues that will require further investigation on campus and discuss these with your Team Chair.

On Campus

Once on campus, you may revise your evaluation after resolving issues by conducting interviews with faculty members, students, and administrators; reviewing documentation; and visiting facilities. You will share your findings with your team members at team meetings on Sunday and Monday nights. This will assist you in refining your recommended action. At the conclusion of the visit, you will provide your Team Chair with the recommended action for your program and an Exit Statement to support that action. You will simulate the on-campus activities during the Face-to-Face Training component.

Post-Visit

The Team Chair develops the Draft Statement to the institution by combining and editing the program exit statement material from the Program Evaluators and adding material that applies to the institution as a whole. The Draft Statement is reviewed for adherence to standards and consistency with other statements by two editors and ABET headquarters staff. It is then sent to the institution, which has 30 days to respond. The Team Chair uses the response from the institution to prepare the Final Statement, which is edited again and then provided to the full Commission for action. In preparing the Final Statement, the Team Chair may consult with the Program Evaluators as needed to determine whether there are any changes to the recommended accreditation action because of the institution's actions since the visit. Final accreditation decisions are made at the Summer Commission Meeting in July.
Evaluating a Program's Compliance to the Criteria
Right now, you are probably thinking, "How do I decide if a program is in compliance with each Criterion? And how do I recommend an accreditation action?" Follow these steps:

1. Identify issues by Criterion. Remember that you may find issues not listed in the Common Issues Associated with Each Criterion section above.

2. Determine the appropriate finding.

In The Accreditation Process, you read about Levels of Compliance, statements of compliance, concern, weakness, and deficiency, as well as observations with regard to your findings when evaluating a program. You can also find these in the Accreditation Policy and Procedure Manual, Section II.E.12.

  

Take a minute to review the Levels of Compliance.
3. Select the key term that applies overall for each Criterion. Each Criterion has multiple elements. You need to use good judgment in weighing the contributions of each element when deciding if a program complies with a Criterion. For example, the Criterion for students requires that the institution evaluate student performance, advise students, and monitor their progress so that the students can attain the program objectives. As the Program Evaluator, you need to weigh each element of an institution's processes regarding students to determine compliance but your decision should be based on the overall Criterion, not solely on one element. Base your decisions on the Criteria, NOT on your opinion. Consider the resulting recommended action. Is it consistent with the nature of the shortcoming?

4. Explain each concern, weakness, and deficiency in relation to the specific Criterion.

5. Recommend the accreditation action.

Prior to the campus visit, your Team Chair will ask you where the program stands in overall compliance to ABET Criteria. Based on your preliminary review, you will select one of the following potential actions as described in the Accreditation Policy and Procedure Manual, Section II.F. This preliminary judgment may be revisited after you gather more information during the campus visit.
NGR Next General Review
IR Interim Report
IV Interim Visit
RE Report Extended
VE Visit Extended
SC Show Cause
SE Show Cause Extended
NA Not to Accredit
T Terminate
General Review Terminology vs. Action

If in your judgment a program has a weakness for a given Criterion, you must recommend either an Interim Report or an Interim Visit action.

If in your judgment a program has a deficiency for a given Criterion, you must recommend a Show Cause action if this is a re-evaluation or a Not-to-Accredit action if this is an initial evaluation.

Refer to the table below:

Weakness? No Yes Yes --- ---
Deficiency? No No No Yes Yes
Actions NGR IR IV SC NA
Definitions Next General Review Interim Report Interim Visit Show Cause (only for a re-accreditation) Not to Accredit (only for new programs)
Duration (years) 6 2 2 2  
Ensuring Consistency
Accreditation actions must be consistent across all programs and institutions. Accreditation actions must be consistent with actions given for other programs with similar shortcomings (concern, weakness, deficiency).

Throughout the ABET Accreditation Process, there are five check points to ensure consistency. (See chart below.) 

Sample Situations
What would you decide? Try out these sample situations. You want to determine the best response given the information you are given.

Situation #1

You are a part of a team evaluating five programs at Bay State University. Four of the five programs do not have their Educational Objectives published anywhere. As a result, the team determines that these four programs are clearly not in compliance with the Program Educational Objectives Criterion of the Criteria, and the team agrees that those programs be given a deficiency with respect to this Criterion. However, the program that you are evaluating has its objectives published on its website but not in the university undergraduate catalogue nor in any of the departmental promotional materials handed out to both enrolled and prospective students.

Select the best Program Evaluator and team response:

  1. The program you are evaluating is not in compliance since the objectives are not published in each location where prospective students and the general public might look for them, and there needs to be consistency among the actions taken for different programs at the same university. Therefore, the team should agree that the program should be given a deficiency.
  2. The team agrees that you will discuss this with the program head to try to resolve the issues before the team leaves the campus. If the response of the program is satisfactory for resolving the problem, you will not report it in the Draft Statement or Exit Interview.
  3. The program is in partial compliance with the Criterion but lacks the strength of full compliance and the team agrees the program should be given a weakness.
  4. Since the objectives are published someplace and the Criterion is silent on where the objectives need to be published, the Program Evaluator should consider this as a concern at most and perhaps not mention it at all to the team.
Answer Key
  1. This would not be appropriate because the Criterion doesn't require that the objectives be published in all documents related to the program. Further, consistency among the program actions is not an issue; different actions for different programs are appropriate if the characteristics that are the basis for a decision are different.
  2. This is not appropriate because all the relevant facts are known, and it is not appropriate to leave without making decisions on all issues relative to the Criteria.
  3. This may be the best alternative, although if there is evidence that the program's website is the most-referenced source of information about the program, then a concern could be appropriate.
  4. This could be appropriate if the website is clearly the primary source of information about the program for prospective and current students. It should be mentioned to the team in any case, however, especially given that the other programs do not publish their objectives anywhere.
  • Note: It is not necessary that the program objectives be published in every document about the program, but they should be included in all documents that are readily used by the public and by current and prospective students to obtain information about the program.
 

Situation #2

You are evaluating a program and its Self-Study Report indicates that a significant portion of the outcomes assessment data will be made available during the campus visit. Once on campus, you find that these materials are neatly compiled statistical results of several assessment instruments - recent teaching evaluations, an alumni survey, and employer interviews. These are claimed to support the Program Outcomes Criterion as evidence of an assessment process with documented results.

Select the best Program Evaluator response:

  1. Review the materials and identify trends in the responses that would suggest appropriate ways to improve the program you are visiting; report these to the program chair.
  2. Request that an executive summary of the results be provided before you depart from campus.
  3. Request additional documentation of how the results of the surveys are applied in program improvement. If such evidence is not available, discuss with the Team Chair and other team members the option of citing the program as being deficient with respect to the Program Outcomes Criterion: assessment results are not applied to the further development and improvement of the program. Suggest to the program chair that direct assessment of student work may provide the most objective form of outcomes assessment.
  4. Tell the Team Chair that the program will be cited for a deficiency with respect to the Criterion and that they face a probable Show Cause recommended action.
Answer Key
  1. The Program Evaluator is not expected to engage in data analysis. It is up to the program faculty members to make the case for their claims.
  2. This approach may seem appropriate on the surface, but the most important issue in this case is how the data are used. (See next option.)
  3. This is the best first step. The Program Evaluator may find that the data has been used in some way but not documented.
  4. This would be an appropriate action if the evaluator determines that this is all that was done, there was no serious attempt to use the data, and the faculty and chair did not seem to be committed to using the data in any manner.
 

Situation #3

The program you are visiting has in its Educational Objectives the program's ability to prepare its undergraduates for successful careers both in the traditional chemical process industry and in the novel field of microelectronics. The faculty C.V.s contained in the Self-Study Report indicates that no faculty member has expertise in the microelectronics area. After the faculty interviews, you confirm that this is the case and that the faculty member in charge of teaching the one senior-level course in microelectronics processing has been learning the course material on the fly from a new textbook in the area. Interviews with the students provide evidence that the faculty member teaching the microelectronics processing course is a favorite instructor. Students like the course and find the material to be very easy. Consultation with the program head suggests that the course content has very little rigor for a senior-level course. You conclude that the evidence indicates that coverage of microelectronics is weak at best.

Select the best Program Evaluator response:

  1. As the Program Evaluator, you encourage the chairperson either to remove the objective from the published information or to provide the faculty with development opportunities to enhance their expertise in the microelectronics field. In the meantime, the program is cited with a weakness because of the lack of strong compliance with the Faculty Criterion: The faculty . . . must have the competencies to cover all of the curricular areas of the program. The faculty . . . must ensure the proper guidance of the program and its evaluation and development.
  2. In your Exit Statement, cite a deficiency for the Faculty Criterion and suggest that the administration provide funds for hiring additional faculty members to provide expertise in the microelectronics field.
  3. You encourage the chairperson for the program's attempts at providing innovation in its Educational Objectives. No weaknesses, concerns, or deficiencies need be cited.
  4. Report a concern about the microelectronics curriculum in regard to the Curriculum Criterion.
Answer Key
  1. This may be the best response if the Program Evaluator believes the faculty member who is teaching the course is committed and capable of developing the needed expertise in a short period of time with the proper support.
  2. This is an inappropriate response unless the Program Evaluator determines that there are other factors, such as local industry and/or the school administration, demanding that the microelectronics option be provided.
  3. This is an unacceptable option in view of the findings of the Program Evaluator.
  4. This is an unacceptable response in two ways. First, there is no explanation of the basis for this concern. Second, it ignores the issue of faculty expertise as required by the Faculty Criterion and the need for this emphasis in the program (constituency support indicated through Program Educational Objectives Criterion).
Analyzing Student Transcripts
Student transcripts provide direct evidence that the institution's program requirements are met. In addition, transcripts provide evidence that the curricular requirements are met. As part of your review of the Self-Study Report, you will need to analyze transcripts. Procedures for doing this include the following:
  • Be sure that the transcripts identify the name (title) of the degree received in a way that clearly identifies the program as an accredited program according to the institution catalog and other documents and in a way that distinguishes it from any non-accredited programs with which it could be confused by a potential employer. Identify any problems in this regard to your Team Chair.
  • Make sure that the courses that are counted toward the degree are consistent with the published requirements of the program. In cases where the transcript is for a graduate of an earlier curriculum, the institution must provide a copy of the appropriate curriculum. The institution also should provide justification for any variances, such as transfer credits or substitutions that are not clearly documented on the transcripts.
  • Check to be sure that prerequisites are taken before each course that requires them and that the course sequence on the transcript does not vary unreasonably from the recommended sequence. If courses are taken out of sequence, check to see if there is an indication of difficulty for the students in terms of the course grades. (If there are difficulties for students, then there could be a problem with the mechanisms for advising and the enforcement of prerequisites. If there are no problems, it could indicate prerequisite requirements that are not needed.)
  • Ensure that the number of transfer credits and the number of course substitutions are reasonable.
  • Request clarification for any apparent problems in the transcripts. Do your transcript analysis and request clarifications soon enough to allow reasonable time for the institution to respond.
Analyzing the Mock Self-Study Report
Are you ready to apply your newly gained knowledge to simulating PEV pre-visit activities? You will:
  • Review a mock Self-Study Report. Use the PEV Worksheet for your commission to walk through the Self-Study Report and capture your initial evaluation.
  • Make a preliminary decision on findings.

This Self-Study Report has been created for training purposes only. It should not be distributed. While based on responses typically seen in a Self-Study Report, this document intentionally contains problems for training purposes.  

Using the appropriate pre-visit forms, review the Upper State University Self-Study Report for compliance to the ABET General Criteria.

IMPORTANT: You are reviewing the Upper State University program against General Criteria ONLY. 

NOTE: This review may take up to eight hours.

Applied Science Evaluation

  • This Self-Study Report will be available soon.

Computing Evaluation

Engineering Evaluation

Technology Evaluation

The following must be completed as part of the PEV Training Pre-Work:

  • Checks for Understanding
  • Program Evaluator Worksheet
  • Program Evaluator Visit Report: Transcript analysis and curriculum analysis ONLY.

No later than three weeks before the Face-to-Face training, you should complete the pre-work and submit the Worksheet and Visit Report to your Member Society, your Mentor, and ABET headquarters (training@abet.org).

 

Copyright© 2008 ABET, Inc.