The evaluation of a program’s compliance with each of the various features of the Program Educational Objectives, Student Outcomes, and Continuous Improvement Criteria (Criteria 2, 3, and 4) is an important element of ABET’s outcomes-based accreditation criteria and the program’s continuous improvement processes. Although you will be reviewing many aspects of the program visited, your evaluation of the program’s process for the assessment, evaluation, and implementation of identified needed improvements relative to its stated program educational objectives and student outcomes will be an important part of your work. This module will provide you with information that will help you in the evaluation of these processes.
A. Terms and Definitions Used by ABET
Unfortunately, there is no universally accepted set of terms used in the assessment field. The table below defines the terms used in the ABET Criteria and defined in the Accreditation Policy and Procedure Manual.
|Program Educational Objectives
||Program educational objectives are broad statements that describe what graduates are expected to attain within a few years after graduation. Program educational objectives are based on the needs of the program’s constituencies.
Student outcomes describe what students are expected to know and be able to do by the time of graduation. These relate to the knowledge, skills, and behaviors that students acquire as they progress through the program.
Assessment is one or more processes that identify, collect, and prepare data to evaluate the attainment of student outcomes and program educational objectives. Effective assessment uses relevant direct, indirect, quantitative, and qualitative measures as appropriate to the objective or outcome being measured. Appropriate sampling methods may be used as part of an assessment process.
Evaluation is one or more processes for interpreting the data and evidence accumulated through assessment processes. Evaluation determines the extent to which student outcomes and program educational objectives are being attained. Evaluation results in decisions and actions regarding program improvement.
Programs may have adopted a specific language of assessment, which varies from the terms above. It is also possible terminology will vary from one program to another within an institution. If a program is using different terms, it is important it defines its terms in its Self-Study and uses them consistently in its documentation for ABET. If the Self-Study does not clearly indicate how terms are being used, this should be clarified before the visit.
B. Review of Program Educational Objectives
Program educational objectives focus on what graduates are expected to attain within a few years after graduation. Due to recent changes in the harmonized Criteria 2, it is no longer necessary to assess the attainment of the program educational objectives but “There must be a documented, systematically utilized, and effective process, involving program constituencies, for the periodic review of these program educational objectives that ensures they remain consistent with the institutional mission, the program’s constituents’ needs, and these criteria.”
The review of program educational objectives requires appropriate monitoring of the currency of the objectives themselves. The currency of the program educational objectives should be reviewed periodically. The time span will depend on the changing needs of the constituents and mission of the program. Programs in dynamic and rapidly changing disciplines will need to have more frequent monitoring cycles to be sure the program educational objectives are current and the student outcomes will enable the attainment of the objectives.
Information on the needs of constituents for the development and revision of the program educational objectives should be gathered in meaningful ways. Determining compliance with this aspect of Criterion 2 will take informed judgment on the part of the Program Evaluator.
Note: The program defines the groups that make up its constituencies, with the exception of ETAC programs. ETAC specifies that representatives of industries that hire program graduates must be included via their advisory board.
C. Assessment and Evaluation of Student Outcomes
For student outcomes, the focus of the data collection is to answer the question, “Can the program demonstrate the level to which students have attained the anticipated student outcomes?” The evidence of student learning is then used to identify student strengths and weaknesses related to each of the student outcomes for making decisions about how to improve the program teaching/learning processes.
This evidence should be the product of faculty reviewing and/or observing student work related to the program requirements. In preparation for reviewing a program’s processes related to Criterion 4, Continuous Improvement, for student outcomes, it is important to understand several principles of a well-constructed process to enable continuous improvement related to program-level student learning.
The following are the underlying principles of continuous quality improvement of student learning at the program level.
- The focus of Criterion 4 (continuous improvement related to student outcomes) is on the learning of students and not the assessment or evaluation of individual students.
- The focus of Criterion 4 (continuous improvement related to student outcomes) is on the learning of students and not the assessment or evaluation of individual courses.
- Student outcomes should be defined in order for faculty to have a common understanding of the expectations for student learning and to achieve consistency across the curriculum.
- A program does not have to collect data on every student in every course to know how well it is doing toward the attainment of student outcomes.
- To provide evidence of attainment of student outcomes by the time of graduation for program reporting purposes, programs may choose to evaluate and report only data collected in courses towards the end of the curricular cycle.
- A program does not have to assess every outcome every year to know how well it is doing toward the attainment of student outcomes.
- The focus is on continuous improvement based on information for decision-making.
D. How do I know if a program has an adequate continuous quality improvement process for student learning?
Evidence of a CQI process would contain the following:
- A timeline of repeated activities related to the assessment and evaluation of student outcomes. Possible question: “What is your data collection and evaluation timeline?”
- Agreed upon student outcomes and how they will be assessed. (Identifying a few performance indicators per outcome is an effective way to develop measurable definitions.) Possible question for faculty: “How does your program assess its student outcomes to ensure consistent assessment across the curriculum?”
- Systematic data collection focusing on direct evidence of student performance related to the student outcomes. Possible question: “Where do you collect the data that is evidence of student learning?”
- Systematic data collection ensuring coverage of each student outcome for the given student cohort. Possible request: “Describe how the data being presented were collected.”
- Data collection and analysis providing information that enables faculty to identify superior performance and opportunities for improvement related to the outcomes. Possible question: “I see X% of your students have attained outcome Y. Were there any notable positive or negative aspects of the students’ performance?”
- An evaluation process clearly communicating to program faculty opportunities for improvement in student learning. Possible request: “Describe how the proposed actions improved student learning (or are anticipated to improve student learning) related to the enhancement opportunities identified.”