Overview

The Program Assessment Policy is a crucial component of the PSU’s continuous improvement process in relation to the Teaching and Learning Quality Framework(TLQF) that complements the Institutional Quality Framework. All programs in PSU are required to implement a variation of the Program Assessment Process described in this policy that is in compliance with the Saudi Arabian Qualifications Framework.

This policy has Spady’s (1994), an educational researcher, outcomes-based education as its premise. He claimed that demonstrating learning is the essential point of assessing learning. It is not just a matter of knowledge of the content but being able to apply it.

Purpose of Assessing the Program

The main purpose of program assessment is to help programs recognize ways in which they can improve the quality of programs with respect to teaching and learning, thereby closing the program assessment cycle loop. It is also to track the continuous improvement of the implementation of teaching and learning within the program, assess the achievement of program learning outcomes, and assess the achievement of the course learning outcomes.

Scope

This policy applies to all programs and departments on campus that have courses that contribute to a degree-granting program at the undergraduate and graduate levels.

Definitions

1. Program Assessment: It is the systematic and ongoing method of collecting, analyzing, and using information from measured outcomes (both direct and indirect) to improve the quality practices of student learning. It is different from an individual course, student, or faculty evaluation assessment. It utilizes the data gathered to make informed decisions about the quality improvement of the program.
2. Continuous Improvement: It is the monitoring, tracking, evaluating, analyzing of data gathered about the quality enhancements needed to maintain the level of quality education expected at PSU.
3. ILO (Institutional Learning Outcome): These learning outcomes focus on the broad conceptual knowledge and adaptive soft skills required by the institution upon graduation.
4. PLO (Program Learning Outcome) or SO (Student Outcome), depending on the program. These learning outcomes focusing on the overall skills students will be able to do upon completion of the program.
5. CLO (Course Learning Outcome): These learning outcomes are what students will be able to do upon completion of course. It should only include one action verb per outcome and refer to learning that is transferable.

(Kindly refer to the Student Assessment Policy for further definitions.).

Policy

It is mandatory that all programs take part in the implementation of the program assessment requirements. This involves attending any necessary training or workshops related to program assessment conducted by the concerned center(s) from DQAD. It also involves submitting documents like the assessment plan for the program and a record of its implementation to the DQAD concerned center(s) for monitoring. It is also mandatory to use rubrics for assessing student achievement of learning outcomes.

Prerequisite to the Assessment Process:

• The program objectives should be clearly defined to all the stakeholders of the Program.
• The Program Learning Outcomes (PLO)1 should be clearly defined to all the stakeholders of the Program.
• Complying with International Accrediting Bodies standards or criteria based upon their specific disciplines. For example, for ABET, a program will have established Program Objectives (PO) to comply with international accrediting standards for a specific discipline that are aligned with its Program Learning Outcomes (PLO) or the terminology appropriate with the accrediting body from their specific disciplines.

III. In some programs, Program Learning Outcomes (PLOs) are called Student Outcomes (SOs) in compliance with the international accrediting body’s standards and criterion.

JJJ. The program curriculum is mapped with its Program Learning Outcomes, which demonstrates how learning activities in specific courses map to these learning outcomes. (Key: I- Introduced, R- Reinforced, A-Advanced)

KKK. The Program Learning Outcomes (PLO) are aligned with the Course Learning Outcomes (CLO) of every Foundation, Core and Elective courses

LLL. The Course Learning Outcomes for a particular course are aligned with various assessment methods.

MMM. The Course Learning Outcomes for a particular course are aligned with various teaching strategies (methods).

his entire mapping of the Program Learning Outcomes with its course learning outcomes, teaching strategies, and the assessment methods has been represented by a diagram, as shown below:

The Assessment Process:

The T&L Quality Assurance and Improvement process are based on a continuing annual cycle based on the principles of PIMRU (Plan→ Implement →Monitor →Review → Update, Figure 1 below).

FIGURE 1: THE PIMRU MODEL

Planning:

Every department is required to prepare an Academic Program Assessment Plan (AP), which shows a roadmap of assessing the programs at two levels.

• Course Level Assessment
• Program Level Assessment

Assessment Plans should be based on OBE principles and Learning Outcomes. Assessment Plans should include course level and program level assessments in accordance with PSU’s rules and regulations. Refer to the program assessment handbook for further details.

Course Level Assessment

The course instructor will select relevant assessment tools for course-level assessment with the help of which each CLO in the course will be measured using both direct and indirect assessment methods (e.g., course exit surveys) via a learning outcome’s rubric. The Academic units will have the discretion of deciding the process of course-level assessment.

• The selected assessment method(s) should cover all CLOs in a course in every academic semester.
• The selected tools account for at least 50% of overall course assessment, such as a final exam or major exam(s) and a course project.
• The course instructor should conduct a Course Exit Survey every academic semester or year, depending on the department policy.
• The Course Assessment Plan should incorporate a Review-Update cycle to illustrate the continuous improvement of the programs.
• The course assessment plan should be approved by the course group supervisor or PAC.

Program Level Assessment

• The Assessment Plan should be prepared in which all the PLOs will be assessed using direct or indirect methods over a period of 3 or 5 years. Every year a subset of the full list of Program Learning Outcomes is required to be assessed.
• The PAC identifies the courses to be used in assessing the selected program learning outcomes for the current year. The courses which are highly contributing to the selected PLO should be used for direct and indirect assessments. It is advisable to use mainly program core courses for assessing the program learning outcomes.
• The Program Learning Outcomes should be aligned with the Saudi Arabian Qualification Framework (SAQF) categorized under three domains of learning (Knowledge, Skills, and Competencies).
• The program assessment plan should be submitted by the PAC to the department chairperson and then subsequently to the department and College Council for approval.
• Once approved by the College Council, the academic unit level Academic Assessment Plans are raised to the Program Assessment Committee (the committee is comprised of the members from the TLC or Quality committees in the program and college) for review and coordination.
• Every semester the courses that have been selected for PLO assessment will prepare a detailed course assessment plan mentioning what CLOs and assessment methods are being used to measure the selected PLOs. The CLOs which are highly contributing to the PLOs should be assessed.

Implementing

• The approved course assessment plan is implemented using direct (such as Rubrics) or indirect methods (using Course Exit surveys).
• Academic departments are free to make use of the appropriate tools and approaches to measure the learning outcomes based on an acceptable standard scale.
• At the end of every academic semester, the course instructors are required to submit the CLOs and the PLO assessment results for the courses selected for the assessment.
• The course instructor will also report all CLO results using direct and indirect methods in the semester in accordance with the course report per NCAAA regulations.

Monitoring

• The PAC members are required to monitor the assessment process throughout the semester.
• At the end of every semester, the PAC and the Course Group or Course Coordinators collect the CLO and PLO assessment results (direct and indirect) for the respective courses and analyze the Program assessment data using appropriate calculations and tools. These data are then submitted to the department chairperson to be included in the decision making process and continuous enhancement of the program.

Reviewing (Auditing) and Updating (Closing the Loop)

Assessment Plans should incorporate a Review-Update cycle to illustrate the continuous improvement of the programs.

• Based on the analysis done in the above step, PAC will prepare detailed recommendations for improvement or promotion of good practices in coordination with program course groups or leaders and the department chairpersons.
• The program assessment results are recorded in the annual program report and reported to DQAD for monitoring and feedback at the institutional level.
• The recommendations for improvement and the collected assessment data mentioned in the annual program report will be reviewed and approved by the PAC and subsequently by the department chairperson.
• DQAD will follow an annual audit session where it will study the recommendations, summarize the issues, observations, and recommendations, and raises a report for major actions to the concerned parties.
• The whole process is documented for accountability and review purposes.

Instruments or Tools used for Learning Outcome Assessment

Rubrics: Course instructors will be asked to design rubrics for a particular learning outcome to assess it in any of the selected course assessments.

Course Exit Surveys: at the end of each semester, students will be asked to rate the achievement of learning outcomes for the respective course through a survey. Faculty members will upload a survey about the learning outcomes achievement onto Moodle to be completed by the students in their courses.

Course Portfolios: A course portfolio will be maintained for each of the taught courses. The collected material may also be used by the course teacher as another means to assess the achievement of course learning outcomes.

using stakeholder satisfaction surveys

1. The following surveys have been adopted at the university level as a measure of Indirect assessment for monitoring the quality of the programs.

Program Exit Survey(PES),
Midway Survey (SES)
Course Evaluation Survey(CES)
Faculty Satisfaction Survey(FSS)

Alumni Satisfaction Survey: Alumni are asked to rate the program outcomes and give their feedback regarding their experience through a survey that will be distributed annually.

COOP Employer Satisfaction Survey

Employer Satisfaction Survey(ESS): Employers will be asked to rate the achievement of program outcomes through a survey that will be distributed periodically.

• These surveys are conducted periodically by the Center of Statistical Unit(CSI).
• The Program/Department Chairs are required to establish the internal and external benchmarks for the survey results.

The Center of Statistics and Information (CSI) is responsible for compiling the PSU survey results annually and submitting the report to the respective departments/Colleges.

The department will analyze further the survey results based on the internal and external benchmarks, program delivery, and recommend appropriate action(s) based on the survey results for continuous improvement of the quality of the program.

The survey results, their analysis, and recommended actions are reported in the annual program report required by PSU in accordance with NCAAA’s regulations.

Key Performance Indicators (KPIs):

1. PSU has adopted NCAAA’s KPIs to assess and exhibit the quality of the academic programs at the Program level, College, and Institutional levels. In addition to that, every department can establish its own KPIs to assess the quality of its programs.

• The CSI is responsible for compiling the KPI results annually.
• The academic departments will establish the internal and external benchmarks for every KPI for their program and analyze the KPI results annually using the internal and external benchmark and recommend appropriate action(s) based on the KPI results to improve the quality of the program.
• The KPI results, their analysis, and recommended actions are reported in the annual program report.

Reporting the Assessment

All programs need to provide their assessment reports along with the annual program report to DQAD, which will become a part of the program’s self-study evaluation process and documentation. The assessment reports need to be evidence-based discussions of student achievement of learning or concrete data. The reports should utilize the template provided by the TLC and reflect their assessment process, results, and actions for continuous improvement

References

NCAAA Handbooks and Standards.
Spady, W. (1994). Outcome-Based Education: Critical Issues and Answers. US: American Association of School Administrators. Retrieved from Eric
https://files.eric.ed.gov/fulltext/ED380910.pdf

List of relevant documents

1. Prince Sultan University's Program Assessment Handbook