We tested our readiness for November’s External Evaluation Review (EER) last month.
During the first two weeks of June, 12 areas of Unitec were involved in a self-assessment exercise using the same process that the NZQA will use for our EER. This was based on a series of interviews with management, staff, students and other key stakeholders, where they reviewed a range of performance and self-review documentation. The interviews were facilitated by staff from Arahanga, supported by five Unitec staff who were trained as co-facilitators.
The facilitation team have now sent us their feedback for each of the 12 focus areas, as well as an overall report that assesses us against each of the NZQA’s Key Evaluation Questions (KEQs).
So, what were the results?
Debra Robertson-Welsh, Dean Health & Community and Environmental & Animal Science, who is leading our Category 1 project, says that the assessors’ report shows us that there are some areas of excellence. However, in some areas, we’ve slipped from good to marginal since the 2016 EER report.
Debra said: “Inconsistent performance is our biggest shortcoming and if we’re going to regain the NZQA’s high confidence in our educational performance and our self-assessment, we need to show less variability in our results.
“The good news is that the assessors have given us plenty of tips to help us on our way”.
The main changes they’ve told us that we need to make over the next few months are:
- Make sure we know our students
- Make sure we know where they’ve gone to / their destinations
- Make sure we know our data (and how to use it).”
The three most practical things that each programme can do immediately are:
- Record feedback from moderation and discuss at Programme Action Quality Committee
- Review all action plans, Monitor’s recommendations and student feedback as a group
- Develop an integrated action plan, using SMART principles, and review regularly.
Here is a summary of the assessors’ feedback
KEQ 1: How well do students achieve?
- There is considerable variation in course completion rates across Unitec and between student groups.
- The continued disparity between overall course completion rates for priority learners and other students is concerning in light of the importance placed on these students in the current and previous Tertiary Education Strategy.
- It is difficult to interpret data due to different methodologies used. It is not clear from the evidence provided what strategies are in place to achieve the target Unitec has set for qualification completion.
- In line with the findings of the 2016 EER there are examples of good practice at Unitec and high levels of achievement. This is not uniform across the organisation and this review has also highlighted a lack of focus on academic processes, poor practice such as clustering of assessments and slow feedback to learners.
KEQ 2: What is the value of the outcomes for key stakeholders, including students??
- Unitec is commended for the richness of the data it collects, but it is not evident if the data has affected change.
- Analysis of individual Programme Evaluation Plans provides little evidence of programme level actions concerning these outcomes. The focus of PEPs is primarily on trends in enrolment and course completions.
- Where a programme is subject to external accreditation, e.g. Architecture, Social Work and ECE the additional peer review process strengthens stakeholder engagement and provides extra validation of the value of the programme to the relevant industry.
- There are some examples of good practice, e.g. ECE and Social Work where there is a close connection between employer/agency feedback and curriculum development.
- In addition to providing well-qualified graduates into the Auckland and wider (domestic and international workforce) many programmes offer opportunities for students to contribute to community initiatives in the region and overseas.
KEQ 3 How well do programme design and delivery, including learning and assessment activities, match the needs of students and other relevant stakeholders?
- In some areas, most notably the Bachelor of Teaching (Early Childhood), Bachelor of Social Practice, and the Bridgepoint portfolio, programme design, delivery and learning, and assessment activities match, and exceed the needs of students. Some highly motivated and enthusiastic teams provide strong evidence they go above and beyond their duty of care to ensure their programmes are engaging, innovative, flexible and up-to-date.
- There appears to be a 50/50 probability that students and other stakeholders will not have their needs either matched or met in terms of this KEQ.
- Where field-based delivery and practicums characterise teaching and learning these modes contribute to maintaining a high level of programme relevance to stakeholders and communities. The Bachelor of Teaching (ECE), Bachelor of Social Practice and the Bachelor of Engineering Technology exemplify good practice and a clear line of sight for students to progress to employment.
- Most of the highly regulated programmes evaluated produced strong evidence that stakeholder needs are met and that programmes are regularly reviewed and updated.
- Instances were also observed where evaluation and self-assessment to implement improvements for the benefit of students did not appear to be prioritised.
- Engagement with key stakeholders was so patchy and inconsistent across the focus groups that it is difficult to verify either the level or appropriateness of key stakeholder engagement across the focus groups.
- Fair, valid, consistent and appropriate assessment methodology and practice is clearly great in parts at Unitec, but seriously deficient in others.
- Overall, evidence of good assessment practice, useful feedback and purposeful alignment with learning outcomes struggled to reach the 50/50 mark across the focus areas.
- Overall, some very good examples of deliberate alignment with industry and great connection and engagement with stakeholders, including students can be seen in places, but where there are obvious gaps and deficiencies these are serious and impact on educational performance, in some cases to an extreme.
KEQ 4: How effectively are students supported and involved in their learning?
- The evidence provided and interviews with staff and students were insufficient to conclude Unitec students are effectively supported and involved in their learning consistently across the institute.
- The evaluators concluded pockets of excellent support and involvement with knowledgeable, approachable and caring teachers exist. Equally there are significant gaps in student support and an unacceptable number of students are unclear about the content and structure of their course prior to, and in some cases during their enrolment.
- Staff and students are similarly frustrated by centralised enrolment systems. Neither comprehensive, nor timely study information nor advice seems to be consistently provided by sufficiently well-informed Unitec staff external to the programme to adequately assist students to pursue their chosen pathways.
- In the main, students felt their teachers were approachable, caring of their well-being and responsive to their needs.
- Where student evaluations are carried out they appear in most cases to be favourable. However, sample sizes are unknown and they are not mandatory at this point in time.
KEQ 5: How effective are governance and management in supporting educational achievement?
- Unitec has been in a period of change since before the last EER, and it is evident that this has impacted negatively on student success and staff morale. Unitec has now recognised these are important issue to be addressed.
- Staff engagement remains lower than target and very low staff NPS scores were noted in several focus areas.
- Change is starting to happen, and the evaluators noted positive staff and student attitudes to the changes in management that have occurred over the half year. In particular staff appreciated the student and education focus of the current leadership.
- In the focus areas for this IER many new initiatives were being rolled out or planned that should improve educational provision.
- Across the focus areas self-assessment was evident largely through the Programme Evaluation Plan (PEP) reports. Variability was identified in the quality analysis used, the identification and follow through on improvement opportunities and the engagement of staff in the self-assessment process.
- Recording graduate destinations for programmes and using this information to inform discussion of the value of the programme is an area for improvement. Knowledge of graduate destinations was generally poor and not systematically collected in the focus area programmes.
KEQ 6: How effectively are important compliance accountabilities managed?
- Performance is inconsistent in relation to this question.
- Self -assessment is not sufficiently robust or comprehensive to ensure all compliance accountabilities are being identified and addressed. The inconsistency in practices and procedures across Unitec that are not adequately monitored or reported, highlight weaknesses in self-assessment.
There will be more conversation about the self-evaluation process and what we’ve learnt from the IER at Tū Arotaki/Checkpoint on the 17th. There are two sessions: 9am and 11am so click here to register.