Monitoring learner experience
Quick links:
Context and background to sector-leading practice
Itec received a contract growth of 100% in August 2011. Learner experience and outcomes during this growth period were of utmost importance and in line with the Learner Involvement Strategy, hence the need for a live reporting system that gave us the ability to observe this centrally – direct from the learner. Sustaining a strong learner experience was identified as key to increasing overall learner satisfaction, retention, progression and outcomes. Also, with the increase in learner numbers and subsequent expansion into our supply chain there would be the need for any new system to be delivered throughout the supply chain.
A system was required that:
a. could give Itec up to the minute access to data about the learner experience
b. could be easily adopted consistently across the supply chain
c. was not limited by geography or accessibility issues
d. could be used across all programmes
e. allowed us to interrogate learner data centrally to explore patterns, react to trends and identify groups or individuals that were not receiving the best possible experience.
f. be compatable with, or easily compared with the Learner voice process.
It was agreed that a transparent, and efficient and performance focussed approach was a necessity. The concept of ‘IDRIS’ – ITEC Data Responsiveness Internal Survey System was developed which allowed a web based performance system to be developed, breaking down access barriers and addressing time and resource limitations. The system provides live satisfaction and response rates performance for all programmes.
The system now encompasses current partners to ensure standardisation and a live holistic view and measurement of satisfaction rates for ITEC can be seen.
Description of nature of strategy or activity
1. IDRIS is a multi-platform learner experience system that provides live data to monitor the quality of learning.
2. At the front end is a user-friendly software tool that instigates genuine feedback from learners through PC, laptop or mobile device.
3. Learners are asked to input their feedback through IDRIS at three points in their learner journey (induction, mid-term and exit) and are supported by regular learner fora.
4. The tool allows data to be amalgamated to observe trends as well as allowing learners to feedback specific comments relating to their experience.
5. The data is reviewed and discussed at every senior management team meeting and action is taken where a need is identified.
6. IDRIS is also available in Welsh. IDRIS has also been delivered a level of compatabilty with the Learner Voice process in terms of questions and categories. This enables ITEC and its subcontractor partners to benchmark against each other and across the entire Welsh network.
What impact has this work had on provision and learners’ standards?
Main areas for improvement identified across in-house IDRIS evaluations:
- Quantity and Quality of Equipment / Resources available in centres
- Information provided on options available after programme completion
- Feedback to learners on views given
- Ensuring training is relevant, or that learners understand the purpose of certain topics within their learning
Changes made by ITEC since the development of IDRIS – Provision wide
- IDRIS live – all provision inclusive of partners
- Additional roles – Transition Co-ordinator
- Building alterations – Cwmbran / Bridgend
- New IT equipment for all centres
- LSC – Learner Storage centres
- New Engagement curriculum being piloted, inclusive of ILP packages and blended learning
- Tutor recruitments – further use of micro teach
- Improved Learner journey documentation and process, including induction and counselling ‘video’
- Partner development days and sharing of best practice
Changes made by ITEC since the development of IDRIS – Local
- Trial of learner representatives (Llwynypia)
- Hours of attendance to suit travel arrangements
- Breakfast clubs and coffee mornings
- Sports events and cultural visits
- More Guest speakers
- Change of structure to prevent overcrowding
- Welsh ‘days’, ESDGC days
- Workshop development
- Televisions and equipment
- Charity walks and events
Impact Assessment
IDRIS is aligned to the Learner Voice (LV) questions, therefore allowing us to measure / benchmark against the Learner voice results. The 2013 LV results showed that although ITEC received good results, the percentage of very good ratings achieved was below the sector average. Impact Assessment has shown that the identification of improvements required via IDRIS and the subsequent implementation of changes have had a positive impact on the provision of ITEC and the standards our learners achieve, raising our Very good rating profile:
Provision
Using the results from IDRIS over a 5month period October 2013 to Jan 2014 – the following increase in % of learners giving a rating of ‘Very Good’ (VG) has been measured:-
Main areas for improvement identified across in-house IDRIS evaluations and increase in % of VG ratings given:
- Quantity and Quality of Equipment / Resources available in centres – 19% increase
- Information provided on options available after programme completion – 10.5% increase
- Feedback to learners on views given – 8% increase
- Ensuring training is relevant, or that learners understand the purpose of certain topics within their learning – 28.4% increase
Further, when reviewing all LV / IDRIS main topic ares., the following positive increase in % Very Good ratings have been measured:-
- Information and Advice – 12% increase
- Support (Extra Learning Support) – 24.4% increase
- Support (personal issues) – 7.6% increase
- Responsiveness – 1.2% increase
- Health and Wellbeing – 3.3% increase
- Teaching and Training overall – 25% increase
- Teaching and Training (one on one time) – 36.2% increase
- Teaching and Training (listening) – 3.3% increase
- Overall rating of provision – 28.6% increase
- Met expectations – 23% increase
Standards
- 594 learners have completed evaluations on the IDRIS site, 129 of these have terminated. The PPR for this cohort of learners is 72.5% v’s 61% for the cohort as a whole.
- ASR of those terminated Learners who have been involved in ITECs Responsiveness strategy and completed at least 1 sitting of IDRIS Learner Evaluation:-
- Engagement – 90.9% (v’s 70.5% as whole)
- Traineeship Lv1 – 69.8% (v’s 56% as a whole )
- Steps- 57.8% (v’s 62.4% as a whole)
- ASR ESW Communications of those terminated Learners who have been involved in ITECs Responsiveness strategy and completed at least 1 sitting of IDRIS Learner Evaluation:-
- Engagement – 100% (v’s 67% as whole)
- Traineeship Lv1 – 72.7% (v’s 67.5% as a whole )
- Steps- 63.2% (v’s 66.4% as a whole)
- Transition co-ordinator role – as a result of the support / guidance offered by this new role, the attendance rate for this cohort of learners in Llwynypia has increased from 46% to 89%. (also see support measurements above)
Way forward:
Plans are in place to further extend and development the IDRIS system:-
- Extend the use of IDRIS to the Employer and Stakeholder Evaluation systems
- Extend the use of IDRIS mobile ‘phone questionnaire to hard to reach Neets provision and other non-Welsh Government provision
- Extend feedback systems to be inclusive of further social media eg facbook and text facility
- Ongoing Trend analysis and bench marking data (sufficiency of data)
- Comparison and benchmark against new Learner voice results
How have you shared your good practice?
The ITEC IDRIS system has been developed further to be used externally by all Apprenticeship subcontractors / Partners.
ITEC work with nine main partners. Our prime contractor module is centred on three core factors:- performance, quality and development. We work with our partner’s to ensure that the service we offer our customers is of the highest standard.
Our current partner base:
MVRRS – South Wales: Health and Social Care
LMJ – South Wales: Health and Social Care, Management
Plato – North Wales: Health and Social Care, Retail, Administration, Management
ELT – South Wales: Management
PTAS – Wales national: Rail
Rossett – Wales National: Health and Social Care, Childcare
ISA – South Wales: Hair and Beauty
ABM Health Board – South wales: Administration
The system was demonstrated to all partners via the ITEC Sharing practice and development days, with ITEC staff being on hand to support all subcontractors as they pilot the system in their own organisations. The system mirrors both the Learner Voice and internal evaluations which enables the subcontractor partners to benchmark their learner satisfaction rates not just against themselves but also others across the entire Welsh network.
The live data is broken down per subcontractor / partner in the form of satisfaction charts per topic, response rates and monthly planning lists to help parthers forward plan data capture. Learner comments can be accessed via the report which gives further insight into learner satisfaction rates. These reports can be broken down per programme, by age / gender / ethnicity, and by dates, to allow comparisons and interagation of data. Responsiveness data can be accessed via the ITEC ‘live’ Partner Portal Intranet system.