Measuring Success in the Learning and Skills Sector

NIACE endorses the belief that “it is important to recognise and celebrate learners’ success” and agrees that public recognition of achievements in the Learning and Skills Sector is a worthwhile goal.

NIACE agrees that there is a need for greater fairness, comprehensiveness, coherence and consistency of approach to evaluating achievement. Uniformity is unlikely to be either desirable or achievable.

NIACE agrees that it will be important to ensure that ‘success’ is defined and measured in ways that are valid and meaningful for learners themselves first and foremost, for employers and for learning and skills providers.

NIACE endorses the approach adopted - which seeks to develop and implement a set of measures which, taken together, can make a significant contribution to the recognition of learner success and provider performance.

The development of valid and reliable measures of success and reporting arrangements which can in due course provide ‘benchmarking’ information will assist providers in making realistic self-assessment judgements.

NIACE applauds the consideration to be given to providers’ missions and purposes in offering a particular range of provision as well as their effectiveness in addressing diversity of learning and support needs.

NIACE agrees that it is not possible to identify one single formula for calculating the value added to learners’ achievements as a result of participating in learning. There is no single valid method for evaluating all forms of attainment. Different types of progress in differing contexts need to be evaluated in a range of appropriate ways. Not all worthwhile learning gain can be quantified.

NIACE recognises the importance of qualifications to many learners. The achievement of qualifications is an important measure of learner and provider success and also indicates whether providers are ensuring that learners are placed on an appropriate, properly resourced learning programme, with relevant support.

NIACE agrees that the use of ‘success rates’ as described should in principle be extended to work-based learning, and eventually to providers of all externally accredited programmes.

Aggregated information about successful achievement of qualifications at different levels will be useful. Taking account of the curriculum profile will be helpful in ensuring that ‘like for like’ comparisons can be made. The inclusion of information about targeted and/or participating learners would also aide understanding.

NIACE supports the proposal that a measure of success be developed for individual learners pursing more than one qualification, based on a national credit framework. As the priority is to develop a unit based credit framework for adults, it is not clear when this would apply to young people, or how non-qualification-bearing learning programmes accessed by younger learners, alongside qualifications, will be accommodated.

Achievement of qualifications and other awards, and of units of credit where relevant, are important indicators of success in learning, and effectiveness in providing equality of opportunity. However there are other forms of achievement that should be recognised. The Staged Process identified in the context of the current LSC RARPA action research project will be relevant in many learning situations in that it takes account of planned and unanticipated learning outcomes and other outcomes of learning.

It is important that definitions of what is meant by the terms ‘added value’ and ‘distance travelled’ are clear and that the terms are used and applied in a consistent way. The meaning of the two terms and the relationship between them is, however, ambiguous in places within the document. ‘Added value’ seems to refer to the additional learning which has been achieved or acquired during the learning programme by an individual learner. The processes described (using numerical data to represent GCSE grades and particular graded outcomes at level 3) appear to provide a measure of both absolute and relative ‘value added’.

The ability to measure ‘value added’ (or not) at an individual learner level is in turn seen as contributing (when aggregated) to an evaluation or measurement of provider performance, described in this document as ‘distance travelled’. Judgements can then be made as to the extent to which providers have added value (or not) in that groups of young adult learners have achieved or exceeded the anticipated aggregated qualification outcomes. This approach is limited in so far as it does not take account of other outcomes of learning.

NIACE is aware that the use of predicted grades based on a numerical calculation has been found in the FE sector to be a useful tool in motivating 16 - 18 year old learners, by raising aspirations and/or helping learners to be more aware of their abilities and potential; and can contribute to improved management of teaching and learning in that it generates data which can contribute to exploration of factors which may account for unexpected differences in performance on the part of comparable learners.

As is recognised, such a method is not directly transferable to the adult learning context, as adults’ prior attainment and individual starting points vary considerably.

However, for adults pursuing qualifications programmes, NIACE recognises the benefits in identifying data on socio-economic background, age, gender and ethnicity in order to contextualise achievement rates and to facilitate effective benchmarking. The difficulties associated with the use of postcodes to generate proxy number values that describe prior attainment do not negate the value and importance of colleges and other providers using them to measure their performance against institutions serving similar communities in supporting learners (both young people and adults) gaining qualifications. Post codes are a blunt instrument with which to measure socio-economic status and deprivation - but, given the cost and time that would be involved in collecting and analysing data from other geographic mapping tools, they provide a quick, straightforward and immediately available solution. Until and unless all LSC providers can use alternative data, postcodes remain an important indicator and not to use them would be unwelcome.

The challenge will be for the LSC to combine the use of postcodes as indicators with other data highlighting educational disadvantage to support providers with improved MIS for benchmarking progress for adults on qualification-bearing courses. This will not be easy.

The use of literacy and numeracy tools at the point of entry is seen by the Adult Learning Inspectorate in their recent (October 2003) Survey Report on Initial Assessment as not ‘fit for purpose’. Testing adults at the point of entry, particularly those who are reluctant to participate, is likely to be alienating and potentially counter-productive in many cases.

Other indicators, such as attributing a number based on length of time since last engaging in learning, are likely to be fraught with difficulties related to definitions and interpretation, also validation - as would be an approach which sought to allocate a number based on prior qualifications not within the current NQF.

Eligibility for fee remission may also be suggested as an indicator from which a numerical value can be derived. At present there is no nationally applicable fee remission policy for the whole Sector. Even if there were, individual circumstances change - would there have to be a revising of the start point numerical allocation if a person’s eligibility (for example, related to receipt of a particular benefit) changed.

If a group of such indicators were to be combined and weighted in some way, clear and definitive rules of combination would be required. The process would be bureaucratic and costly, initially and where adjustments were needed (as in ‘eligibility for remitted fees’ example).

Many adults engage in part-time and often short learning programmes, in some cases sequentially, in other cases in parallel with other programmes including non-accredited programmes. If a process of calculating a numerical starting point were to be required for each learning episode, it would be necessary to take account progressively of any externally accredited achievements. The process would inevitably be bureaucratic and the burden on learners and providers would be considerable.

Given the lack of valid relationships between GCSE performance at 16 and non-graded NQF qualifications or awards outside of the NQF it is hard to see what purpose a ‘starting point’ numerical proxy for attainment at the start of a learning programme would serve. For many qualifications/forms of accreditation sought by adults, ‘terminal’ numerical proxies are not currently available. What numerical proxy for attainment at the end of the programme would then be used, and how would such proxies be arrived at?

NIACE supports the approach currently being taken in relation to non-accredited adult learning. The RARPA Staged Process has the potential to improve the quality of the learning experience for learners in a range of contexts. It can provide robust evidence of individual progress and achievement. It can also support providers in improving the management of teaching and learning processes, in that the outcomes for groups of learners can be examined as part of internal quality assurance processes. It cannot however provide valid aggregated data at provider level to enable comparisons between providers.

NIACE recommends that the RARPA approach be adopted as a means to recognised and record progress and achievement in all adult learning contexts, and alongside the ‘value added’ processes suggested for 16-18 year olds, to take account of achievements in additional to qualifications.

NIACE supports the collection and analysis of data and the use of locally negotiated targets to evaluate the extent to which providers are successfully addressing the learning needs of adults from all sections of the communities they serve. The principle of aiming for a common framework for disaggregation is supported.

NIACE is however concerned that providers with significant numbers of part time learners (small providers from the voluntary sector in the ACL context, or small work-based providers) will struggle to meet the data management requirements.

Learners’ views are an important source of evaluative information in the context of evaluating provider performance and the success of the Sector as a whole. Some learners may resent being asked to complete standard questionnaires at regular intervals. Sample sizes for national surveys should be properly representative in that they represent comparable proportions of the overall numbers of learners in different parts of the Sector.

NIACE endorses the approach whereby providers are required to demonstrate that they seek learner feedback in a variety of ways and use this to help them improve provision and services, as obtains currently in the contexts of inspection and performance review.

The difficulties in collecting such data in FE are rightly acknowledged in the document. The burden on small providers and/or those with large numbers of part time learners could be considerable, were they to be made responsible for providing comprehensive data. The proposal to seek other ways to obtain such information, possibly on a sampling basis, is welcomed. Again, sample sizes and the range of learners participating need to be representative. This will be challenging, given the range of provision across the Sector.

However the information is obtained, it should capture information about a range of types of impact that learning has had, as well as progression to further learning or employment. For example, individuals’ learning achievements may impact positively on families and communities, and in the workplace. This will be necessary if as stated a longer-term aim is to inform judgements about impact (para 67). Given the government’s priorities it will be important to capture evidence that current policies are effective in generating a range of types of impact at several levels.

The principle of seeking information about staff experience and qualifications, skills and competence, and the effectiveness of continuing professional development is endorsed. What is at issue is the best way to do this.

Teaching staff are critical in relation to supporting learning and achievement. Many other staff also make an important contribution, in ACL, FE and work-based learning. A range of qualifications relating to a variety of roles and responsibilities should be considered, and in addition other ways of evaluating capacity, given that many current staff may not have teaching or other relevant qualifications (as these were not previously required in some parts of the Sector) but nevertheless contribute to provider capacity.

The principle of seeking reliable ways of measuring value for money for parts of the Sector and the Sector as a whole is endorsed. However the lack of a common funding base suggests that it will be some time before this is possible. There is a need for sensitivity to the range of missions, strategic priorities and target learners.

A successful quest for valid measures of value for money would enable comparison of relative effectiveness and efficiency, when combined for example with inspection grades and performance review ratings, across the Sector. It would also inform inspection judgements (the CIF makes reference to value for money in relation to judgements about Leadership and Management.)

 The range of measures used should be as limited as is compatible with being sufficiently comprehensive to give a fair and accurate picture. New measures should refine or replace previous measures.

The principle of selecting specific and appropriate measures or specified groups of measures to make useful judgements about the performance of individual providers, and/ or where appropriate to compare performance of different types/groups of providers, is endorsed.

NIACE promotes the interests of learners. In all parts of the Sector measures relating to learners should carry the greatest weight.