6.1. The distance learning provider
6.2. The student
6.3. The employer
6.4. The financial sponsor
6.5. Relevance to the workplace
In the distance education process, there are typically four players:
1. The teaching institution
2. The student
3. The company or employing organisation
4. The financial sponsor (which could be 2 or 3 above, or an external organisation)
Clearly, different views on effectiveness and cost-effectiveness are taken by each of these players. As Calder (1994) indicates:
'One of the main problems in trying to determine the effectiveness of anything is that you need to be clear not only about what it is you want to achieve, but also what relative weightings you would attach to each of those goals.... In looking at the effectiveness of different types of education and training provision in relation to learning, the problem is compounded because of the number of stakeholders involved, and because of the range and often contradictory nature of the goals which they hold for a particular course or training programme.' (10)
The institutional provider of distance education often considers effectiveness as meaning learning effectiveness i.e. the success of the instructional programme in achieving its objectives or pre-determined outcomes.
From the point of view of the teaching institution this is invariably measured in terms of performance indicators such as assessment of student work through continuous assessment and examinations and in the form of course evaluation conducted by in-course and end-of-course questionnaires. Such course evaluation is commonplace. Rowntree defines it thus:
'Evaluation is the means whereby we systematically collect and analyse information about the results of the students' encounters with a learning experience. (11)
In a review of the research evidence currently available on learning effectiveness of open and distance learning, Calder (1994) states that the criteria most commonly used by distance learning providers in assessing the effectiveness of their training programmes are:
· extent to which access to courses is facilitated
· extent to which study skills are acquired
· the use of deep (as opposed to surface) learning by students
· extent to which students operate as self-directed learner
· course retention rates course pass rates
A typical checklist for assessing learning effectiveness is shown below:
· Will the learners Understand what is expected of them?
· Will the learners have difficulty achieving any of the listed objectives?
· How long would they be expected to take over each section?
· Does the material seem pitched at the right level of difficulty and interest?
· Do the examples, analogies and case studies seem relevant and illuminating?
· Are any sections likely to cause problems (for example, for a different culture)?
· Are new terms adequately explained?
· Are there sufficient activities and self-tests?
· Are the activities worthwhile and practicable?
· Is there appropriate assignment and follow-up activity?
Evaluation before, during and after training should answer many of the above questions and the collection of information via student and/or employer questionnaires is a common technique. However the study identified examples of monitoring of student opinion and performance which stopped short of data analysis and evaluation with the consequence that course modifications were not made and recommendations not acted upon. To be effective and useful, course evaluation should be more than a 'public relations' exercise. Indeed, the process of evaluation and the modification of material, especially for rapidly changing bodies of knowledge like engineering and technology, should be a key and inviolate part of a distance learning provider's service.
A common indicator of success frequently used by distance learning providers is a measure of student drop-out from courses. The use of this indicator is fraught with problems and ambiguities and more mature distance learning providers believe that there are more effective measures which can be used to describe, for example, completion and non-completion rates and which take into account students who postpone and then re-enter the system. The Open University of Sri Lanka, as an example, would be advised to consider alternative methods for the measurement of student participation and satisfaction.
For the student, success in achieving the qualification is frequently seen as the prime measure of learning effectiveness, thus laying great store by the providing institution's examination and assessment procedures.
Central to the model of a learning process is the interaction between the student, trainer and the learning materials. However the primary stakeholders are the learners. This learning effectiveness should not be reduced to goals or outcomes as frequently considered pre-eminent by students; the quality of the learning process must be recognised as a crucial factor in the assessment of learning effectiveness.
For the employing organisation effectiveness is more likely to be viewed in terms of 'problem-solving' and acquisition of skills.
Within companies, success in 'passing' the course is less of a concern, training often being undertaken following an employee's appointment or promotion; so to a certain extent 'success' has already been achieved. Training needs often arise through appraisal of employees and again there is an expectation of 'success' in the training process. Learning effectiveness is assessed primarily in terms of improved work processes or safe and correct working practices.
A 1989 report from the Department of Employment, 'How to Profit from Open Learning Company Evidence', indicates that 94% of the 50 firms involved in the survey regarded their open learning programmes as successful.
Those companies that used open learning identified a number of benefits, including:
· higher pass rates than before
· line manager satisfaction
· better retention of information
· a better record of promotion
· an increase in the number of employees working toward vocational qualifications.
However there was little systematic evaluation of the programmes involved - frequently because even conventional training was not evaluated, thus providing no basis for comparison.
For the sponsor, effectiveness is likely to be measured in terms of the extent to which their own (often wider) objectives have been achieved.
Few of the indicators used to assess effectiveness are concerned with the benefits to the employer. Indeed, there is little evidence of distance education producers analysing in depth the skills necessary to complete tasks in the workplace in order to design programmes to satisfy these needs - despite the fact that employers are frequently the sponsors of training and their primary objective is that the training should impact positively on the student's work performance. There is clearly considerable scope for an in-depth study of the specific benefits of training on task performance.
To date very little work has been done in terms of measuring effectiveness and, by extension, the cost-effectiveness of training.
