LEARNING
TECHNOLOGY
DISSEMINATION
INITIATIVE

Evaluation Studies

Route map: Home => Publications =>Evaluation Studies=>CALculating Success?

CALculating Success?

Joanne Smailes, University of Northumbria at Newcastle
to contents
to next page
to previous page

Also available
as a pdf file.

Abstract

Quantitative subjects form an integral core of the degree programmes offered at Newcastle Business School (NBS). Preparation and delivery of the subject matter assumes that all students have the standard requirement of GCSE Grade C or equivalent in mathematics. Increasingly, however, experience indicates that many students, on entry, exhibit weak numerical aptitude. In an attempt to address this, an additional option unit (elective) has been introduced allowing students the opportunity to update their skills base and develop more confidence in handling numerical problems. Consequently, business modelling staff at NBS have selected this unit to pilot the use of in-house computer aided learning materials as the principal vehicle for the delivery of the syllabus.

This paper evaluates the effectiveness of this software based approach as a teaching method and its subsequent impact on developing expertise. A discussion of how the customisation of the program has created advantageous by-products of administrative data capture and analysis, as well as accountability of students' individual learning needs, is also addressed.

Furthermore, the changing roles of both student and lecturer within a computer aided learning environment are examined, with emphasis placed on the possible formulation of a "best practice" strategy for the future integration of computer aided learning within the mainstream curriculum.

Introduction

At Newcastle Business School (NBS), as with most others, each student must study quantitative methods as an integral core of their degree programme. Increasingly, however, numerous press reports and lecturers' experiences indicate that many students embarking on a higher education course actually have a weak numerical aptitude. This concern has even been expressed with regard to mathematical degrees, for example in October 1994, Manchester and Southampton Universities introduced four year extended mathematics degree courses in order to cope with the dwindling numerical knowledge base of incoming students.

At NBS a similar decline has been observed and there is no doubt that the majority of students would benefit from numeracy skills revision at the beginning of their course of study. In reality however, there are a number of factors preventing this via the traditional lecture / tutorial route:

At present such revision is only available by choice. The University of Northumbria, of which NBS is a faculty, operates a Centre for Writing and Numeracy Development. Students of any discipline can consult staff and the resources of this centre. NBS students can also elect to study a 12 week unit during the second semester entitled basic quantitative techniques for business. Having a personal interest in numeracy problems within business education, it was inevitable that I would be selected to lead this unit. In its present form, being lecture and seminar based, it can only cater for a relatively small number of students. Nonetheless, I believed that this would be a perfect opportunity to consider the ways in which this provision could be expanded and I believed computer aided learning was a possible solution which could perhaps overcome the stumbling blocks mentioned above. Research from previous studies and evidence collected via the optional unit would provide me with the opportunity to prove this.

This paper intends to firstly justify the choice of Computer Aided Learning as well as defining exactly what constitutes business numeracy in NBS. This will be followed by an appraisal of the use of such materials on the basic quantitative techniques for business unit. Finally, in connection to this, general issues relating to the future, which arose during this evaluation will be discussed.

Why Computer Aided Learning ?

The use of computer aided learning (CAL) in science and mathematics is by no means a new idea. One of the first reported instances was the Programmed Logic for Automatic Teaching Operation (PLATO) in 1960 at the University of Illinois. It is important to note, that prior to this, was a facility called 'Skinner's box', a linear teaching machine which perhaps could be regarded as a form of automated instruction. Studies into the use of CAL for mathematics teaching are somewhat limited and in the main tend to be associated with children of school age. Not until the late 1980's and early 1990's did CAL within further and higher education emerge as a prominent field of study.

In Multi Media Computer Assisted Learning (edited by P. Barker), Yeow-Chin Yong (1989) from Ngee Ann Polytechnic directly discusses the considerable value of CAL in the teaching of mathematics and science. This study recognises that to make the most of student teacher interaction, in these areas, class sizes should be ideally around ten but that in reality within higher education class sizes are much larger (as is the case in NBS). Yong offers CAL as a practical solution to support teachers who deal with classes of this size, hence complying with the first of three stumbling blocks. As CAL is a physical rather than a personnel resource, contact between staff and students can be greatly reduced satisfying the second of the three factors mentioned in the introduction. To overcome the third and final factor once again, in terms of resourcing, physical versus personnel plays a major role. Vinegrad (1987) emphasises that CAL can easily provide "a learning by example paradigm" which is especially relevant to areas such as numeracy where expertise is based on the "acquisition of procedural knowledge". Yong states that CAL is an ideal vehicle to consolidate, revise and self assess especially in areas, such as numeracy, where students can develop an inferiority complex. Hence, these studies suggest that CAL indeed helps conquer an inherent fear of an academic specialist.

Whilst conducting the research for this paper, the definition of numeracy typically varied from one study to another. Therefore in the following section as a term of reference for this study its meaning is clarified.

What exactly constitutes basic numeracy in business education ?

On entry to the university, students (excluding mature and overseas) must have at least a grade C or equivalent in GCSE mathematics. Obviously, as business studies students they are not expected to be highly proficient mathematicians and to be competent in all subjects covered by the GCSE syllabus. However, there are certain areas which are assumed in the preparation and delivery of the core NBS business modelling programme. These are:

The aim of the basic quantitative techniques for business unit is to establish the basic numeracy skills which are assumed within the core business modelling units whilst developing confidence in handling numerical data. Accordingly the subjects listed above are defined as "business numeracy" and form the outline syllabus for the unit.

Software "fit for purpose"

Initially, rather than trying to "reinvent the wheel" I looked round for existing CAL materials covering the defined areas of business numeracy which could be adopted for use. As a first port of call, the Teaching and Learning Technology Projects (TLTP) were considered. Two of these projects one of which is led by the Imperial College London and the University of Leeds, the other led by the University of Newcastle, are concerned with remedial mathematics. Over the last couple of years I have attended conferences were both of these projects were demonstrated. In both cases, the software is of the highest calibre, yet in my opinion more geared towards engineering and science. Secondly, I referred to an Internet web page set up by Sheffield Hallam University. This web site (last updated in October 1997) fortuitously reviews all available mathematical software to date. By consulting these extensive reviews again none were deemed suitable in this instance.

Besides, Yong asserted that it is imperative that teachers develop their own in-house materials in order for general enthusiasm, optimism and momentum to be gathered so as to cultivate CAL within an organisation. I strongly subscribe to this view and consequently created a business numeracy package using 'Asymetrix Multimedia Toolbook' which was an authoring tool. This package (present working title of which is CRUNCH) consists of a diagnostic check, revision modules, informative assessments plus an associated management information system.

In 1998 these in-house computer aided learning materials were complete enough to pilot on the basic quantitative techniques for business option unit. The general focus of this pilot was to assess whether numeracy standards were improved by this CAL medium, and to examine whether information gathered by the management information system is beneficial for administration purposes. Additionally, I was particularly interested in pursuing the following questions:

I would also have liked to confirm whether or not students have problems admitting that their numerical ability is weak. However, because the pilot group are there through self selection, does this not suggest that this problem can not be adequately addressed without bias? It is this concern, that at present, unfortunately, can only be flagged for future investigation.

General evaluation

A variety of data collection methods were employed including questionnaires, semi-structured interviews and observation.

The pilot group was composed of nineteen students from an assortment of first and second year courses within NBS. Nine of these students were categorised as being mature and three overseas (of these two were also mature) hence giving a diversity of backgrounds. It should be noted that all of the participants have studied at least one of the core quantitative methods courses in NBS.

In the first session of the course the students undertook a diagnostic assessment. This consisted of twenty questions each designed to relate directly to elements of the mathematical skills covered by the syllabus. The management information system was structured to collect some basic background information (e.g. age, GCSE grade) along with data pertaining to question responses entered. Once students completed the questions to their own satisfaction, they simply clicked a button which automatically marked their assessment, then offered some general feedback and issued them with their own unique study plan. This study plan simply consisted of a list of the modules they were expected to work through in the succeeding sessions. In the aim of increasing self-confidence in number handling calculator use throughout the course was restricted. As an additional aside: mental arithmetic is a useful skill particularly as psychometric testing of numerical skills is increasingly used by employers in recruitment and selection; a number of these tests do not allow the use of a calculator. (Example: Graduate Managerial Assessment (GMA) - Numerical).

As this assessment was purely a diagnostic check, numerical results were not formally collected but student feedback was accumulated via questionnaire. On the whole, 69% of the students considered that their performance on the assessment was as expected, 18% thought that they had done better than expected.

When reflecting on their general opinion of the assessment's difficulty, 63% believed it to be achievable while the remainder found it to be a challenge. This provides reassurances that the diagnostic and informative assessments have been pitched at an appropriate level.

Data collected by the management information system did permit an analysis of group performance. Interestingly, some particular areas of weakness emerged, these being the questions on the order of operator manipulation, multiplication and division of fractions and algebra. This information proved very useful in preparing extra support material.

In subsequent sessions, I became a bit of a spare part. I just wandered round whilst the group studiously worked through modules listed on their individual learning plans. Many informal comments were made during these sessions all of which were very positive and attendance levels were good. The students, who knew this was the first time CAL had been used, were very keen to contribute and reported any glitches and, on their own initiative, made useful suggestions on how the materials could be adapted.

The group was asked to estimate how much time had been spent on the individual modules and on average this was estimated as being between 20 and 25 minutes. In the case of fractions, one of the highlighted areas of weakness, generally a little more time was required, an average of 30 minutes. Encouragingly, around a quarter of the students invested significant amounts of time on the modules outside of the formal sessions. The commitment of the students and the employment of this software based approach, resulted in the syllabus being completed in two thirds of the overall allotted timetable provision. At this point, students were given an informative assessment its format being identical to that of the diagnostic check. On this assessment, all substantially improved their performance and it is predicted that all will perform well in the terminal examination in June. However, once again the management information system indicated algebra as a continuing problem (the other two now being virtually eliminated). Consequently, the final third of the unit will be used to concentrate on this area in more detail, albeit by traditional means. On the whole, the evidence gathered from this pilot along with previous studies has provided sufficient corroboration that computer aided learning materials can improve numeracy skills whilst reducing the need for staff student contact.

The paper will now move on to contemplate the questions posed earlier on student and lecturer perceptions of a CAL environment.

The student and the CAL environment

Specific reference to student reaction to CAL for basic numeracy is made by Clare (1991). He provides an honest objective view in the use of in-house materials at the Matthew Boulton College of Further Education (project started in 1986) with varying student intakes. Overall, the reaction to and the success of the materials was fairly mixed "...student reaction is not as positive as we have experienced before...". Nevertheless, it was noted that, in the case of algebra, even though students' resistance had not be eliminated teachers did express it had been reduced and in their experience students were happier than previous classes had been. In contrast Tan, Yong and Tan (1987) carried out a survey where feedback from both staff and students to CAL materials was very positive, particularly with regard to an eagerness to attend sessions. My experiences have been more in line with those of Tan, Yong and Tan.

As mentioned earlier, most of the time I felt that my presence was surplus to requirements. So, via questionnaire, followed up by interviews, students were asked whether they felt it was necessary to have a subject specialist in attendance. Opinions on this were split pretty much fifty-fifty. The proposition was introduced that if the numeracy sessions were unstaffed, would they still have turned up, seventy-five percent replied yes. Consequently this was contradicted, with around 60% of the group stating CAL sessions should be formally timetabled with a lecturer present. At interview, this was expanded upon by a couple of students who asserted staff presence was recommended for discipline. To illustrate, they quoted an instance where a workbook had been issued on another unit. Admissions surfaced that a great deal of the work had not been started, let alone completed, even though it was understood that assessments were to take place at the end of the semester.

Moving on to points related to independent learning, the question "If CAL was available as an optional extra to the main course of study would you make use of it?" was posed. An overwhelming majority honestly declared that they would only do so at revision time with the rest saying that they would perhaps use it occasionally to back up their study. Assuringly however, no one stated that they would not bother with CAL, but again a desire for a little more enforced discipline was expressed.

Following on, the idea of CAL materials, for instance, replacing lectures was debated. Even though general opinion affirmed that the CAL materials were excellent and had without any doubt improved their understanding, the preference was for lectures with the addition of staffed CAL sessions. Justification being that at times there were still certain things requiring clarification; in those situations a lecturer was essential for consultation.

"Computers cannot anticipate exactly what you don't understand."

During interviews, it was stressed that these staff reliant attitudes were mainly based on their feelings towards the new environment of university life, particularly the social aspects.

"It can be quite a culture shock"

So, in terms of academic learning they simply desired more direction, for example registers to encourage attendance.

The lecturer and the CAL environment.

Past studies have revealed that, from a lecturer's perspective, a common fear associated with using CAL is its potential threat to their jobs. For instance Clare assures us that the CAL materials at Matthew Boulton College, were not complete courses, and that skills such as problem solving were not included, therefore it would not be envisaged that teachers would be replaced by such materials. Yong also stressed that CAL should be used to "support not threaten" teachers. Obviously, as I have a personal interest in the development of CAL, my views could be seen as optimistic and biased.

In order to gain a more objective view from my colleagues, a staff seminar reporting the findings above, followed by an open discussion, was held to assimilate views on CAL's role in quantitative subjects. The outcomes were incredibly heartening. Over the last couple of years lecture sizes have in most cases tripled and now a typical lecture is presented to 200+ students. Smaller group sizes (approximately 25) are only encountered during workshop sessions where emphasis is placed on spreadsheet use. Against this backdrop, all members, including myself, assigned to the view, that CAL is a very important support tool of the future. Student views regarding the timetabling of CAL sessions were passed on and even though there was great empathy, all lecturers recognise that an independent learning culture within universities is now common place. Staff agreed that students require guidance to ease them into this culture at induction (programmes of this nature are being developed) but as subject specialists we should concentrate on strategies for using CAL, not only as an additional student resource but also as a lecturer support tool. Subsequent dialogue revolved around how the numeracy materials can be incorporated into the mainstream curriculum, plus a confident expectation of additional materials to be developed. Many decisions were made which I will be introducing in the conclusion.

Conclusions and future developments

The cultivation of the CRUNCH materials referred to in this paper has been a long process. Primarily, this was due to myself being the sole developer along with a number of technical issues encountered with network compatibility. During 1997 however, NBS kindly funded a placement student dedicated to this project, the result of which instigated a speedier completion. The pilot group was small and by nature of self selection, one important factor relating to numeracy (i.e. student fear of admission) has had to be excluded from this research. There were a sufficient number however, to be representative of general issues and root out any technical errors. The students have expressed gratitude to the use of CAL and all have reported significant benefits in terms of both ability and confidence. From the student perspective, there was a little reluctance towards CAL as an independent learning resource, but I believe that with efficient coaching in "learning to learn" this will be overcome. The expectation of staff resistance to CAL was unfounded and comments were exclusively directed towards expansion. As these have radical implications for the near future they will now be elaborated upon. To start with, word on the success of the pilot had been passed on to higher management. They have advocated a larger trial in September 1998, on the BA Business Administration course. The intention is to timetable the diagnostic checks into the induction program and introduce the CRUNCH materials as an independent study resource.

As a "carrot", students are informed that three or four weeks into the main quantitative course a follow up assessment will be issued (the decision is yet to be established as to whether this will be formative or informative). A full implementation is being planned for the following year.

One of the considerable dilemmas recently encountered, has been the provision of both informative and formative assessment, other than by a terminal examination, for large student numbers. As the management information system and assessments have proven successful and easily adaptable for data capture, it was agreed that a secondary system of computerised assessments should be adopted in the mainstream quantitative subject areas. Ergo, equipping both staff and students with scope to review performance at regular intervals. Again this will be piloted during the next academic year.

Finally, the Dearing Report on Higher Education identified four "key skills" of which numeracy is one. In 1997 the University of Northumbria's Academic Board approved the implementation, across all courses, of a Key Skill Model over a period of three years. Subsequently a representative panel, across all university facilities was constructed to consider ways in which this can be achieved. An advisor to this panel attended the staff seminar whereupon it was suggested that these materials may be put forward as a possible delivery method for the numeracy key skill.

In its entirety results have been extraordinarily favourable, and personally I am thrilled that CAL has been highlighted as an issue for wide ranging expansion, with future possibilities for research. To sum up then, (pun intended) I envisage a CALculated success!

References:

Cann C (1997) Key skills at UNN: an introduction, Red Guides, University of Northumbria.

Clare P (1991) Basic Maths Project, A CBL approach to basic maths and numeracy in FE, Learning Resources Journal, Vol. 7 Part 2 p26-30

Patel K (1993) Autumn Launch for four-year maths degrees, The Times Higher, April 23 1993

Tan BT, Yong YC and Tan HG (1987) Validation of the effectiveness of CAL as a Form of Remediation , Proceedings of Educomp' 87 Conference, Kuala Lumpar, Malaysia

Utley A (1990) North opts for extended maths, Times Higher Education Supplement, December 28 1990

Vinegrad MD (1987) Learning by Example: a CBT approach, Program. Learn. Educ. Tech. Vol. 21 Part 3 p218-222

Yong Yeow-Chin (1989) CAL for mathematics and science teaching, Chapter 4 Multi-Media Computer Assisted Learning Edited by Philip Barker, Kogan Page

Non Authored References:

Sheffield Hallam Website: http://www.shu.ac.uk/services/lti/mcr/intro.html (Last update:October 1997)

TLTP Catalogue - Phase 1 - Spring 1995

TLTP Catalogue - Phase 2 - Spring 1996

to previous page to contents to next page

To contact the maintainers - mail (ltdi@icbl.hw.ac.uk)
HTML by Phil Barker
© All rights reserved LTDI.
Last modified: 29 September 1998.