Evaluating training outcomes: some reflections on an online and in presence modality
| Author | Paola Nicolini |
| Position | Department of Educational and Training Sciences, Macerata University |
| Pages | 113-122 |
Page 113
According to the objectivist point of view, knowledge consists in correctly conceptualizing and categorizing things in the world and grasping the objective connection among those things and those categories (Lakoff, 1987, p. 163). So there is only one correct possibility to reach this kind of correspondence and only one correct understanding of any topic (Vrasidas, 2000). In this framework the work of learners consists in the achievement of the correspondence between abstract symbols and real world. Evaluation is goal-driven (Jonassen, 1992a) and it can be very similar to a paper and pencil test (Bennet, 1998): trainers ask students to make a activity, then the answers can be compared to a correct model by teachers or students themselves. This is a quite easy way to do an evaluation and also a self-evaluation. There are also software able to provide this kind of assessment, both in presence and in online experiences (Rafaeli & Tractinsky, 1989; 1991; Rafaeli, Barak, Dan-Gur & Toch, 2003).
On the contrary, in a constructivist perspective, the world is mostly created by the human mind (Piaget, 1970) so that knowledge is mainly considered as an interpretive process (Kuhn, 1996). In addition, in socio-constructivist approaches knowledge is considered the result of construction of meaning and negotiation that happens within social exchanges (Bruner, 1990), so that teaching is not just a simple transfer of information, but an active building of data and understanding situated within authentic relationships and activities (Scardamalia & Bereiter, 2002). As there is not one correct understanding and there is not one correct way of solving problem (Vrasidas, 2000, p. 10), the exclusive use of testing is clearly not adequate to individuate this kind ofPage 114learning achievements (Lesh & Doerr, 2003; Sternberg 1997). Constructivist and socio-constructivist teachers allow learners to have an active role along the whole training path and in the evaluation process (Jonassen,1992b). Evaluation of one's own work promotes self-reflexive processes, which is another goal of constructivist learning (Vrasidas, 2000, p. 12). Self- regulation is also enhanced by peer interactions.
According to socio-constructivist theories, we assume learning as an outcome of social interactions, both disagreeing and concerted (Doise & Mugny, 1981; Pontecorvo, 1993; Carugati & Selleri, 2001). In this approach, teaching is mainly regarded as an activity of scaffolding offered to students to facilitate an active and self directed learning (Scardamalia & Bereiter, 2002). It means at least to start with any concept or knowledge is already available for students and to facilitate a better form or reorganization of concepts. The basic role of prior knowledge in training process is evident: trainers have to deal with previous opinions, ideas and judgments of the trainees to promote new understanding. Training can be considered not a simple grow of information, but a real cognitive and affective conceptual change (Mason, 2001; Mason, 2006). The new contents have to be translated in individual competence, what permits to learners an adequate application and creative use of knowledge and expertise (Gardner, 1991). The change can be achieved only on the base of direct experience and a subsequent reflection (Bion, 1961; Knowles, 1986; Arfelli Galli, 1997; Bruscaglioni, 2002).
We followed the socio-constructivist assumptions in different versions of the same course, the Workshop for observing children at school. We now intend to illustrate and discuss this model, to provide a study case.
The Workshop for Observing Children at School is an obligate formative course at the University of Macerata. It is addressed to students that will be teachers in their professional future. Assuming observation as a specific competence required to teachers, the Workshop is finalized to train skills in observation method. In fact teachers are supposed to adopt an expert approach when observing learners at school.
The Workshop consists of a system of progressive proposals, both subjective and collective. The online version of the Workshop is articulated in 8 activities related to specific goals. In the first activity the participants have to write their observation text using the video available online. The video reproduces a real school situation, in which some children are building a tower. The video has a duration of 60'. The goal of the first observation activity is to activate knowledge and competences owned by the students before the study of the textbook. The students are then asked to discuss (within the forum in online modality, in face to face interaction in the in presence lessons) about analogies and differences aroused among the individual observation texts (activity 2). The peers' discussion is finalized to recognize differences, limits and errors of the subjective point of view (Chinn & Brewer, 1993). Moreover while the students do argue their divergent point of view to support their own opinions, they are building a new and stronger structure of ideas (Nussbaum & Novick 1982). At this point there are bases to activate aPage 115negotiation of meanings (Bruner, 1990). In fact, in the third activity the students are asked to negotiate a shared list of indicators for child observation, looking for a possible agreement (Doise & Mugny, 1981; Carugati & Selleri, 2001; Pojaghi, 2000). Then the students have to read the recommended books (activity 4). The understanding of scientific theories is supposed to be facilitated by the naïve theories recognition and activation. Another peer discussion (activity 5) provides the possibility to further revise the list of indicators. In the fifth activity the participants have to write a new observation text. The video is similar to the first; it shows two children collecting a puzzle in an infant school. This activity aims to enable the students to experience observation in the light of the just learned concepts. The participants are then invited to speak about the previous activity within their group in a web forum, expressing an assessment on the Workshop and formulating a self-assessment of their own learning process (activity 6). To conclude the curriculum, the students are requested to send a personal dossier (activity 7) composed by written texts of every activity. Collecting and composing a
personal dossier is a further strategy planned to promote considerations and metacognitive attentiveness. It is a way to support self assessment. The in presence version of the Workshop has the same activities and goals: there are 7 meetings of 2 hours each, along two months time. The only difference is in the third activity, that is replaced with a teacher lesson. The interactions in web forums are substituted with face to face discussions. 3.2. The Samples
The online and in presence version of the Workshop developed during the academic years 2007-2008 have quite the same number of participants (125 subjects online, 117 in presence). In both cases the great majority are women, the course was attended by younger students with respect to the online Workshop. Besides in the online course there are a lot of students already graduated, whereas the Workshop in presence is mainly attended by students with a high school degree. The table below shows the characteristics of the participants (Table 1).
The characteristics of participants Table 1
In presence Workshop
Online Workshop
Number of participants
117
125
Year birth range
1966-1986
1956-1985
High school degree
96
30
University degree
21
95
Full time students
98
30
Workers
19
95
Geographic origin
5 South of Italy
68 South of Italy
110 Centre of Italy
55 Centre of Italy
0 North of Italy
1 North of Italy
2 Foreigners
1 Foreign
To compare the observation text made by every student at the very beginning of the course and the text written in the sixth activity can be a possibility to assess the training efficiency. This kind of analysisPage 116has been carried out by two independent researchers by the means of a list of features, as in the following table (Table 2). Using the above indicators we identified three quality level of observation text:
· Low Quality (LQ): short and not structured texts, without separation between description and interpretation, characterized by the presence of generalizations, deductions without argumentations, use of personal point of view as an absolute one;
· Medium Quality (MQ): longer and more structured texts, with a better separation between interpretative and descriptive data, the point of view more frequently expressed as relative;
· High Quality (HQ): texts which present all or most of the indicators typical of an expert approach.
Examples of observational texts are soon provided.
Indicators to evaluate the qualitative level of observation texts Table 2
Naïve observation text
Expert observation text
Text structure
Short and free text
Long and structured text (titles, paragraphs, bullet points,
tables)
Context
Absence of information about the
focus of attention and the aim of the...
To continue reading
Request your trial