Peer Assessment - TU Delft Medewerkers

Peer Assessment
Measuring & Monitoring Team Performances
Ir. Vincent Brugemann and Robert-Vincent de Koning Bsc
Delft
University of
Technology
Challenge the future
Presentation for TBM staff
• About Peer assessment
• Peer Feedback/Peer Review
• Peer Evaluation (PE)
• How to use peer assessment?
• Peer Feedback/Peer Review
• Via Blackboard
• Peer Evaluation (PE)
• Scorion
2
Context
Faculty of Aerospace
Engineering
• 2000+ students in BSc and MSc
• In BSc much use is made of project based
learning (approx. 35 ECTS)
• Students work in groups of 8-10 on a
variety of projects
3
Problem: Assessment
• There is a large need amongst lecturers for
assistance in assessing individual
performance in a group working
environment
• Major problem areas:
•
•
•
•
Who did what?
What is the size of the individual contribution?
What is the quality of that contribution?
In what way did a person contribute to the group
process?
4
What is Peer Assessment?
• Peer Assessment is an umbrella term for a number of different
teaching methods in which peers assess each other’s products
and/or behaviour.
• Peer Assessment is always formative (diagnostic). The aim of Peer
Assessment is to learn from feedback; there are no summative
consequences. In other words, assessments by peers will not
result in grades.
5
When should Peer Assessment be
used?
• Peer Assessment is a useful way to get students to
collaborate intensively on the subject you want them to
learn.
The students:
1. Learn to assess each other using assessment criteria
2. Analyse the subject matter more thoroughly by applying
the criteria
3. Learn to give and receive constructive feedback
4. Learn to take responsibility
5. Appreciate being taken seriously
6
Two forms of Peer Assessment
• We distinguish two different forms of Peer Assessment so that
you can choose the appropriate instrument for your specific
teaching situation.
• The forms of Peer Assessment applied at TU Delft are:
• Peer Feedback/Peer Review
• Peer Evaluation (PE)
7
PEER FEEDBACK/PEER REVIEW
8
Peer Feedback/Peer Review
• Students provide commentary on each other’s products.
• Similar to the Peer Review used in the scientific world to assess
scientific articles.
• In an educational environment it can also be used to describe
assessments of products (which may include scientific articles).
9
Working with
Peer Feedback/ Peer Review
• If you want your students to assess each other’s products you can
use the Self and Peer Assessment tool in Blackboard
• The students submit their products and Blackboard then
distributes them among their peers so that they can assess each
other’s work.
10
PEER EVALUATION (PE)
11
Peer Evaluation (PE)
• Peer Evaluation is a form of Peer Assessment in which
students provide commentary on each other’s behaviour
during cooperation (for example in project groups).
• The PE instrument Scorion has been developed especially
for TU Delft.
12
Added value of Peer Evaluation
• PE can be embedded in teaching and function as an early-warning
system, helping projects to run more smoothly.
• The tutor’s own impressions of how the project groups are
performing can be compared with the findings of the students
themselves.
• PE enables the students to develop social skills while
simultaneously acquiring the relevant subject knowledge.
13
Solution: Peer & Self
evaluations
Peer evaluation:
Assessing of team members on their contribution
to the project and the way that contribution came
into being.
Self evaluation:
Assessing of one’s own contribution to the project
and the way that contribution came into being.
14
Types of peer evaluations
Three types of peer evaluations:
1. Ranking (from top to bottom)
2. Dividing (Dividing up a fixed number of
points or amount of money amongst team
members)
3. Rubrics (Using qualitative criteria to have
students assess each other)
For project based learning at Aerospace
Engineering it was found that option 3 works
the best.
15
Criteria
Op L&R aan de TU Delft worden 5 criteria
gebruikt:
• Job performance
• Attitude
• Leadership / Initiative
• Communication
• Management of Resources
16
Job Performance
Attitude
Leadership/Initiative
Management of
Communication
Resources
++
+
Consistently does more than
required. Work is of
exceptional quality.
Positive and professional
attitude which favorably
influences other company
members.
Takes initiative to seek out
work, concerned with getting
the job done. Very involved in
the technical project.
Uses time effectively in and out
of group and works to get
others to do the same. All tasks
completed on or ahead of
schedule.
Oral and written skills
excellent. Very effective
within the group and to
reviewers.
Sometimes does more than
required. Work is of high
quality. A producer.
Positive attitude toward
project and the team.
Readily accepts tasks,
sometimes seeks more work.
Gets involved in the project.
Uses time effectively in and of
group. Completes all tasks on
time.
Usually effective.
Performs all assigned tasks.
Quality of work is acceptable.
Neutral attitude.
Gets involved enough to
complete tasks. Does his/her
share.
Wastes some time in group, but
works hard when a deadline is
near. Most tasks completed on
time.
Generally gets the point
across. Tries to improve in
weak areas.
Performs all assigned tasks.
Work must be redone or
repaired to meet standards.
Negative attitude toward
project and/or project.
Tends to watch others work.
Gets involved only when
necessary. Volunteers to help
when it will look good.
Wastes most of group time.
Seldom seen doing productive
work. Some tasks completed
late.
Skills ineffective. Makes an
effort to improve.
Performs some assigned
tasks. Work must be redone
by others to meet standards.
Negative attitude which
adversely affects other
company members or project.
Lets others do the work; does
the minimum he/she thinks is
needed to get by.
Does little useful work in group
or out; wastes his/her time and
others. Work is constantly late.
Skills ineffective. Makes
little or no effort to
improve.
0
--
17
Criterion: Job Performance
++
+
Consistently
does more
than
required.
Work is of
exceptional
quality
Sometimes
does more
than
required.
Work is of
high quality.
A producer
0
Performs all
assigned
tasks.
Quality of
work is
acceptable.
Performs all
assigned
tasks. Work
must be
redone or
repaired to
meet
standards
-Performs
some
assigned
tasks. Work
must be
redone by
others to
meet
standards.
18
Criterion: Attitude
++
Positive and
professional
attitude,
which
favourably
influences
other team
members.
+
Positive
attitude
toward
project and
the team
0
Neutral
attitude.
-
--
Negative
attitude
toward team
and/or
project.
Negative
attitude
which
adversely
affects other
team
members or
project.
19
Criterion: Initiative
++
Takes
initiative to
seek out
work,
concerned
with getting
the job done.
Very involved
in the
technical
project.
+
Readily
accepts
tasks,
sometimes
seeks more
work. Gets
involved in
the project.
0
-
--
Gets involved
enough to
complete
tasks. Does
his/her
share.
Tends to
watch others
work. Gets
involved only
when
necessary.
Volunteers to
help when it
will look
good.
Lets others
do the work;
does the
minimum
he/she thinks
is needed to
get by.
20
Criterion: Management of Resources
++
+
0
-
Uses time
effectively in
and out of
group and
works to get
others to do
the same. All
tasks
completed on
or ahead of
schedule.
Uses time
effectively in
and of group.
Completes all
tasks on
time.
Wastes some
time in
group, but
works hard
when a
deadline is
near. Most
tasks
completed on
time.
Wastes most
of group
time. Seldom
seen doing
productive
work. Some
tasks
completed
late.
-Does little
useful work
in group or
out; wastes
his/her time
and others.
Work is
constantly
late.
21
Criterion: Communication
++
+
Oral and
Usually
written skills effective.
excellent.
Very effective
within the
group and to
reviewers.
0
Generally
gets the
point across.
Tries to
improve in
weak areas.
Skills
ineffective.
Makes an
effort to
improve.
-Skills
ineffective.
Makes little
or no effort
to improve.
22
Experiences with Peer evaluations
• To make optimal use of peer evaluations they
should be carried out at least twice per
project (Mid way and at the end)
• Comment functionality is very useful
• Because the results are not directly converted into
a grade the system is accepted by students
as fair
• A good IT structure is a MUST to avoid
overburdening lecturers
23
Example: Peer Assessment in the
Design and Construction project
• Background:
•
•
•
•
•
1st year L&R Bachelor
14 week project (Sem 2, period 1 and 2)
5 EC
40 groups of 10 Students working together
Assisted by a group of 20 student assistants
• Peer assessment conducted in weeks 3.7 and 4.8
• Results:
• Made available to the tutors
• Discussed in a meeting with each tutor
• Are used as input to improve the group process (formative) & assist
the grading process
24
Examples:
Overestimating
Good student
Underestimating
25
Any questions?
• If you should have any additional questions contact Vincent
Brügemann, tel. 84363, [email protected]
26
Peer Assessment
Implementatie van de tools
Delft
University of
Technology
Challenge the future
PEER FEEDBACK/PEER REVIEW
28
Peer Review in Blackboard
• Mogelijkheden
• Via blackboard het verslag laten inleveren
• Deadline is in te stellen
• Aantal reviewers is aan te passen
• Review kan anoniem gedaan worden
• Feedback kan per vraag gegeven worden
• Docent kan model-antwoord geven
• Docent kan criteria en waardering per vraag instellen
• Connectie met de GradeCenter van Blackboarden
29
Peer Review on Blackboard
• Tekortkoming: Inactieve studenten
• Geen of weinig feedback op verslagen
• Minder nakijkwerk voor sommige studenten
• Minder inzicht wordt vergaard
• Ongelijkheid
• Workaround
• Verwijder inactieve studenten van cursus  Moeilijk te managen
30
Peer review assignment in BB
31
32
Peer review assignment maken
Per vraag kan ook een model antwoord gemaakt worden.
33
Peer review assignment (2)
34
Opdracht inleveren
35
Opdracht inleveren (2)
36
De opdrachten beoordelen
•Beoordeling van eigen verslag mbv model antwoorden
•Beoordeling van twee andere verslagen (anoniem)
37
De opdracht beoordelen (2)
38
Samenvatting van de review
• De docent kan een samenvatting bemachtigen:
• Ga naar ‘Course Tools’  ‘Self and peer assesment’
• ‘View submissions’ toont de inlevervooruitgang
• ‘View evaluations’ toont de verdeling, voortgang en scores. Evaluaties
kunnen ook ingezien worden.
• ‘View results’ toont de scores en laat een sync toe met Grade Center
39
PEER EVALUATION (PE)
40
Scorion
•In gebruik sinds 2011
•Aanzienlijke groei sinds 2011
•Afgelopen jaar ingezet voor 29 vakken
•Aantal gebruikers vorig jaar was ~7000
•Rubrics gemaakt door LR is de standaard
41
Scorion – gang van zaken
•Docent doet aanvraag bij E-Learning Support (ELS)
minimaal twee weken van tevoren.
•Docent dient keuze te maken in:
•Aantal evaluaties
•Formulier template
•Rapport template
•Docent deelt groepsindeling met ELS
•liefste groepsindeling op Blackboard
•Vragen over mogelijkheden?
 Bel E-Learning Support voor advies!
42
Scorion – gang van zaken (2)
•ELS maakt formulieren
•ELS verzendt de formulieren op afgesproken tijdstip
•Docent stuurt eventueel herinnering naar studenten
•Studenten vullen formulieren in (inloggen met NetID)
•Formulier sluit automatisch op afgesproken datum
 Docent kan groepsrapporten inzien
 Student is in staat persoonlijk rapport in te zien
43
Examples
Closed questions
Title
Scoring
explanation
(optional)
Scoring
All students in the group
have to be graded
44
Examples
Open questions (standard)
Open questions gives
the teacher the
possibility to allow the
students to give
feedback on their peers
publicly or in private
(to their teacher).
Feedback for every student.
Not compulsory in standard
form (can be changed)
45
Examples
Group report (for teacher)
The standard group
report contains the
average peer scores
for a student for
each question and
the score the student
gave oneself. The
column Delta gives
the difference
between these two.
46
Examples
Group report (for teacher)
The standard group
report also contains
the public and
private remarks.
These can help the
teacher to find
problems within
groups. If there are
any.
47
Examples
Group report (for teacher)
It is also possible to
plot the individual
results of students in
so-called Spiderplots.
These plots can be
added to the report
if it is requested by
the teacher.
The total average
score can also be
shown in the report
if requested.
48
Examples
Individual report (for student)
The student will only
see the average
scores he/she has
been given and will
not see the scores of
others.
The student will also
get a personal
Spiderplot.
Only the public
remarks will be
shown.
49
Further Possibilities
Questions
• New questions can be implemented. These
questions can be open or closed.
• The title, explanation of the scoring and scoring
can be altered if desired.
• Questions do not necessarily have to be about the
students, but can also be used to evaluate the
teaching assistents.
• It is up to the teacher which peer results are
available to the student.
• Open questions can be made anonymous.
50
Implementing Peer Assessment
• Do you want to implement Peer Assessment and/or do you
have further questions?
• Please contact E-Learning support ELS , tel. 89194,
[email protected].
• We need to have your application for a Peer Evaluation with
Scorion at least two weeks before the evaluation date.
• If you will be working with PE for the first time, then we will
need to have your application at least two weeks prior to the
start of the course.
51
Questions?
• If you should have any additional questions contact Vincent
Brügemann, tel. 84363, [email protected]
• For questions regarding the implementation of Peer Assessments,
contact E-Learning Support:
[email protected]
89194
52

Download Report