fmprovement Plan MA Media Studies Erasmus University

fmprovement Plan MA Media Studies Erasmus University Rotterdam
February 2014
1.
Introduction
On 28 November 2073 the assessment committee CIW and Media Studies released the final report
containing the results of the extemal review of the MA programme Media Studies of the Erasmus
University Rotterdam. This report was based on the site visit that took place on 28 and29May 2013.
The programme w¿ìs evaluated on the three standards of the NVAO's Assessment Framework for
Limited Programme Assessments in the following way:
Standard 1: Intended learning outcomes
Satisfactory
Standard 2: Teaching-learning environment
Satisfactory
Standard 3: Assessment and achieved learning outcomes
Unsatisfactory
Overall assessment:
Unsatisfactory
On 1l February 2014 the Erasmus University Rotterdam received the official outcome of the external
review from the NVAO, and was informed that the MA programme has the opportunity to apply for
reaccreditation by presenting an improvement plan before
3l
March 2014. This improvement plan
should contain an overview of the adjustments and changes which the programme
will make, or has
already made, to address the deficiencies signaled by the committee, in order to meet the requirements
of Standard 3 within two years (at the latest). The planning should be realistic and should enable a
reassessment by the panel in the autumn of 2075, on the basis of the results of the2014-2015 graduate
cohort.
In the report, the committee motivates the negative evaluation of Standard 3 by the occurrence of too
many unsatisfactory MA theses. This concems 2 out of 17 theses in the original sample (all grades);
and 3 ottt of 10 extra theses (this time only "weak" theses which had received the grade 5.5,6, or 6.5
were assessed by the committee). The committee explicitly states they "only had a problem with the
minimum criteria the programme found adequate" (p.26). The committee had no issues with the other
grades: "the level of grades awarded by the programme was not too high in general" (p.26).
Regarding the assessment procedure as such the report notes that "the committee \ryas impressed with
the quality assrrance of the programme for the master thesis assessment" (p.26).
I
The committee concludes its considerations by stating that
minimum level of the criteria used to
"if
the programme were too increase the
assess the theses the committee is confident this
would lead to
an improvement of the quality of the theses within one year." (p.27).
The Programme Management
-
in close collaboration with the Examination Board - developed
a plan
for improvement which is presented in this document. Despite the specifically targeted critique of the
committee, we have used this opportunity to critically evaluate multiple aspects of our MA thesis
procedure. Therefore, we propose a broader range of measures and adjustments to ensure that the
assessment in our programme is more than satisfactory. Our plan takes into consideration the specific
points of criticism with regards to the assessment of MA theses that the committee raised, as well as
other remarks by the committee which are relevant for Standard 3 and the overall quality of the
programme. Our measures relate to the following aspects:
General improvements MA Programme
MA Thesis ClassAvIA Thesis trajectory
1. Restructuring
2. Additional Methods Course
3. Introduction Methodological guidelines MA Programme
fmprovements MA Thesis Procedure
4. Changes in set-up
MA Thesis
5. Research proposal
MA Thesis
6. Assessment of
Class
MA Thesis
As can be seen (and will be discussed in more detail below), our measures concern
a better
methodological training of students, a more critically informed judgment of the proposed research
design of the MA Thesis, improving the quality of the supervision of the MA Thesis, and fine-tuning
the assessment procedure of the
MA Thesis.
For every point, we first disctrss the remarks of the asse'ssment committee, followed by the concrete
actions that has been (or
will
be) taken, and thirdly the expected outcomes of the measures for
improving Standard 3.
2
2. Improvements in the
MA Programme structure
Three adjustments are made to the genpral strncture of the
MA Prograinme. They
are meant to
allocate more time for methodological training and to reinforce student's general knowledge of how
to apply and report on methods.
Improvement 1: Restructuring the MA Thesis Class / MA Thesis trajectory
Standard 3 relates to the quality of assessment and the achieved learning outcomes. The negative
evaluation of this standard by the committee resulted from a negative judgment of how the minimum
criteria to assess MA Theses were applied. However, the committee also voiced some concerns about
the set-up of the MA thesis project. In their view, too many EC are awarded to the MA Thesis (20
EC) and the MA Thesis Class (5 EC), in which the research proposal for the MA Thesis is written
as
well as other methodological skills are trained. The committee suggests to restructure the MA Thesis
Class/tr4A Thesis trajectory, leaving space for another methodological course (p.17, l8). We have
decided to follow the advice of the committee.
Summary Improvement
l:
Action
20EC for MA Thesis --> 18 EC
Restructuring the MA Thesis Class / MA Thesis trajectory
5 EC for
MA Thesis Class --> 2 EC
Implementation
2074-2015
Expected olrtcome
More efficient writing of MA Thesis; extra time for additional method course
Explanation:
From the year 2074-2015, finishing the MA Thesis will be awarded with 18 EC. At the same time,
finishing the MA Thesis Class will yield2 EC. The 5 EC that are spared will be used for an additional
methodological course (see Improvement2). This allocation of credits also implies that the MA
Thesis Class needs to be restructured (see Improvement 4).
Improvem ent 2: Additional Methods course
While the committee
w¿ìs "very positive about the strong focus on empirical research" (p.14), they
found that there are too few rnethodological courses in the programme and also expected a broader
range of methodological techniques to be taught (p.17). They suggested including an extra
methodological course in the programme. We have followed this suggestion and introduce an extra
methods course.
J
Summary Improvement 2: Additional methods course
Action
Data analysis course (5 EC) will be added to the programme
Implementation
2014-2075
Expected outcome
Stronger research skills which
will
lead to higher quality
MA
Theses
Explanation:
This new methods course will be compulsory for all students and will be scheduled in term 3. The
focus of the course
will
be on data analysis, since the methods course of term 1 already covers various
data collection methods (survey, open interview, content analysis). In the new data analysis course,
students
will receive in-depth instruction about and hands-on-training in various frequently
used
forms of qualitative and quantitative data analysis, including the use of SPSS for various forms
of
statistical analysis. Other methods for data analysis will be offered as elective elements within the
course (e.g. critical discourse analysis).
We should emphasize here that the majority of students who enter our programme are already
educated in methodology and statistical analysis in their BA programme or premaster programme.
Students whose methodological and/or statistical backgtound is considered insufficient are required to
participate in the "lntensive methods" course which is taught before the start of the programme.
In other words, the additional data analysis course offered in term 3 builds upon and seeks to extend
previous methodological knowledge and skills of students and ensllres that students receive
significantly more training in applyingdataanalysis methods, which will improve the methodological
level of the MA Theses in our programme.
The choice for scheduling this new course in term 3 is motivated by (a) the positioning of method
collrses in the programme, (b) the connection to the MA Thesis, and (c) the balancing of the work
load for students. Since the additional method course focuses on data analysis, we find it appropriate
to offer the course after the course on data collection (term 1) and after the MA thesis class in which
students write their MA thesis proposal (term
I
and2). Furthermore, the course will also follow the
research workshops of term 2 where students already completed the empirical cycle, including some
kind of data analysis. Scheduling this additional method colrrse in term 3 is also motivated by
spreading the work load for students in a reasonable way.
While this new course will be introduced in20l4-2015, we are already preparing the course in20132014by offering extended workshops on quantitative and qualitative data analysis.
4
Improvement 3: Introduction Methodological guidelines MA Programme
The committee found that techniques and methods were sometimes "used in a less adequate way"
(p.26). Also, they stated that in some MA theses variables were sometimes not well defined which,
according to the committee, led to "measured variables that were superficial" (p.26).ln order to meet
this criticism, we developed more detailed Methodological Guídelines for applying and reporting on
research methods. Applying the Guidelines is compulsory for students and supervisors as from
January 2014.
Summary Improvement 3: Introduction Methodological Guidelines MA Programme
Action
Development of document containing methodological guidelines which
students should follow in writing the MA Thesis and other research papers)
Implementation
2013-2074
Expected olrtcome
Clearer reporting on the used research design and data analysis which
will
lead to higher quality MA Theses
Explanation:
Students and lecturers receive Methodological Guidelines providing clear and explicit standards for
the methods and results sections ofthe thesis (and other research papers that the students have to
produce in the context of the Research'Workshop which are scheduled in term 2).
This document integrates and expands on the already existing "quality guidelines for MA thesis
research". Whereas the latter guide mainly focused on what quantities of data are expected from
students in various research designs, the new Methodological Guidelines also detail how students
should apply and report on operationalization, sampling, and data analyses. This is explained for the
most commonly used qualitative and quantitative data gathering methods (e.g. open interview, survey,
experiment, qualitative/quantitative content analysis), and qualitative and quantitative data analysis
techniques (e.g. Grounded Theory, statistical data analysis).
The Methodological Guidelines were written by members of the programme management, who
consulted key methodological publications, examples from other communication departments, and
experts within our o\ryn department. Although the Guidelines are primarily meant to reinforce the
quality of the MA thesis, we are confident that the Guidelines will also contribute to improving the
overall methodological quality and consistency at the programme level. What is more, it is not only
the students who will have to consult the methodological guidelines, also the lecturers will work with
these guidelines when supervising students. Introducing these guidelines therefore also improves the
level of supervision (see Improvement 6).
5
3. Improvements in MA Thesis Process
Improvement 4z Changes in set-up MA Thesis Class
The
MA Thesis Class prepares and supports students in designing and writing their MA Thesis, and
aims to contribute to students'general academic skills. The committee reports the positive evaluations
of the MA Thesis Class by students and says it "is positive about the content of the MA Thesis Class"
(p.18), yet also is critical about the course. The committee finds that too much time (5 EC) is allotted
to the course, and reports findings which suggest that the research proposal -- produced in the MA
Thesis Class -- is not always the basis of the
MA Thesis (p.17-18). From this follows the
recommendation to, firstly, integrate the 5 EC of the Master Thesis Class into the20 EC of the MA
Thesis, and secondly "make more efficient use of the proposals" (p.18).
In line with these recommendations, we will change the set-up and the EC allocated to the MA Thesis
Class from September 2014.
Summary Improvement 4: Changes in set-up MA Thesis Class
Action
(a)
(b)
MA Thesis Class -> 2 EC
In 2013-2014: Students will be required to take 3 workshops on
5 EC for
argumentation and quantitative/qualitative data analysis.
(F
or 201 4-201 5
see Improvement2: Additional method course)
(c) Earlier selection of MA thesis topics
(d) Earlier start with development of Thesis research proposal in MA Thesis
Class
(e) Communication Guidelines for Thesis Supervision
Implementation
2014-2015 (a)
2013-2014 (b, c, d, e)
Expected outcome
Research proposals which are produced at the end of the
will have higher quality, and thus form
MA Thesis
Class
a more solid basis for the thesis.
Making the workshops compulsory will aid the quality of data analysis in the
thesis (in 2013-2014 when the new method course is not yet in place).
Explanation:
As was already explained in Improvement
the
MA Thesis project from2} to
1, downsizing the
MA Thesis Class from 5 to 2 EC (and
18 EC) creates space for an additional methods course.
At the same
time, the data analysis workshops in the MA Thesis Class, which in the past were offered to students
on a voluntary basis, are made compulsory. Note that this latter measure is only for 2013-2014, since
6
next year the extra 5 EC method course will be introduced (see Improvement 2). Together these
measures are expected to substantially increase the methodological quality of the
MA
Theses.
Another aspect of reforming the MA Thesis Class concerns a more timely and intense guidance of the
writing process of the research proposal. To this end, students will select their thesis topic already in
term
1 (end
of October). Students can subscribe to projects and topics submitted by supervisors. The
Master Thesis Coordinator acts as a match-maker between thesis supervisors and students, in order to
allocate students to supervisors' thesis projects. The Thesis Coordinator thus ensures that for each
topic or project, a student works with a supervisor with appropriate methodological expertise.
Also, students are asked to start developing their proposal earlier than in previous years (November).
In the
MA Thesis Class extra assignments
are introduced to help students in preparing. Both the
research question and the research design are already discussed in the MA Thesis Class. By
rescheduling the design phase ofthe proposal, and offering extra guidance (on top ofthe supervisor's
feedback), we expect the MA thesis research proposals to be more developed, and thus providing a
stronger basis for the thesis itself.
Finally, to ensure that students and supervisor are aware ofthe existents regulations concerning the
process and duration of the
MA thesis supervision
and the consequences of submitting an insufficient
thesis, we have tightened up the wording of the regulations for MA Thesis supervision, and informed
students and staff explicitly about the
re
gulations.
Improvement 5: Research proposal MA Thesis
Students first write a research proposal which must be approved by the supervisor before they can
start writing their
the
MA Thesis. The committee
argues that the procedure for the research proposal
MA Thesis is currently not optimal. They recommend that
of
each research proposal is not only
evaluated by the supervisor, but also by a second reader. We follow this recommendation, and
complement this improvement by further fitne-tuning the guidelines and evaluation process, and
extending the pool from which second readers are drawn (the so-called Second Reader Panel, or
sRP).
Summary Improvement 5: Research proposal MA Thesis
Action
(a) Fine-tuning ofguidelines and assessment form ofthe research proposal
(b) Assessment ofthe research proposal by a second reader (SRP panel)
(c) Extension of SRP panel
(d) Evaluation ofthe procedure sub 3b and identification ofpoints
improvement for 201 4-201 5
(e) Finalizing procedure for 2074-2075
7
of
Implementation
2013-2014 (a, b, c)
2014-2075 (d, e)
Expected outcome
MA thesis research proposals will be evaluated more critically and reliably,
which increases the quality of the proposals, and, ultimately, the quality
the
MA
of
theses.
Explanation:
The existent guidelines and criteria for the research proposal have been critically evaluated and
improved. This was done parallel to the development of a corresponding assessment form for the
research proposal. In developing this latter form, the learning goals of the MA Thesis and,
consequently, the assessment criteria for the
MA Thesis were
used as a starting point.
Developing this research proposal assessment form enabled us also to achieve a more reliable
assessment of the proposal by comparing the judgments of the supervisor and a second (independent)
reader. This second reader is member of the Second Reader Panel: a selected group of lecturers from
our department with various forms of research expertise, but all holding a PhD and having significant
experience in supervising MA theses and doing research. Thus, the assessment procedure for the
research proposal has been extended and more formalized which should warrant a higher quality
of
the research proposals, and ultimately, a higher quality of MA theses.
These measures have already been irnplemented in term2 and 3 of the current academic year (2013-
2014). Still, we see them as a first step, and will evaluate this revised procedure at the end of the
academic year in order to finalize the procedure in20l4-2015.
Improvement 6: Assessment of MA Theses
The committee explicitly stated they "only had a problem with the minimum criteria" (p.26) that were
used to assess the theses, since the report also noted that "the committee was impressed with the
quality assllrance of the programme for the master thesis assessment" (p.26) and "the level of grades
awarded by the programme was not too high in general" (p.26). The committee asked to increase the
minimum level of the criteria used to
assess the theses. We have taken the
following five measures to
meet this request.
Summary Improvement 6: Assessment of MA Thesis
Action
(a) Improvement assessment form
(b) Instruction of lecturers how to apply the assessment form
(c) Fine-tuning assessment procedure
(d) Quality assurance by the ESHCC Examination Board
8
(e) Composition of Second Reader Panel
Implementation
2013-2014
Expected outcome
MA Theses will be evaluated more consistently, and more critically, and this
will prevent insufficient
theses to be passed.
Explanation:
a)
Improvement assessment form. Every MA Thesis is evaluated using a standardized form,
containing 5 criteria (Definition of the problem; Incorporation of the literature;
Accountability and Desìgn of research; Conclusion and discussion; Argumentation). For each
criterion, the assessor give a score on a 5-point scale (ranging from'poor'to 'very good')
which is averaged and transformed into
a grade on the scale
from I to 10. Although the
committee was positive about the assessment form, we have carried out some adjustments,
particularly with regards to assessing the methodology of the thesis.
Whereas in the old form research design and reporting of results was evaluated within the
same criterion, we deemed
it more appropriate to distinguish between an assessment of the
Accountability of the Research Design, and the Analysis and Results.In addition, the fivepoint scale was extended to a six-point scale by adding the explicit category of "fail".
b)
Instruction of lecturers how to aþply the assessment form. The instructions for filling out the
assessment form have been made more explicit in a document which is sent to all supervisors
and members of the Second Reader Panel. The
MA Thesis Coordinator explains the
assessment form and procedure in an information meeting (term 3 of the Academic year)
for
all supervisors and second readers.
c)
Fine-tuning assessment procedure. The assessment procedure which in the standard situation
consists of two readers (supervisor and second reader) was fine-tuned in the following way.
So far theses received an extra third evaluation when:
(i)
the proposed grade was an 8.5 or higher;
(ii)
the grades of the two readers differed by one point or more;
(iii) at least one of the two readers considered the thesis (almost) a (minor) fail (5-5.5).
(Note that clear fails, 4.5 or lower, were not assessed by a third reader, but
immediately failed).
We have made this third rule (iii) more strict: all theses which are evaluated as a "6" are
now also
d)
assessed by a
third reader.
Ouality assurance by the Examination Board. The assessment procedure of the MA thesis
Media Studies is regulated and audited by the ESHCC Examination Board (EB) in the
following ways.
(i)
First, the EB decides which lecturers, based upon their teaching experience and
expertise, can perform examination tasks (such as supervising and evaluating theses).
9
The EB has formally delegated the final assessment of the MA theses Media Studies
to the SRP. Members of the SRP are proposed and appointed by the EB in agreement
with the Chair of the Department Media and Communication.
(ii)
Second, the assessment procedure of the MA thesis Media Studies is part of the
Assessment Protocol of the EB. This protocol describes the procedures and quality
guidelines that all examiners
-
including thesis evaluators
developing exams and grading. The EB
will
-
must follow when
select every year a sample of theses to
check whether the assessment procedures for the MA thesis have been followed
correctly.
(iii) Third, the EB will check for this sample of theses the quality of assessment by
establishing whether the assessment is properly substantiated and in accordance with
the Methodological Guidelines (see Improvement 3).
e)
Composition of Second Reader Panel
Finally, we have extended the SRP panel to l0 members, including all MA programme
coordinators, to increase the diversity ofresearch skills, critical ability, and reliability
evaluations.
10
of

Download Report