(Based on my work with Margaret Kiley),
How can supervisors advise students about what to expect in their doctoral examination and viva? What do examiners say they are looking for in successful PhD theses which make a contribution to knowledge? How do examiners go about their reading and examining a thesis? Do examiners cross conceptual thresholds when examining theses? Do they need to see , and what is the evidence of moments and stages in a thesis which evidence a sound contribution to knowledge, originality, coherence, and publishability? How as supervisors can we support students to both produce a competent, coherent, well written thesis and also to make these moments or new insights, explicit? So that examiners recognise them?
The only evidence of the work a PhD or Masters student produces is the written thesis or dissertation, so everything hangs on its quality, its ability to articulate the research processes, and the fully understood discoveries, well articulated. This might seem a little unfair if the student is good at research but not a very good communicator, but it is up to the student, with our help, to express what the enquired, how they conducted the research, what they constructed or found and what it contributes to knowledge and meaning in the field. Section 4 discusses writing the thesis, at length, and emphasises the importance of a narrative and argument to articulate, present, share the new knowledge. For students undertaking, for example, maths or music research, much of the notation is not in an argument form, but a presentation in the language of those disciplines. Even with these cases, as with highly creative, performative pieces, where ballet or the artwork would seem to speak for themselves, it is still necessary to consider the readership, in this case the supervisor and the examiners, and articulate the research process, situate the work in the field, and similarly articulate what has been constructed, or found, and what it contributes. For many undertaking artistic research work, this means sharing both the product, for example a performance, a sculpture, a fashion design, an engineering design, a series of poems – and communicating what and how they contribute to knowledge, through the shared language of a thesis text. For those whose work accompanies their professional practice developments, evidence of the effect of the research in their professional practice is also important, but there is the narrative, the articulation of the research processes and what has been constructed or discovered and what it contributes to understanding and knowledge in the field, as well. The thesis is a shared text, it must be readable.
Doing the research and writing the thesis are difference but linked activities and it is possible you might have a student who is better at one of these than the other.
Trafford and Leshem (2008) suggest that it is useful to be guided from the start of the research and writing, by knowledge of what the examiners will be looking for – in other words to look at the research literature and gather information about the experiences of others so that supervisors can help direct students towards achieving according to accessible criteria for a passable, or good, or excellent thesis.
How do examiner examine?
Examiners of doctoral theses effectively have no formal training for this role, instead they learn by doing. It could be useful for supervisors and for new examiners to reflect on the learning gained from the role to feed not only into their supervisor practice but their activities as examiners.
Examiners we interviewed indicated that as a norm they tend to assume the work is passable or it wouldn’t have got as far as to be submitted, so the supervisory team are at least reasonably satisfied with is (however, in some universities, students can submit without the approval of their supervisor and sometimes without their knowledge). Supervisors tend to feel often that they are under almost equal scrutiny to that directed at the student and their work, because in letting the work go forward they are giving is a stamp of approval. So this assumption about quality and review cuts both ways. Examiners tend to read the abstract to gain an idea of the students awareness of their major contribution to the field, their confidence, ownership and argument, then they read the conclusion to see the equally coherent investigation ? of major contributions and awareness of limitations and further work in the conclusion ntyers are both factual and conceptual conclusions so the conclusions chapter pulls the whole thesis together and emphasises its findings and their importance and accuracy. As they continue to read, they identify theme, coherence, the storytelling narrative and argument, and the confident recognition of new knowledge which is situated in the context of previous work, well produced through robust research design in practice. They also check for e?? the introduction followed by the rest of the thesis comes next, although some examiners look at the references at this stage to ensure that the main areas of work in the field have been recognised. One examiner in our study said their next activity was to wait to be surprised and to hear something quite new. They expect a coherent, well organised piece which is readable and also links the work of each section together so that the new work is situated in a dialogue with theorists and other critical voices, is clear, well informed, confident and also realistic about the limitations, the methodology and methods, has taken ethical ? of the whole work, and the ight os anthing and ayine involved in it, has a robust design which is worked out through the whole thesis, shows how the data has been analysed, informed by the rcetuives of the theories, developing themes, major issues, and commenting also on the interpretation of the analysis, in the discussion, on what kind of patterns, what tensions, what issues and what perhaps surprising revelations or and revelations have been discovered.
From this scrutiny they produce early reports which are either held back until the morning of the exam, or shared by a committee, or shared between the examiners, holding back ensures that separate individual responses are gathered and then discussion either face to face in a pre-viva meeting or virtually online or through the mediator of a committee, can enable the examiners to come to conclusions about the thesis and its merits and an corrections needed, from minor corrections which are largely cosmetic through to a resubmission, Universities tend to have a system to classify PhD’s, a four of five point system – pass without any alterations, minor modifications, major modifications, resubmit, fail – though some also have a middle stage which is substantial modifications without the need to resubmit and if there are viva’s involved, the decision to hold the viva again is made on this significance of the modifications in the resubmission. In some countries where the entire process is conducted via email, examiner comments are given back to the students but do not affect the judgment. In most countries they directly affect the judgment and a joint report is constructed from the agreed set of examiner responses, and shared with the student so that they, working with their supervisor, can re-work the thesis for minor or major resubmission. This is a moment when supervisors need to be engaged and supportive as well as practical, since the student is usually surprised that their work has not passed outright first time. In my own work with students, I ask them to consider the examination process as resembling a peer review, i.e. a stage towards completion rather than actual completion. Like previous formal stages such as transfer, they engage with experts who read their work and make suggestions to rnahnev t and they must work with the supervisor to ensure these enhancements are taken into the thesis, for those who have a lot of work to do this could be a protracted experience, but it must be more than cosmetic, the poorest responses to peer review in journals is to cosmetically change a few words and hope that the reviewer does not re-read carefully, examiners will require the re-written thesis and a well written mapping of the work needed to fulfil the recommendations or the conditions, a set of changes related to expected responses, and a narrative accompanying these changes which might be recorded in sort of a grid, then signposted around the re-written thesis, the aim is to make the re-written work as clear, as obviously responding to the examiners recommendations, so that the whole is seen to be an improved and asable piece. One of the first things for us as supervisors to consider are – how do examiners recognise and comment on doctoral level achievement? How do they discuss theses considered marginal? What are the characteristics of marginal theses, and of good or excellent theses?
Research conducted on examination reports from doctoral examiners in Australia 2002, and more recent work interviewing doctoral examiners internationally suggest they are guided by both qualities of cohesion, organisation, comprehensibility and other characteristics of doctoral level achievement including conceptual threshold crossing which is indicated in the moments where new understanding emerges, and the doctoral students clear awareness and control of that. One of our interviewees noted their interest in seeing work which goes beyond a standard thesis, where there is some lively ‘illuminated’ thinking , new ideas, engaged expression. They said:
‘It’s not illuminating, I want to see lights come on in what the person is saying and what they’ve achieved, if it’s flat……. If it’s pedestrian and flat you know they can tick the boxes, they have done this, they have done this, and they have done that, then that’s the straightforward thesis work but something that’s moving on from that, the exception is where I see illuminated thinking coming through in the sections.’(Examiner interview)
This examiner is acknowledging that theirs is a difference between a satisfactory piece of work and one which shows awareness of enhancing understanding, making a significant contribution to knowledge.
Early work was carried out by Margaret Kiley and Gerry Mullins (2002) and latterly Margaret Kiley and Gina Wisker (2010).There is a wealth of work studying examiner reports mostly in Australia and the UK (Holbrook et al, **) all of which works from the report written and the statements made by examiners who reflect on the characteristics of a marginal or good thesis, and often also comment on the overall shape of a thesis, and progressions through an argument, and some exciting, unusual findings, referring back to Section 6 when we considered conceptual threshold crossing, it could be that recognising the moments when students have articulated the new understandings in their work, is a recognition of conceptual threshold crossing moments – and it is advisable to let students know that examiners are looking for these ‘illuminating’ moments, so they need to be clearly noted and mentioned in the thesis, not just allowed to emerge. It is important to emphasise early in throughout where appropriate and at the end, what breakthroughs in thinking and findings have been achieved. This is a bit like a marketing technique but it also offers an important signal to examiners that the work is of doctoral standard.
**
Research conducted on examination reports from doctoral examiners in Australia 2002, and more recent work interviewing doctoral examiners internationally suggest they are guided by both qualities of cohesion, organisation, comprehensibility and other characteristics of doctoral level achievement including conceptual threshold crossing which is indicated in the moments where new understanding emerges, and the doctoral students clear awareness and control of that. One of our interviewees noted their interest in seeing work which goes beyond a standard thesis, where there is some lively ‘illuminated’ thinking, new ideas, engaged expression. They said:
‘It’s not illuminating, I want to see lights come on in what the person is saying and what they’ve achieved, if it’s flat……. If it’s pedestrian and flat you know they can tick the boxes, they have done this, they have done this, and they have done that, then that’s the straightforward thesis work but something that’s moving on from that, the exception is where I see illuminated thinking coming through in the sections.’(Examiner interview)
This examiner is acknowledging that there is a difference between a satisfactory piece of work and one which shows awareness of enhancing understanding, making a significant contribution to knowledge.
Early work was carried out by Margaret Kiley and Gerry Mullins (2002) and latterly Margaret Kiley and Gina Wisker (2010).There is a wealth of work studying examiner reports mostly in Australia and the UK (Holbrook et al, **) all of which works from the report written and the statements made by examiners who reflect on the characteristics of a marginal or good thesis, and often also comment on the overall shape of a thesis, and progressions though an argument , and some exciting, unusual findings, referring back to section 6 when we considered conceptual threshold crossing, it could be that recognising the moments when students have articulated the new understandings in their work, is a recognition of conceptual threshold crossing moments – and it is advisable to let student know that examiners are looking for these ‘illuminating’ moments, so they need to be clearly noted and mentioned in the thesis not just allowed to emerge .It is important to emphasise early in throughout where appropriate and at the end , what breakthroughs in thinking and findings have been achieved. This is a bit like a marketing technique but it also offers an important signal to examiners that the work is of doctoral standard,
Our research explored whether it was possible to identify from thesis examiners comments how they recognised and commented on doctoral level achievement. We also sought to identify how the examiners discussed theses that were considered marginal.
Examiners report on examining processes and specific examples of stages in the thesis and viva when students are clearly evidencing work of doctoral quality, have ownership of their project, are articulate and confident about its achievements and discuss its workings and effectiveness in a collegial manner. Examiners talk about going through ‘conceptual thresholds’ (Kiley, Wisker 2009, 2010) in their recognition of the quality of the work.
From the analysis we argue that a marginal doctoral thesis will be characterised by ‘patchy’ work that indicates that the researcher/ writer is still in a developmental process. In contrast, a candidate who presents work that is
Conceptually coherent and
Intellectually rigorous
Indicates they have crossed various conceptual thresholds in their learning.
They show in their work, written and in the viva, if they have one, that they have undergone an ontological shift that has changed them as a researcher, and an epistemological shift evident in the articulated quality of their work and their contribution to knowledge.
The examiner reports analysed offer a useful checklist for supervisors and students to consider. We can discuss what each item means, and how in practice and in the students own work, and ask them to consider how they can move from exhibiting the characteristics of a ‘marginal’ thesis (which might still be one which passes) to a really good thesis. The discussion of these characteristics, made real by considering their characteristics in the students work, offers an opportunity to produce a reflection, and an action plan, moving from, for example, excess detail without a clear organisation or story or argument, to a sense of scholarly cohesiveness which nonetheless identifies key terms, concepts, arguments and pulls these into a coherent whole. It helps students identify the steps for moving from a poorly presented thesis to one without typos, with organisational coherence and structuring narrative story and argument, which enables examiners and other readers to see the main thread of the argument and how the research design in practice has yielded the results and findings and what they contributed, what they mean, and why it matters.
Some of the action points lead to a systematic audit of the work eg. for typos, and some lead to a rethinking of it’s overall cohesion and quality in the lught is the identified attributes of a good thesis. Different students and their work will have different issues. This audit and reflection and planning is an activity which could be useful conducted with a small group of doctoral students who can discuss between them how to get from there to here, and what work is needed in their own writing and research to do this.
What the reports say (Mullins & Kiley, 2002)
Add chart here from ppt.
| A ‘good’ thesis has… • Critical analysis & argument • Confidence & a rigorous, self-critical approach • A contribution to knowledge • Originality, creativity & a degree of risk taking • Comprehensiveness & scholarly approach • Sound presentation & structure • Sound methodology. | A ‘less than ideal’ thesis has… • Too much detail with lack of analysis • Lack of confidence, energy & engagement by the candidate • Lack of argument and rigour • Shoddy presentation (typos etc) • Lack of critique of own analysis/ sweeping generalisations based on opinion rather than analysis • Inadequate or poorly expressed methodology & scope. |
Some examiner comments from reports and interviews follow.
A “good” thesis from the reports
Its strength lay in the depths to which the candidate was able to pursue numerous disciplinary insights and the capacity demonstrated to keep these various insights focussed on his complex topic. (Sci 12)
It shows an extensive knowledge of relevant literature, a comfort with conceptual development, an ease with qualitative research techniques, a talent for the analysis of data, and a facility for writing up results. (Soc Sci 55)
A ‘less than ideal’ thesis
What he provides instead is a comprehensive historical/political description of…It is certainly a well-written account but it is also quite superficial. (Soc Sci 71)
The thesis requires a critical analysis of the method chosen (Soc Sci 85)
It is clumsy and repetitious, and too much emphasis is given to trivial matters…and too little to the actual implications of the results (Sci 104)
What do interviewees say about a ‘good’ thesis?
· Sound design, methodology in action
· Good qualities and cohesion throughout plus that extra ‘newness’ and ‘flourish’ which goes beyond the thesis
· Engagement with the literature in dialogue
· Real sense of mastery and adding something new
· ‘The magic ingredient’
I’m looking for somebody who really, really knows this subject so well that they’re able to step beyond the subject and go somewhere new with it. That for me is, it demonstrates their confidence, that they thoroughly comprehend where their subject is situated, that the theoretical underpinnings of it, but also the margins of the theory that they’re using, because then when they get to the margins of the theory and they’re challenging the theoretical underpinnings they’re then ready to move into a new realm and taking the methods of enquiry into that new realm to produce something, as an extension of the knowledge that they already had. (1)
However when identifying a marginal thesis they noe:
· Too complex without order or focus ‘too many beads’
· Too mechanistic
· Marginal
· If they’ve really significantly missed something (1)
Some examiners gave examples of a thesis which indicates enough work but probably without a sense of ownership or the identification of the contribution of an individual this kind of work often is an individuals thesis which exists in relation to a larger funded project, so that their work needs to be individualised, it’s unique contribution to knowledge is also highlighted alongside the completion of a set task of the shared funded project. Since examiners note the rather mundane workday qualities of such work, and the missing element of ownership and identifiable individual contribution to knowledge it would be useful to consider how to encourage and enable students to identify their own stamp on their work, their own element of a joint prkevey their own contribution. They are after all not merely technicians?, they are also doctoral students – and developing something unique to their own interest for the larger project will enable them to earn the research poirc teams oirces writing processes and to see how to transfer these to future projects.
Characteristic of a workaday –project are defined by this examiner @ although they talk about a science? project there are also large social science projects, for instance, which will equally need to enable students to differentiate their contribution to knowledge from that of others.
‘The odd PhD programmes in some of our best universities where the sciences you know, you come along you knock on the door and somebody says well the team’s down there, this is the actual metaphor that was used when I was first told this story, there’s a gang of people down there in the quarry we’ll give you a bucket and a pick and you can get down there and when your bucket’s full we’ll pull you up.’ (2)
What do they say about poor theses?
The lack of cohesion of conceptual/critical level seems to be a result of poor supervision
The student has been drawn into a research group and just carried out the workaday work
It is competent enough but lacks the magic ingredient
The cohesion and spark only emerge during the viva (and so what happens in systems without one?)
Poor work and poorer supervision
To be brutally honest his internal supervisors were not that hot, in terms of their knowledge of the subject area, kind of dumped on… one way or another he got dumped on them, they got dumped on him,(5)
Too busy and unstructured
About four things wrong with it which are relevant to this discussion. One was that the hypotheses weren’t followed through, two was it wasn’t particularly up to date, with some quite old references, some key references and some key ideas hadn’t been explored, and the final thing, where you think at the end of the thing, things will come together into a nice coherent whole and in particular he was offering a model, which was a diagram, and the links between the components of what you call the model were lined in the diagram but were not explained, so it was a kind of conglomerate, (5)
Because he had about six hypotheses, some of which he dabbled with (5)
These examples suggest poor? these are partly responsibility of the supervisor, whether that is true or not it is worth knowing that this is thought by examiners – the comments above reveal inconsistency, lack of coherence, lack of subject knowledge and something which is quite capable but has not made a contribution to knowledge, understanding. How might we support students to move beyond such potential limitations and flaws?
Not every system has vivas in fact they are rare in Australasia. In Scandinavia they are a large scale event involving as many as perhaps a hundred and an invited ‘opponent’ who is an expert who engages in a public dialogue with the student, on a stage. Most UK viva’s are quite private events with a Chair, an internal and an external examiner, perhaps the supervisor (student choice) and the student. The Chair is independent and ensures that the regulations are understood and the whole viva is conducted with appropriate decorum according to the rules and is fair to the student.
Role of the viva
To confirm the quality and enable a collegial dialogue
Explore the decisions made
Question certain errors and complexities Sometimes this enables students to make a more coherent form and argument – in person, in context – move through the conceptual threshold on the viva e.g.
R (7): In the viva we were able to ask questions in such a way that he indicated that he had made some of the connections and so we said okay.
INT: So the viva added a conceptual level to the paperwork?
R: Yes. And that’s how I see it, I know there’s a whole range of different approaches but I think the viva is so important in that to establish a relationship .. with the student or candidate however we refer to them and to ask the kind of questions no-one will ever ask them again, to you know honour their work and take it seriously and ask pressing questions.
Those where there’s a leaning forward and they’re enjoying the conversation because you’re taking it seriously, and they often think that you’re there to catch them out.
My own viva I learned something in the viva, I understood something from the question they asked me and I thought ‘Ah oh I see yes’.
INT: You went through a conceptual threshold in the viva.
Some examiners report that they transfer learning from the examination process to their own supervision practices (Wisker and Kiley, 2011)
As Halse and Malfroy (2010, p. 89) express it: ‘doctoral supervision is a specific, specialized type of professional work’. Their research discovered that supervisors have seen a distinct shift in the ways in which they now conceptualise and practise doctoral supervision. This shift is towards increased professionalisation, the distance of personal relationships, and intervention in students’ work to ensure quality. Supervisors are often isolated in their work,
We interviewed 7 very experienced examiners.
The simple thing you learn is the rules of the game—they help you to prepare [your own] students (Soc Sci Male A: L ater study)
Lessons learned and passed on when I have engaged supervisors in workshops what these experiences and research based ideas about examining they have reflected on their usefulness and said they have become aware of:
· a clearer understanding of how a candidate can effectively ‘defend their thesis’ either in the text of the oral examination.
· knowing their own stance and also being aware of different opinions, opinions that might be held by one or more examiner.
· writing and the written text as a coherent document in argument and style as well being presented in a scholarly fashion.
· ‘rules’ around the doctorate, including format and approach.
· an understanding of quality and standards, both in terms of Masters and PhD differences,
· an appreciation of the significance of having up to date literature and analysis rather than description
· how different examiners approach the assessment process.
Examining a doctoral thesis provides a valuable opportunity for professional learning for supervisors who learn first-hand of the examining processes and of the expected qualities of a thesis. Examiners in our study indicated how they took that learning directly back into their own supervisory practice. Demystifying the processes and practices of doctoral thesis examiners is a valuable outcome from research into the ways in which examiners approach, read and process doctoral theses, and the generic and discipline related quality indicators they seek in the
passable thesis and the thesis of merit (Mullins and Kiley, 2002; Holbrook et al, )( ki;ey an wiser 200 9?)
How can we learn from this information about examiners’ processes and practices to inform our work as supervisors and students?
References
Kiley, M. & Mullins, G. P. & University of Canberra (2002) Quality in Postgraduate Research Conference, & The Centre for the Enhancement of Learning, Teaching and Scholarship. Quality in postgraduate research [electronic resource] : integrating perspectives / edited by Margaret Kiley and Gerry Mullins
Kiley, M. and Wisker, G. (2010), Learning To Be a Researcher: The Concepts and Crossings, in Threshold Concepts and Transformational Learning, Land, R., Meyer, J.H.F. and Baillie, C., (eds), Sense Publishers, Rotterdam, pp. 399−414
Trafford, V. & Leshem, S. (2008) Stepping Stones to Achieving your Doctorate: By focusing on your viva from the start, Open University Press: Maidenhead.
Wisker, G. & Kiley, M. (2009). ‘Threshold Concepts in Research Education and Evidence of Threshold Crossing’. In Higher Education Research and Development.
Wisker.G (2005,2011 forthcoming) The Good Supervisor Palgrave Macmillan
Wisker, G., Kiley, M (2011 under consideration)Professional learning: lessons for supervision from doctoral examining. IJAD
Gina Wisker 