Programme of Assessment

20190624 v10

What is the purpose of assessment in the RCEM curriculum?

The curriculum provides clear guidance on the knowledge and activities of an EM clinician. It also provides a map to negotiate the route for the novice entering training to become an expert clinician, ready to lead the line.

All learners are different- at the start of training, in how they respond to EM work, the experiences they have and the resources available to them-including time.  It is vital that learners take responsibility for their training. They must seek out opportunities to learn, considering their trainers as coaches, with vast experience of the clinical arena and of others who have worked in it. They seek must out their honest feedback and be prepared to accept and reflect on it.

Trainees-if you reach a point of difficulty in training it is usually not that you can’t do something- it’s that you can’t do it yet. Taking time to develop expertise is not a mark of failure.

It is important that learners are able to cross thresholds of responsibility in training as safely as possible. That is the role of those providing training- to support, guide, and mentor.

This Programme of Assessment sets out clearly how learners will be assessed and how decisions will be made about their readiness to progress across the key thresholds in training.

We define assessment as all activity aimed at judging a learner’s attainment of curriculum outcomes, whether for summative (determining satisfactory progression in or completion of training), or formative (developmental) purposes.

We have a programme of assessment that integrates different ways of assessing learners that is blueprinted to the Specialty Learning Outcomes (SLOs) and the underpinning syllabus that we have defined.

That means that each of these contributes to an holistic view of how a learner is progressing and informs decisions about whether a learner is ready to cross thresholds in training.

What are thresholds in training?

Thresholds are points in training where the responsibility shouldered by learners significantly changes. Key thresholds in EM training are between Intermediate and Higher Speciality training, when the trainee will typically become the most senior clinician in the ED overnight, and between Higher Speciality Training and completion.

The need to ensure that these changes in responsibility are safe and reasonable steps for a learner to take is the main purpose of the assessment programme.

We explicitly outline what is required at these stages across each of the Speciality Learning Outcomes (SLOs) that make up EM training. The means by which a learner will be adjudged ready to cross these thresholds are summative assessments.

Summative assessment

There are two modes of summative assessment in EM training. These complement each other and offer a ‘programmatic approach’ [van Vleuten 2011]. That means that a number of different modes of assessment are used to deliver an holistic judgement on progression. The modes of summative assessment are ‘blueprinted’ to the curriculum to outline how each of the SLOs are covered and how the programme works as a whole.

Table 1. RCEM assessment blueprint

1 Complex stable patient 2 Answering Questions 3 Resus 4 injured patient 5 PEM 6 Procedural skills 7 complex and challenging situtaions 8 EPIC 9 Educate and supervise 10 Research 11 Patient safety and QI 12 Manage, administer and lead
Examinations
EM Knowledge Base (EKB)

SBA

** * ** ** *
EM Knowledge in Practise (KIP)

SBA

** ** ** ** *
MRCEM Readiness for Higher Training (RHT)

OSCE

** * ** ** ** * **
Advanced Knowledge in Practice (AKP)

SBA

* ** * * * * ** * * * **
FRCEM  Multi-Station Oral exam (MSO)

OSCE

* * * * * ** ** ** ** * **
FRCEM CLA

SAQ/SBA

 

**
QIP project **
ESLE * ** ** * * * ** ** *
Mini CEX ** ** ** ** **
CBD * ** * * * **
DOPs **
SLO Entrustment statement ** ** ** ** ** ** ** **
Management portfolio **
Teaching assessment tool **  

Formal RCEM examinations

The suite of RCEM examinations relate to the two key thresholds in training we define.

The MRCEM examinations are part of the preparation of learners to become HSTs. The MRCEM primary focusses on the basic science that underpins EM care and on how that knowledge base is used  to inform the skills and attributes needed to practice safely as an HST in EM.

The FRCEM examinations builds on this and is blue printed to the learning outcomes as they reflect the development of growing expertise in EM care. These tests of knowledge are grounded in the more complex considerations of someone preparing for independent practise. There is also a face to face examination that tests the application of knowledge, communication in complex situations, and the ability to analyse, synthesise and use data or evidence to inform decision making.

The examination schedule is as follows.

MRCEM

The MRCEM suite of examinations are designed to support assessment of trainees from entry to the speciality to the end of intermediate training. It is designed to ensure trainees have a wide ranging and comprehensive knowledge base that reflects the nature of the speciality, and have the techniques of history taking, examination, diagnostic decision making and communication that are needed to function safely and effectively in Higher training.

The components of the examination are:

  • EM Knowledge Base (EKB)-Single Best Answer. This examination samples the basic science syllabus, ensuring a sound background knowledge in the basic science underpinning EM care. It can be undertaken at any point post registration as a medical practitioner.
  • EM Knowledge in Practice (KIP)-Single Best Answer paper. This examination samples the clinical syllabus and ensures a sound understanding of the full range of conditions and presentations that may present to the Emergency Department, also sampling how trainees interpret data and examination findings to make decisions and inform management plans.
  • MRCEM Readiness for Higher Training (RHT)-OSCE. This examination objectively samples the clinical skills of trainees and ensures they are those of someone ready for Higher training. It includes history taking, examination, communication, decision making, dealing with challenging situations and resuscitation scenarios.
  •  Critical Appraisal (ECA)Short answer paper /Single best answer (following psychometric review). This paper is blueprinted to the Research SLO and assesses the EM trainee’s appraisal of the medical literature and decision making about its relevance for EM patient care. This paper can be sat at any time in training, from commencing ST1. It can be taken in HST if not done so in core or intermediate.

FRCEM

The FRCEM suite of examinations is designed to support assessment of trainees in higher training as they prepare for consultant level work as an independent practitioner. The examinations are blueprinted to the curriculum as it articulates the advanced knowledge, skills, behaviours and attitudes required to work at this level. It includes assessment of trainee management of complex clinical scenarios and the support and leadership of others.

  • Advanced Knowledge in Practice (AKP)Single Best Answer paper. This examination is blueprinted to the advanced clinical elements within the Speciality Learning Outcomes and the Clinical and Supporting Syllabi. This paper can be sat from the start of ST5
  • Multi-Station Oral exam (MSO)Face to face, multi station examination. This examination is blueprinted to the complex or challenging situations an EM clinician will face, the requirements of leadership and support within the ED and the supporting SLOs and syllabus. This examination can be sat from the start of ST5. (This is exam is currently under construction with a review of the ‘state of the art’ in exit exams and psychometric input. It may include data interpretation and communication, teaching activities, prioritisation exercises, identification and management of patient safety concerns and the skills needed to support the team as shift leader).
  • Quality Improvement Project (QIP)Submitted project. This project is centrally marked and reflects the trainee’s ability to understand and implement QIP methods and includes change management and leadership elements of the SLOs and Supporting syllabus. This can be submitted at any point from the start of ST4.

The Programme of Assessment integrates these formal examinations with assessment in the workplace. The formal examinations represent psychometrically reliable instruments and have strength in giving reliable estimates because of the standardisation of the content within them. The nature of EM care in the workplace is that it is delivered in situations characterised by a distinct lack of standardisation. This makes reliable point estimates of trainee’s capability very challenging.

The integration of the knowledge skills behaviours and attitudes, as tested in the formal examination schedule, into the demanding and random nature of EM care is assessed in the workplace over a period of training. The WPBA schedule is designed to offer formative feedback of these aspects of EM care and inform summative multi-consultant entrustment statements about readiness to progress.

In essence, both aspects of assessment are required to make a robust and holistic decision. The examination schedule is not the gateway to the next phase of training, or completion. It is information that is required to contribute to a decision that a trainee is ready to progress. That being the case, it is not necessary that the exams are sat at the end of a block of training, but can be taken when a trainee is ready to do so. The schedule is designed to reflect the variation between how learners progress and their autonomy as adult learners. The sequencing of examinations is, therefore, designed to give the most flexibility possible.

Entrustment decisions

The complementary mode of assessment to the formal examinations is entrustment decision making.  Entrustment is defined so:

‘ to charge or invest with a trust or responsibility: charge with a specified office or duty involving trust’

The content within the SLOs that relate directly to patient care are sequenced to match changes in responsibility. As stated, training is about preparing trainees to cross these thresholds safely. The decision about when a trainee is ready to do so will be based on their performance in the workplace and the results of the formal RCEM examinations that relate to that stage of training. These make up a programme of assessment. Trainees will have received regular feedback in the SLOs with clear guidance about entrustment for the cases they have seen. They will have had the opportunity to reflect on this feedback. The purpose of training is to build opportunities for learning and to strengthen areas of practise that need to be strengthened. These will vary for different trainees and the opinion of the learner themselves on their readiness to progress will also be important.

The assessment schedule includes the requirement, therefore, that the training faculty give an opinion about the learner’s readiness to cross a threshold in training. It is, in effect, a prediction of how the learner is likely to respond to situations and circumstances they find themselves in when the training faculty are not there. It is therefore incumbent on the training faculty to have provided support to learners and for learners to have prepared themselves to reach these points.

A scale that describes the degree of independence with which a trainee may be entrusted is shown in in figure 1. The clinical responsibility required of trainees at the key way-points of training in each of the SLOs is summarised in figure 2. The requirement relates to the entrustment scale shown, reflecting the fact that differing levels of responsibility or expectation exist in different areas of EM work for learners at the same point in training. (This scale is the one used in ACCS and provides consistency across each of the ACCs specialities to ease transferability).

By making these waypoints explicit to both learners and assessors the decision making about progression is clear. The formative assessments within the programme of assessment are geared to supporting learners in their development towards these key progression points. 

Figure 1. RCEM Entrustment scale:

1) Direct supervisor observation/involvement, able to provide immediate direction/ assistance

2a) Supervisor on the ‘shop-floor’ (e.g. ED, theatres, AMU, ICU)  monitoring at regular intervals

2b) Supervisor within hospital for queries, able to provide prompt direction/assistance

3) Supervisor ‘on call’ from home for queries, able to provide directions via phone

4) Would be able to manage with no supervisor involvement

Figure 2. Entrustment by stage of training across all Speciality Learning Outcomes

Core:

ACCS: 6 months covering each of anaesthetics, acute medicine, intensive care medicine and emergency medicine. Can be worked in any order. SLOs reviewed in each post. Entrustment level required by end of ACCS

Progression Point

Intermediate:paediatric emergency medicine, leadership roles, support wider EM team

Progression Point

Higher Specialty Training:

leading, EPIC, delivering challenging cases, management and administration toolkit, research, supervision and teaching

Training level ACCS  

ACCS

 

Intermediate ST4 ST5 ST6
Care for physiologically stable patients attending the ED across the full range complexity 2b 3 4
Answer clinical questions 2a 2a 3 4
Resuscitate and stabilise 2b 2b 3 4
Care for an injured patient 2b 2b 3 4
Care for children in the ED 3 4
Deliver key procedural skills 2b 2b 3 4
Deal with complex situations on the shop floor 2a 2a 3 4
Lead the ED shift 3 4
Provide basic anaesthetic care (ACCS) 2b 2b
Manage patients with organ dysfunction and failure(ACCS) 2b 2b
Teach and supervise               4  
Participate in research               4  
Lead quality improvement               4  
Administer               4  

No entrustment decisions needed but must be ready for independent practice by end of training. Progress is sequenced throughout training

Faculty Educational  Governance (FEG) Statement

What is it?

This is a statement that summarises the collated views of the training faculty as to the progress of a trainee, specifically, their suitability to move to the next stage of training. This judgement is based on the observation of the trainee in the workplace, on feedback from staff and patients and what faculty members have learned about trainee’s performance  in conducting WPBA (Individual WPBAs and reflections will not be viewed by the training faculty). Within this statement the strengths of the trainee are also summarised as well as areas to develop thus giving the opportunity to reflect and encourage excellence.

The FEGS was introduced in RCEM training in 2015, with a decision relating to the whole training year in general. The evolution in this current programme of assessment is that the decision is broken down into the relevant Speciality Learning Outcomes. Anchoring this decision to independence with a clear description of what is required will be a significant benefit to trainees and trainers in making these decisions explicit.

The FEGS does not replace an ARCP decision but is a powerful statement that is triangulated with other information from the Educational Supervisor’s Structured Training Report to inform the ARCP decision. The FEGS is held on the e-portfolio and is accessed by the Educational /Clinical supervisor and Training Programme Director only.

How is it done?

The FEGS can be made in different ways according to local arrangements. It must represent the collated views of the training faculty as to whether they believe a trainee has met the requirement for practise in each of the relevant Speciality Learning Outcomes (SLOs) at the level of independence specified for their stage of training. The decision will relate to the key capabilities for each SLO.

The faculty is bound by the requirements on them of the GMC’s Good Medical Practice guidance, by the requirements for fairness and transparency, the requirement that equality and diversity is respected and by the personal ethics and probity of individual members.

Good practice from a number of centres has been that ‘educational governance’ is a standing agenda item at consultant meetings and discussions of all trainees occur at regular (e.g. two- monthly) intervals. This approach ensures that concerns are shared early and trainees can be better supported. It facilitates encouragement of trainees and the feedback of excellence. It is also fair to trainees who will receive a summative decision from the same panel that they are fully aware of how that group are minded towards their progress in each of the relevant SLOs.

The final meeting is for the purposes of FEGS completion. A quorate meeting would include at least three consultants, who must be trained Educational Supervisors.

Other centres have a designated training faculty from among their consultant body that perform this function at a formal Educational Governance meeting comprised of the College Tutor (or equivalent), Educational /Clinical supervisor and at least two other consultant trainers. At this meeting the progress of each trainee against each SLO is discussed and the output of this meeting is the Faculty Education Governance Statement.

Example:

SLO1. ACCS trainee: ‘ Can this trainee take a history, examine the patient and elicit key clinical signs, construct a differential diagnosis that considers a realistic worst case scenario and describes an appropriate management plan with senior help available, but not directly overlooking their work?’

The panel’s view on this will be sought. Panellists will be asked to reflect on their experience of trainees across the full spectrum of cases. This decision is a statement about the reliability of clinical technique and the confidence of the team that a learner can be relied upon to make a safe assessment and seek help as needed.  A yes/ no answer is required.

This process is repeated for the other Learning Outcomes that are relevant to the current phase of training.

The FEGS is recorded in the trainee’s e-portfolio by their Educational or Clinical Supervisor. The FEGS also includes general feedback on trainee strengths and areas to work on.

When is it done?

Faculty Governance statements are made towards the end of a given block of training in an Emergency Medicine placement. This is typically six months (whole time equivalent) during ACCS and Intermediate Training and yearly in Higher Training.

What if a trainee is deemed not ready to progress?

For the large majority of trainees these decisions will be positive. However if problems or concerns are raised about a trainee in departmental education governance meetings, or by other means, these can be fed back with learning needs identified and a plan to remediate put in place. If these persist throughout an entire block of training this will be reflected in the FEGS and the subsequent ARCP panel will outline an appropriate training plan.

An opinion that a trainee is not ready to progress should not come as a surprise at the end of a placement and should not be seen as punitive by the trainee or trainers. It is a formal recording of the opinion of the faculty on progress at the end of that training block and reflects support and deliberation throughout the block.

Indeed, the fact that not all trainees will reach the threshold after the same duration of training is a realistic reflection of the variation that exists between learners, their experience and the complex and highly responsible role a higher trainee and consultant in EM clinician embodies.

Example

SLO1.  ACCS trainee:  ‘ Can this trainee take a history, examine the patient and elicit key clinical signs, construct a differential diagnosis that considers a realistic worst case scenario and describes an appropriate management plan with senior help available, but not directly overlooking their work?’

The panel’s view on this will be sought. Panellists will be asked to reflect on their experience of trainees across the full spectrum of cases. This decision is a statement about the reliability of clinical technique and the confidence of the team that a learner can be relied upon to make a safe assessment and seek help as needed.  A yes/ no answer is required.

This would be repeated for the Learning Outcomes that are relevant to that phase of training.

For the majority of trainees these decisions will be positive. The faculty will also be asked to record strengths and areas to develop in general. This is an opportunity to reflect and encourage excellence.

The faculty governance statement is a collective view that is available to the Educational Supervisor when compiling their structured training report. In the STR this statement is integrated with the result of WPBAs, MSF, any significant incidents and a review of activity within the e-portfolio. This is then reviewed at ARCP. There is, therefore, externality and overview of the activity of Faculty Educational Governance panels. The FEG does not replace an ARCP decision, therefore, but is a powerful statement that is triangulated with other information to inform that decision.

Formative assessment

All individual assessments in the workplace are formative, and therefore developmental, in nature. That means they cannot be failed.  These episodes are an opportunity for learners to receive feedback about progress towards key progression points.  They are designed for that purpose.

WPBAs are anchored to the entrustment scale used for summative decision making. They provide the opportunity for detailed, individualised feedback to be given across the SLOs. The purpose of these assessments is to support learners in readying themselves to cross into the next stage of training. To that end they are designed to be used where the trainee is working towards the limit of their current capability. They are an opportunity to get feedback in a supervised setting where safety can be ensured by the experienced faculty present.

Assessment in the workplace should start right at the beginning of training and continue regularly thereafter. It is the responsibility of the learner to seek out, with the full support of the training faculty, learning opportunities that allow progress against each of the relevant SLOs to be fed back.

This means, for example, that learners in core training should be observed and receive feedback about their role in resuscitation, challenging episodes on the floor and those in intermediate should do so on answering clinical questions, developing safe discharge plans and providing leadership to areas of the ED. Trainees would not be expected to be declared ready for independent practice at the start or even the middle of training. The collation of evidence by WPBA that attested to that would suggest that feedback on cases well within the capability of the learner have been sampled. This does not reflect the growth that is required to take a novice to the points in training outlined and therefore can be seen to somewhat miss the point. RCEM has developed an approach that is designed to support purposeful practice by trainees. That means to get the best out of training, both learners and assessors gear the interactions in the workplace that the formative WPBA schedule represents largely to work that will be required to be delivered in the next stage of training.

The summative decision point is at the end of a period of training and is holistic. It takes account of what has been revealed about learners by observing them in WPBA as one of the sources of information that is triangulated.  Using the learning opportunities available to their fullest, seeking and reflecting on feedback and changing when needed offer the best chance of readying oneself for the transitions in responsibility that accompany progression.

The collation of a range of evidence in formative assessment is a clear indication of engagement in training and shows an understanding by the learner that expertise in Emergency Medicine is complex, multi-faceted, hard won and is best achieved using the training faculty as expert coaches.

Concerns raised in training

As stated, formative assessment is developmental and therefore individual episodes cannot be failed. However, concerns about patient safety, trainee wellbeing or personal/professional conduct may be raised as a result of an assessment in the workplace. Each of the formative assessments has a prompt that requires concerns, should they arise, to be raised with the learner’s supervisor.  These can then be discussed and may form part of an educational approach designed to address the issues identified in a clear and transparent fashion.

Although a trainee may not elicit particular safety concerns, they may not be progressing in a way that makes meeting the waypoints realistic. This would be evidenced, for example, by the outcome of several assessments that suggest there is greater need for supervision than might be expected in the areas sampled. It would then be for the learner and supervisor to explore these and ascertain the causes, with individualised plans then put in place for further development. Concerns may relate to areas of clinical work that trainees find difficult, or it may be there are other factors that are affecting performance. The narrative feedback from assessors will be very important in making this as effective as possible.

Training, and therefore the crossing of thresholds, is competency based. Trainees will move across when they and their supervisors think they are ready. As set out in the above, all learners differ. It is to be expected that learners will develop at different rates. The recommendation that further development is needed to cross thresholds in training should not be considered punitive, but may indicate more support is required.  It reflects the challenge of being the most senior clinician in the ED, or being an independent practitioner and patient safety dictates that a good decision is made. In preparing, it is logical that learners seek feedback on areas they are not comfortable with and develop across the full spectrum of practise. The curriculum makes key areas of EM practise explicit for the first time and some may lead to feedback that is challenging to accept. There is a great responsibility on the faculty to treat learners with respect, fairness, sensitivity and support- but also to be prepared to be honest. Learners also have a responsibility to take control of their training, seek out feedback across the breadth of the curriculum, be prepared to be challenged and accept feedback that is offered in the terms outlined above.

Workplace based assessment and ARCP and decision making

The assessment requirements for each of the Speciality Learning Outcomes are listed. There is no absolute number required for WPBA, but there are recommendations. In particular, the Key Capabilities in each of the SLOs will need to be evidenced in the e-portfolio.

Each trainee is different, and an individualised programme will develop as trainees and supervisors learn more about strengths and areas to work on.

Some SLOs have a requirement for evidence of specific interaction with a particular learning element in the Programme of Learning, e.g. Safeguarding training or Good Clinical Practice Training for research. Otherwise, a checklist approach has been minimised.

An example ARCP decision aid for ST3 is included below (table2). ACCS decision aids are currently being developed with ACCS partner specialities.

Table 2.  Example ARCP decision guidance for trainees for end of intermediate training

 

Trainee:

 

Educational Supervisor:

 

Date:

 

 

Mandatory Summative Examinations (all must be achieved prior to entry to HST)

EM Knowledge Base SBA: 

EM Knowledge in Practice SBA: 

Readiness for Higher Training OSCE: 

 

 

Further Summative Examinations (can be taken from ST1 onwards):

Critical Appraisal SAQ/SBA: 

 

 

Mandatory WPBAs (demonstrating no significant concerns that would prevent progression to HST)

Three ESLEs in ST3: 

At least one MSF/TAB: 

 

 

Entrustment level 3 and satisfactory evidence provided of attainment in each of the key capabilities of the following Specialty Learning Outcomes

Care for physiologically stable patients across full range of complexity: 

Answer clinical questions: 

Resuscitate and stabilise: 

Care for an injured patient: 

Care for children in the ED: 

Deliver key procedural skills: 

Deal with complex situations on the shopfloor: 

Lead the ED shift: 

 

 

Satisfactory progression towards the supporting Specialty Learning Outcomes Evidence linked to each of the SLOs

Rated Below/Meeting/Above expectation by ES

Reviewed by ARCP panel

 

Teach and supervise: 

Participate in research: 

Lead quality improvement: 

Administer, manage and lead: 

 

 

 

Procedural logbook

Satisfactory progression of competence in procedural skills: 

 

Work place based assessments (WPBAs) :  No requirement for a minimum total overall (other than the ESLE), although evidence in each of the SLOs, mapped to the Key capabilities, is required. Aim for quality over quantity. Useful WPBAs will challenge, act as a stimulus and mechanism for reflection, uncover learning needs and provide an opportunity for developmental feedback. You should aim to get feedback on your clinical work on ~ 50% of your clinical shifts. This may include brief learning encounters, for example those captured using the RCEM assessment app, as well as more in depth observations or discussions

RCEM Assessment App*, CEX, CBD, ACAT, DOPS, Resus leadership*, shift handover tool*

*New assessment tools.

 

Educational Supervisor Structured Training Report
Form R
Completion of GMC National Trainee Survey
Attendance at Regional Training Days

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.