Methodologies for evaluating short fellowship programmes
The ACU manages several short fellowship programmes which offer academics the opportunity to spend a period of time at universities or organisations in the UK and undertake professional development tailored to their specific professional needs. The fellowships can be seen as a collection of individual training paths, and their evaluation can therefore present some distinct challenges. Here, we take a look at two specific fellowship schemes and discuss the methodologies at play for assessing impact.
CSC Fellowships: As part of the UK’s contribution to the Commonwealth Scholarship and Fellowship Plan (CSFP), the Commonwealth Scholarship Commission in the UK (CSC) offers fellowship programmes for academics from developing Commonwealth countries to develop knowledge and expertise in their specialist areas. The programme, funded by the Department for International Development (DFID) of the United Kingdom, include Professional Fellowships for mid-career professionals to spend time at a UK host institution, Academic Fellowships for early career academic to conduct research at a UK university and Medical Fellowships to provide doctors and dentists the opportunity to update their clinical skills at UK hospitals.
CIRCLE Visiting Fellowships: CIRCLE (Climate Impacts Research Capacity and Leadership Enhancement) visiting fellowships commenced at the beginning of 2015. The programme, also funded by DFID, is set to run over three years, supporting 100 CIRCLE Visiting Fellows (CVFs) in total with a target of 60 post-doctoral and 40 post-masters Fellowships and a 50/50 gender allocation. The programme is relatively small in scale and timeframe compared to some of the larger scholarship programmes, however it is ambitious in its objectives and is striving for a sustainable legacy of improved institutional research output by building the capacity of institutions involved in the programme to provide systematic research support through an Institutional Strengthening Programme (ISP).
Evaluation challenges
Firstly fellowships are designed to provide a development opportunity to individual fellows and therefore the evaluation of these programmes needs to examine the extent to which they deliver on the knowledge, skills and learning for which they are developed, in addition to the longer term outcomes and impact of the individuals. This can be particularly challenging when the knowledge and skills developed are being examined across a wide spectrum of disciplines and specialities.
The dual objectives of these programmes to strengthen both the individual capacity of the fellows as well as their institutions also present an evaluation challenge. While understanding the impact on individuals and tracking longer term careers is important, it is balanced with the need to understand the immediate transfer of knowledge and implementation of skills to the organisations where fellows are employed. Long-term evaluations aimed at understanding the impact on organisations also have to account for individuals who move on in their careers.
Targeted approaches
For the CIRCLE programme the relatively small number of fellows and the short timeframe of the fellowships allows for a more in-depth, hands on approach to programme management and monitoring. The contact with CVFs and their institutions is more frequent than may be the case with larger programmes. CVFs, their supervisors and their institutions regularly report on progress, both in terms of the fellowship research as well as the CVFs’ engagement in disseminating the outcomes of their research. Additional resources such as a research support fund and a conference and training fund means that the programme managers are also able to closely track the activities undertaken by the fellows. Furthermore, the CVFs have been very active in discussing the progress of their fellowships on an online forum, which led to the development of a blog to showcase their achievements and reflections externally. These various forms of tracking ongoing progress give a rich picture of the activity of the CVFs and their engagement with the programme.
In contrast, the ISP component of the programme is, by nature, a more gradual and complex process, often involving multiple individuals and departments, and measures of outcome and impact are therefore less tangible and quantifiable than individual interventions. A number of training events have been run and as the first year draws to a close, institutions are setting up their ISP teams and preparing to assess their institutional capacity, based on the understanding they have developed over the past year. A gap analysis will inform a workshop in December on the next steps the institutions will take in strengthening their capacity.
In addition to the ongoing monitoring of the programme, surveys on application serve as a baseline to evaluate the progress of the CVFs and their institutions. A counterfactual survey has also been conducted among early career researchers who have not participated in the CIRCLE programme. These initial surveys have already thrown up some interesting insights, suggesting that the CVFs selected represent researchers already showing more activity and engagement before the start of the programme than others at their level. This will need to be taken into consideration when conducting the final evaluation of the first cohort of CVFs as well as comparing them to future ones.
In contrast to the CIRCLE programme, the CSC’s Fellowships are long established and on-going. Their evaluation was initiated in 2008 with a survey sent to all fellowship alumni since the inception of the CSFP scheme (1959) to gather data on their career trajectories and fellowship outcomes. These Fellows have subsequently been sent an additional evaluation survey between 2011 and 2015. However, recognising the importance of following up with fellowship alumni soon after the completion of their UK visit, the CSC has implemented a longitudinal approach for Fellows commencing their Fellowships in 2012 and onwards. At the same time the CSC conducted a review of its Professional Fellowships scheme which highlighted the need for data collection from other stakeholders, including the organisations which host Fellows in the UK and employers in their home countries.
Fellows now receive two evaluation surveys within the first two years after the completion of their awards. New methods have been developed to capture feedback from the UK hosts and home country employers for the Professional Fellowships scheme, with survey data collected from these groups and focus group discussions with Fellows and hosts conducted to understand experiences of both parties during the fellowship. For the Academic and Medical Fellowship programmes, baseline surveys have been developed in 2015 to understand the context within which Fellows are working prior to their fellowship and host and employer surveys will be extend to these schemes from this year. Qualitative data through interviews will also complement the surveys by providing a richer context for greater understanding in, for example, issues Fellows may face when they return to work such as barriers to application of the knowledge and skills that they gained in the UK.
The blend of methodologies employed in the evaluation of fellowships demonstrates the rich, in-depth data that can be obtained for these short programmes, as evidence beyond the experience of the fellows can be gathered by working with other key programme participants. However the range of methods also reflects some of the complexities in measuring the impact of fellowships and by capturing the perspectives of multiple stakeholders it is possible to gain a holistic understanding of the learning, institutional improvement and longer term outcomes of Fellowships programmes.
Dr Caroline Cage is CIRCLE Programme Officer at the ACU, and Shireen Quraishi is CSC Programme Officer (Evaluation) at the ACU.
‘Measuring success?’ – This blog series draws from the ACU’s experience in scholarship design, administration, and analysis, and our connections in the sector, to explore the outcomes of international scholarship schemes for higher education. New posts are published every three to four weeks, authored by experts from all around the world. Find out more
If you are interested in contributing to the series, please email [email protected]
