‘Measuring success?’ year one – The scene, the motifs, the actors

The end of October 2016 marked the one-year anniversary of 'Measuring success?'. The blog series has been developed – or perhaps curated – without a strong editorial line. We have helped develop some themes in the blog, but have not guided the topics much beyond defining the basic parameter that posts had to be relevant to understanding the outcomes of scholarship programmes. In light of that, we felt that the anniversary post should be a reflection on the year from the editors, Matt Mawer and Sian Julian.

For those interested in the history of such projects, a word on the genesis of the blog series. To accompany a panel discussion at the 2015 NAFSA conference in Boston, Joan Dassin and I (Matt) decided to organise a small, informal side meeting of colleagues both involved with scholarship programmes and likely to already be at NAFSA in some capacity. We forewent renting the 'fiendishly expensive' projector and screen, leaving us with three hours of barely structured discussion among a group of scholarship researchers, policymakers, and administrators. The meeting was entitled 'Consolidating an emerging field of scholarship evaluation and practice' and many of those colleagues in attendance have subsequently worked together in various capacities over the intervening year and a half.

One part of the 'consolidation' was a resolution that those involved in scholarship programme policymaking, design, administration, and analysis, needed to write together more regularly and start a dialogue that could enrich the whole community. We initially conceived of a newsletter, and credit is due to our friend Robin Marsh – who later wrote for the series – for suggesting that a blog might be both better received and easier to sustain than a community newsletter. The readership of the blog that became 'Measuring Success?' has since spiralled out of all proportion to the meeting that spawned it. While only around 15 of us attended the meeting in Boston, the feeling that a much larger community would be enthusiastic to join an international dialogue about current trends in scholarship programmes has been roundly proven. A glance at the readership statistics for the series proves the point:

• In 12 months - 14 posts, 4400 views!
• The 'Measuring success?' landing page has been viewed by over 700 people (top countries: UK, USA, Canada, Indonesia, Australia, Nigeria, India, and Kenya)
• Overall readership of the blog series comes predominantly from Europe and the Americas, mostly through direct referrals (i.e. word of mouth)
• 'The impact of home-country context on scholarship outcomes' by Dr Anne Campbell is our most read post, published on 29/01/2016, with 767 unique readers, mostly from the USA (30%), the UK (19%), Canada (4%), and Albania (2%).

A question we have never fully answered within the blog series is 'why is the Association of Commonwealth Universities (ACU) so invested in this topic?'. The most immediate answer is also the most obvious: the ACU administers several scholarship programmes – including circa GBP 80m per annum in UK government funding for the Chevening, Commonwealth, and Marshall Scholarships programmes – and that gives us both a practical and reputational stake in facilitating dialogue on how these programmes can be designed, assessed, and improved. A more philosophical, and equally applicable, answer is that part of the ACU's mission is to help create dialogue between experts and other interested parties globally. Some time ago we concluded that there was a space for a more voluble and detailed discourse on approaches to understanding the outcomes of scholarship programmes: this blog series is an attempt to help fill that gap.

The shifting scene: October 2015-2016

In the run-up to the release of the Sustainable Development Goals (SDGs), a wide range of stakeholders called for increased recognition of the role of higher education in addressing global challenges, including the ACU, through the Beyond 2015 campaign. Since the SDGs were formally announced in September 2015, scholarships in particular have asserted themselves in the collective consciousness of education and development focused professionals more than ever. The SDGs have sharpened attention on the need to measure success (to borrow from the blog title) in more readily quantifiable – or, at least, identifiable – ways, starting with simply identifying what constitutes a 'scholarship' in the SDG space. Conversely, target 4b – '...By 2020, substantially expand globally the number of scholarships available to developing countries...' – is an aspiration linked to the quantity of funding available: it has little to say about the quality or the impacts of these scholarships (John Kirkland, our Deputy Secretary General, noted this in our second post).

Thus, while some thinking has begun on how to measure progress towards the goal, there is little evidence that the topics we have been examining in 'Measuring success?' – study and return experiences, brain drain employment trajectories, communal social impacts, and more – are under critical consideration. Not surprisingly, we believe focusing on the funding without focusing on the impacts is unlikely to produce a satisfactory result.

At the same time as an international agenda begins to unfold, many traditional scholarship 'donor' governments (particularly in Europe) are increasingly preoccupied with their own concerns about economic stability and the geopolitical and demographic challenges of managing migration into the continent. Others promise conditional uncertainties: the impacts of a British exit from the European Union remains largely an unknown for scholarship funding (although a search for 'Brexit' in a leading UK higher education policy blog is hardly reassuring), and who really knows what the future for international aid would look like under a US regime headed by President Trump? Commitments to scholarships have ebbed and flowed. The Norwegian government recently decided to wind up the last of its major (standalone) scholarship schemes. Financial pressures have also yielded shocks to the scholarship ecosystem (Brazil), with potentially more to come (Saudi Arabia).

Conversely, commitments from private sources seem especially plentiful. The Schwarzman Scholars boasts a USD 435 endowment from a broad base of corporate contributors, while the Knight-Hennessey Scholars at Stanford claims to be the '...largest fully endowed scholarship program in the world' (based on a USD 750m endowment). The MasterCard Foundation's large-scale African scholarship programme also continues apace, now listed as a USD 700m commitment (compared to the USD 500m announced in 2012). The Open Society Foundations has recently extended their scholarship programmes into Africa. On a smaller scale, the Atlantic Philanthropies has renewed their investment in scholarships with the Atlantic Fellows Program – perhaps drawing on their experience in the Atlantic Philanthropies Vietnam programme. The amount of private foundation and philanthropic capital flowing into scholarship programmes seems to have been particularly large – or particularly visible – in the last year.

The Institute of International Education (IIE) recently completed a baseline paper for UNESCO in which they outline (a conservative) estimate of the scholarships funded by governments of high-income countries and targeted at citizens in 'developing' countries each year. In light of the private commitments we have seen in late 2015 and 2016, we have been wondering what the equivalent estimate would be for philanthropic scholarship endowments. The IIE concluded that these sources should not be relied upon – in the sense that the commitment is unaccountable to an overseeing political authority – and should not be included in addressing the SDG target for scholarships (4b), but they remain substantial sources of funding. Their advocates are also particularly engaged in discussions of social change and intercultural bonds, especially in contrast to the turn toward an 'aid and trade' or 'soft power' agenda that seems to have characterised some of the more recent governmental outlooks on scholarships.

Musings and motifs: A year of 'Measuring Success?'

'Measuring Success?' was originally conceived and launched as a methodological blog series. Those who received early invitations to write for the series were told that the blog would address topics in how outcomes of scholarships are measured, with some attention also to substantive findings from major studies. As the year (and commissioning process) has progressed it has become increasingly clear that many 'methodological' issues are rather more conceptual than technical. Robin Marsh, for instance, has highlighted that the way we – that is, programme designers, administrators, and researchers – conceive of return trajectories post-scholarship shapes the way a successful return rate or return experience is defined. That dynamic becomes even more complex if the context in which returners are attempting to reintegrate and excel is taken into account, as both Anne Campbell and Halil Chalid have observed. While we vigorously advocate for for detailed, technical thinking about methodology (particularly given this field's propensity for one-off online surveys), it is the conceptual work and framing of outcomes that is taking the foreground as the most pressing methodological issue for scholarship researchers.

Looking back at the 14 blog posts in year one, and considering some of those currently being written, two other motifs emerge:

1. Working together?

There is clear and growing interest in how the outcomes of scholarship programmes can be assessed and evidenced, not least by the engaged readership of this blog. The frequency of semi-permanent assessment or scholarly initiatives is increasing, for example: the International Fellowships Programme 10-year tracking study; the launch of a Global Tracer Facility in Australia; the regular convening of European Higher Education and development agencies at the Donor Harmonisation Group; the forthcoming Palgrave MacMillan book on 'Scholarships for social change' (in 2017 – more to follow on this topic). Here at the ACU, we have two in-house evaluation teams for the Chevening and Commonwealth Scholarships secretariats, with ten staff directly contributing to the analysis of these programmes, plus colleagues involved in the monitoring and evaluation of CIRCLE fellowships. The status quo is thus that at any one time there are several large projects underway, including both a mix of one-off tracer studies that have been the mainstay of scholarship evaluation historically, and long-term programmes with permanent staff and an evolving portfolio of scholarship-related research activities.

One consequence of this more regular research activity is that comparative research may be possible in the future. We have seen some (albeit nascent) examples in Trevor Grigg's use of Commonwealth Scholarships data as a comparator for his analysis of Atlantic Philanthropies scholarship recipients. Whether there is appetite for further coordination and collaboration is an intriguing question. The Commonwealth Scholarship Commission and EP-Nuffic – both major administrators of scholarship programmes – have been piloting a project collating tracer survey data and building 'harmonised' basic outcome indicators (e.g. on employment trajectories) that could be used across future survey projects. The pilot was well received when presented at the Donor Harmonisation Group seminar held in Vienna from 19-21 October, but whether it is possible to establish a 'standard' for basic research topics that could be used in truly comparative studies remains to be seen (we are optimistic!).

2. Who does the research?

Another intriguing theme in commissioning for the blog series is the background of those who we invite to write. High profile analysis of scholarship programmes is typically carried out by consultancies and, in some rare cases, administrating bodies. In 2014, the CSC published a study of this literature in which almost all of the texts examined were produced by programming agencies or contracted private consultants.

Commissioning for the blog, however, we have been struck by how much detailed, attentive, and insightful analysis has been (and is being) conducted as part of doctoral research. Aryn Baxter, Anne Campbell, Halil Chalid, and Tamson Pietsch (in part) have all conducted substantive research on scholarship programmes as part of their doctoral degrees. Informally, we know of many others whose work has neither been widely publicised among the community of scholarship administrators nor found its way onto the blog (yet). We might also factor in those professionals undertaking a doctorate partly concerning scholarships, alongside their regular engagement with scholarship evaluation, such as Mirka Martel.

What should we take away from this reflection? Certainly it comes as no shock to those involved in this area that the academic and programmatic dimensions of scholarship research are often unhelpfully disconnected: this disconnect was one of the drivers behind our original meeting in Boston in 2015. Yet that perception of disconnect can also be taken too far. Aryn Baxter and Anne Campbell both have extensive (past and present) professional engagements with scholarship programmes: their doctoral research hardly 'stands alone' from their professional commitments to the field. Current doctoral students researching scholarship programmes, such as Anna Kent at Deakin University, often also have extensive engagements with administration and policymaking for scholarships.

The disconnect appears rather more in the commissioning of evaluation studies by large government agencies, which are almost always awarded to consultancies and rarely to academic research institutes or individual researchers. The large private foundations seem to have a different outlook to many government agencies in this respect: the Ford Foundation, MasterCard Foundation, and Atlantic Philanthropies have all contracted academic teams to conduct research for their scholarship programmes. These are not universal trends: the foundations still contract private research consultancies, and some government programmes work directly with academic researchers (e.g. Joel Negin and colleagues' work for the Australian Department for Foreign Affairs and Trade). Nonetheless, we might reasonably conclude that there is cause to plan another 'Boston-style' meeting, focused on building dialogue between the academic researchers and administrating agencies, with the aim of cementing ties for more active research uptake in the future.

The actors: Authors and their blog posts

In one year the blog has covered a remarkable breadth of topics, including: the design of new and old programmes; individual vs societal outcomes; soft power; brain drain; the intersection of expectations and experiences; policymaking and evidence; and much more. We have discussed scholarship programmes just beginning in 2016, and scholarship programmes that have been running for over a century. We have discussed the technical and the conceptual, the theoretical and the practical, and the past, present, and future of scholarships. You can find all of that – and more – by visiting the 'Measuring success?' homepage.

Finally, a word of thanks to all of those who have written for the series, put us in contact with potential new authors, and shared or promoted the series through your networks of friends and colleagues. The 'Measuring success?' blog remains, at heart, a community project and we can continue to rely on your community spirit and desire to collaborate as we move into our second year.

The editors: Matt Mawer and Sian Julian

Dr Matt Mawer is one of three researchers responsible for providing policy-relevant analysis of outcomes from a circa £25m p/a government-funded scholarship programme: the UK's Commonwealth Scholarships and Fellowships. In this role he has conducted both extensive quantitative and qualitative research on scholarship programme outcomes and analyses of evaluation practice within the sector, some examples of which can be found in the reports 'A study of research methodology used in evaluations of international scholarship schemes for higher education' and 'Successes and complexities: the outcomes of UK Commonwealth Scholarships 1960-2012'. In his wider role at the ACU, Matt is a member of a Commonwealth Tertiary Education Facility (CTEF) expert working group and of the Donor Harmonisation Group (DHG) of agencies and researchers involved in higher education development cooperation. Alongside Professor Joan Dassin and Dr Robin Marsh, Matt is a co-editor of the forthcoming Palgrave MacMillan book: 'International scholarships in higher education: Pathways to social change'.

Matt's background is in education and research methodology, with expertise in UK higher education, international education, and educational technology. He received a PhD from Coventry University, UK, working on a Leverhulme Trust-funded project examining the socio-political impact of virtual worlds in higher education. He subsequently worked on projects funded both by the UK's Quality Assurance Agency and by the European Union's DAPHNE research strand, and is now Senior Research Officer at The Association of Commonwealth Universities.

Sian Julian is Communications Officer at the ACU, with responsibility for editing the 'Measuring success?' blog series – her first foray into the world of scholarship evaluation (and one which has turned out to be fruitful and interesting). Prior to joining the ACU, Sian worked in various communications roles, including a spell in local government and, more recently, managing the social media and digital channels for a food/cookery author, where writing and editing blogs was a substantial part of her day-to-day existence!