Can individual outcomes lead to communal impacts? Measuring social change using longitudinal studies… A challenge and an opportunity

'A transformative experience for me and my community'. 

These are words I often hear when speaking to alumni of the Ford Foundation International Fellowships Program (IFP), a programme for 4,305 emerging social justice leaders from marginalised communities in 22 countries, which ran from 2001 - 2013. But how can we transfer these individual stories to measure the success of a global social change programme like IFP?

The Institute of International Education (IIE) is privileged to study, over ten years (2013 – 2023), how the personal trajectories of the IFP alumni can lead to impacts beyond the individual level. In more specific details, how an individual scholarship programme like IFP, which offered Masters or Doctorate education opportunities to its fellows, contributes to changes at an institutional and societal level. The IIE has recently released the first report of the IFP alumni tracking study, titled Social Justice and Sustainable Change: The Impacts of Higher Education. This report presents the findings of the first IFP global alumni survey and reflects on the responses of 1,861 alumni who participated.

Unique research hypotheses

The research questions in the study's methodology reflect the overall vision of IFP in linking higher education opportunity to social change. To be clear, our evaluation team embarked on this study with a key assumption. Having built on a comprehensive evaluation of IFP by the University of Twente's Center of Higher Education Policy (CHEPS) that culminated in 2013, we began the tracking study with the assumption that the programme successfully addressed the individual needs of IFP fellows. This was the first research hypothesis of IFP: that the IFP fellowship opportunity supported the participants in developing their role as social justice leaders.

Based on the findings of the summative evaluation, we concluded that IFP succeeded in providing emerging social justice leaders from marginalised communities with the necessary knowledge and skills to advance their professional lives. CHEPS found, for example, that over 90% of the IFP Fellows had finished the advanced degree they had pursued as part of IFP, and 83% of IFP alumni indicated that they were applying the knowledge learned during IFP in their professional work.

With this assumption in place, we could move on to the second research hypothesis: that the acquisition of knowledge and skills by the IFP fellows would in turn lead to social change beyond the individual sphere of influence. As such, our team designed a theory of change specifically around the transformative impact of IFP. We decided to focus on how the individual change within each fellow could lead to a multiplier effect that went beyond the individual. We chose to expand Donald Kirkpatrick's 'Levels of Evaluation' to study this multiplier effect.

Source: Social Justice and Sustainable Change: The Impacts of Higher Education

What have we found thus far?

'Programs that target individuals can nonetheless have significant multiplier effects for communities, societies, and organizations.'

Perhaps the most important finding in Social Justice and Sustainable Change is evidence that a fellowship programme like IFP can have multiplier effects. Examples include:

• IFP alumni report that new programmes and organisations they have created, which relate to social justice, have impacted an estimated 9.5 million adults and children in IFP countries and 860,000 additional individuals worldwide.
• 84% of alumni report that they have made improvements in the organisations where they work or volunteer, impacting an estimated 68,000 employees and volunteers worldwide.
• IFP alumni have created nearly 34,600 products and forms of outreach related to social justice, including over 12,000 conference presentations and almost 15,500 books, reports, journals or news articles.

Source: Social Justice and Sustainable Change: The Impacts of Higher Education

While our study focuses specifically on the multiplier effects of IFP, we have also found significant findings related to the power of the IFP alumni network, and the reflections that IFP alumni have on themselves as social justice leaders. 88% of IFP alumni feel empowered to confront issues of injustice as a result of their IFP fellowship opportunity; and 63% indicate that others in their community look to them when advocating for social justice.

A learning opportunity

We always considered the IFP alumni tracking study as a learning opportunity for those interested in studying the long-term impacts of scholarship programmes. What key considerations can we share three years into our work?

• Studies that are committed to studying long-term impacts are difficult to implement... and as such are quite rare. We have found that we are, at times, treading into unfamiliar territory, as very few studies have taken on a longitudinal trajectory of this kind. With time, we face the challenges of attrition as IFP alumni move on and lose touch. We have found that putting a strong alumni engagement strategy in place improves our chances to continue engaging with alumni over ten years.
  
  
• Limitations cannot be underestimated. We are very cognisant of the limitations of our research, and have articulated these from the beginning. Our first report reflects the quantitative and self-reported findings of 1,861 alumni, representing 43% of the programme population. What about the other half? In 2016, we are embarking on qualitative fieldwork in each region, using local researchers – this will enable us to study the work of alumni in their countries more closely. This will help us reach alumni who may be in more remote locations, and will allow us to triangulate self-reported findings of alumni who are based in these communities.
  
  
• We want to share our story. IIE knows that this study is unique in its size and scope, and as such we are committed to sharing our experiences in the field of research and evaluation, international education and social justice. We hope that by learning about our experiences, donors and programme managers can consider the worthwhile investment of studying the long-term outcomes and impacts of scholarship programmes.

While no single statistic can be used as the only success measure of the IFP, this report contributes to a legacy that is sustained by studying the personal and professional pathways of the IFP alumni, and the potential impacts they have on their communities and society more broadly. The global themes that have emerged in this first look at the IFP global alumni survey findings indicate that, while the programme was decentralised in its design and implementation, there are success factors and challenges that unite the IFP alumni on a local, regional and global scale – indicating that both personal and communal benefits are far-reaching.

Mirka Martel is Assistant Director of Research and Evaluation at the Institute of International Education (IIE) and manages performance and impact evaluations of international fellowship and scholarship programmes, including many that seek to develop leadership skills or promote access to higher education among underserved populations. She is the Project Director of the Ford Foundation IFP Alumni Tracking Study. Read the full report here

'Measuring success?' – This blog series draws from the ACU's experience in scholarship design, administration, and analysis, and our connections in the sector, to explore the outcomes of international scholarship schemes for higher education. New posts are published every three to four weeks, authored by experts from all around the world. Find out more

If you are interested in contributing to the series, please email [email protected]