How it works

What is evaluative benchmarking?

Evaluative benchmarking is a self-improvement evaluation for organisations. It allows them to identify their comparative strengths and weaknesses, and identify possible routes to improvement. It is a way of finding and adopting good practices with a view to improving management processes.

The ACU approach to this goes beyond the comparison of data-based scores or conventional performance indicators (SSRs, unit costs, completion rates, etc); it looks at the processes by which results are achieved. By using a consistent approach and identifying processes which are generic and relevant, regardless of the context of the organisation and how it is structured, it becomes possible to compare across sector boundaries (geography, size, research/teaching balance, etc).

‘Process’ is used here to mean a sequence of activities made up of tasks or steps that cross boundaries between an organisation’s various functions. Focusing on these processes helps to understand how well the institution delivers effective services. The strength of benchmarking lays in the way it leads organisations to question the impact of such boundaries on the effectiveness of their processes. It is most effective when it is ongoing and becomes part of the annual review of a university’s performance.

What is the methodology?

The programme involves carrying out an initial institutional self-review process, using frameworks prepared by the programme team and the specialist assessors (the latter are appointed for their international expertise in each area under review). Each university participating will receive a handbook, incorporating guidance notes for completing those reviews.

Participating universities prepare responses (primarily using existing material), which are then submitted and evaluated by the topic assessors with assistance from the Programme Manager. Prior to the workshop, a set of good practice statements are produced, derived from the responses and the assessors’ comments, which then form the basis and framework for the discussions.

Each university is also asked to prepare a brief environmental scan of the major factors which it is currently facing. These will be used by assessors and participants in the workshop discussions of common challenges and opportunities, and provide contextual data to enable a judgement to be made about ‘fitness for purpose’ in respect of the chosen management approaches that are presented.

The framework process itself is a valuable change management tool, as it focuses attention on the issues that are relevant, and causes the operational managers to reflect on whatever they are doing.

The methodology is designed to encourage a robust debate of the key management issues at the workshop, where senior representatives are engaged from the participating universities in high-level strategic issues and policy discussions.

How is the process assessed?

The assessment process has been refined over the years that the programme has been in operation. It is based on the process used by the European Quality Awards scheme, which considers the following:

• Approach – defined as the policy or technique adopted, and whether it is right for the task (fitness for purpose)
• Application – defined as the extent to which this policy or technique is applied across the university
• Outcome – defined as how successful the approach is at achieving the objectives, but also encompassing the extent to which the approach is monitored to ascertain whether it continues to be fit for purpose, or whether it requires adaptation to meet changed circumstances

There has to be latitude in assessing the different approaches to meeting the challenges, given the diversity of participating universities and the contexts in which they are located. ‘Fitness for purpose’ will be an essential factor in assessing the appropriateness of the chosen approach.

Consequently, the assessors will be looking to see whether a clear approach is defined and used and whether it achieves the desired results, rather than using a predetermined list of standards for specific techniques, structures, or tools. A five-level rating system is used to indicate how they perceive a university has performed in each process under review.

What does the programme entail?

Applications for the programme are open From December and selection and notification of participants will be shortly thereafter. A handbook will be sent to participating universities in order to prepare the necessary information for evaluation. Participants will need to provide information on the basic management processes that are under review, and the amount of work involved will depend on the size and diversity of the university, and the extent to which present procedures are well documented.

As the evaluation process is based on how universities operate at present, participants should not have to generate or collect much new information; the work will focus on describing the processes currently in place. In addition, participants may need to answer certain specific queries from the assessors when they review institutional submissions.

After the workshop, each university is also invited to declare a self-assessment mark (using a simple 1-5 scale) against each good practice element. This is intended primarily for use as a guide to other participants, enabling them to make contact and collaborate with a colleague from a university professing particular strength and experience in a topic, if they are seeking to make improvements in their own approach.

All participating universities are required to send at least one (but no more than three) delegates to attend the face-to-face workshop within their cohort year in order to complete the programme.

How many participants?

In order to ensure participative and engaging discussion, there is a minimum of eight and a maximum of 15 universities in any one cohort which meets face to face. To promote a diverse spread of international experience, the number of universities from any single country may be restricted.

If more than 15 universities are accepted on the programme, the ACU may run an additional cohort with the same topics and assessors for both groups which would meet in October of the cohort year. In this case, cohort allocation will be at the discretion of the ACU.

Fees, course materials, and outputs

The fee per university for the programme is GBP 7,000 (for ACU member institutions) and GBP 10,500 (for non-ACU member institutions) and is independent of how many topics are undertaken. This fee includes participation in the three-day workshop, but excludes flights, airport transfers, and accommodation (although accommodation will be booked by the workshop organisers at a designated hotel at an agreed special conference rate). Participation in the programme brings the following benefits and services:

• A handbook from the ACU setting out the methodology and detailed frameworks to guide the university in its self-review.
• An initial draft report, following the self-review, giving written comments from the assessors and listing the key elements of good practice.
• Participation in the August (or October) workshop, to hear presentations by the exponents of good practice, consider the key issues, and agree on good practice in the processes under review.
• The opportunity to contribute to future years’ programmes in terms of scope, and process.
• Interactive discussion and advice from your cohort on issues arising from the evaluative benchmarking process.
• The final report from the ACU which includes the statements of good practice which can be used to help managers enhance practice within their own universities.

Universities participating in the programme are also able to buy copies of topic reports from previous years (up to a maximum of two reports for each year that they participate, at a cost of GBP 500 per topic). The list of all topics covered is set out in the handbook.

Confidentiality of data collected

The programme operates on the basis of being an international collaborative activity through which universities can use benchmarking methodologies to examine their own performance in a comparative context.

It follows that universities joining the programme must be prepared to allow other participants to see some of the results, subject to confidentiality rules. Indeed, without sharing these results it would be difficult to interpret comparative performance in the international context. In this respect, the programme differs from some national benchmarking services where competitor institutions do not share information, but measure their performance against the range and distribution of other scores.

All data collected will be the property of the ACU, and all prospective participants will be asked to ensure confidentiality. Of course, vice-chancellors and other executive officers will be able to inform their own governing bodies about relevant data, providing it is not subsequently made available for external use.