All members of our cooking skills study group are aiming to use a combination of three different evaluation sources to find out if their courses are ‘working’, by sourcing evaluation materials from 1) the course practitioner (as an ‘observer’), 2) the participant (‘self-reporting’) and 3) and a third source such as a ‘third party’ who can verify for the participant. (More information about other sources, such as quizzes and assessments can be found in blog 19) Using three different sources as a way of evaluating is known as ‘triangulation’ and results in a more robust evaluation.

In this guest blog, my colleague Jacqui discusses some of the difficulties we have experienced as a study group to define what we mean by –

third party or observer?

‘Remember we have waxed lyrical about triangulation, i.e. getting evidence from different perspectives like self-reporting, observations, testing/assessment or third party feedback? Well, the most important thing is to define these early on, be clear about how they are defined, and then consistently apply them. Here’s how we have tried to define these.

A third party is someone who:

• knows, or has an ongoing relationship with the course participant, independent and outside of the cooking skills course environment, and
• can speak with some authority about changes or differences that may be due to that participant’s involvement in the course, or / and
• can confirm the accuracy of things that participants have self-reported that are pertinent to the impact of the course or how it has been run

Let’s apply this to a few examples:

The person from a referral agency who referred the participant to the course, (and who you could interview to find out about any differences the course has made to the participant’s eating, shopping or cooking habits) = third party

A support worker who attends the course with the participant and tells you about how the participant has tried to make recipes again at home themselves = third party

Another family member you meet in the street or supermarket who gives unsolicited feedback about the course participant (e.g. how they are replicating recipes from the course again at home, or the family are enjoying more fruit and veg) = third party

A student or volunteer who sits in on course and makes notes on what is happening on it, or about what the participants are saying = observer (who may record objective assessments or note self-reporting from participants)

A participant tells you about changes that they have made at home = self-reported feedback that you may want to note in your observation notes

And a couple of tricky ones:

A support worker who attends the course and you hear talking with the participant about how they could be doing this at home = something you may want to jot down in your observation notes to follow up later to see if the participant has actually tried to do it, at which point they may self-report things to you or the support worker may be able give you third party feedback

A colleague in your organisation who referred a participant to the course tells you about things the participant is saying about changes they are trying to make = while they are a referrer they are not independent from the cooking skills course environment as they are within your own organisation. They can repeat what the participant is self-reporting but cannot confirm if this is accurate, so this could be recorded as observations.

This may all be a bit pedantic and nuanced given the practical challenges you may be facing capturing evidence at baseline, during course, end of course or follow up. Triangulation is tough. Getting clear and personalised information from third parties that evidences changes can be challenging.  If you can only gather evidence from two perspectives it is important to be able to separate these out too.’

jacqueline.mcdowell@nhs.net

kim.newstead@nhs.net