You are here:

  • HOME
  • 5. Measuring change

5. Measuring change

5.1 Measurement approach

Modelling the quantity of young people that will experience change

At the commencement of this project, there had been little quantitative data collected over time to indicate the changes experienced by stakeholders and the quantities of stakeholders experiencing those changes. Section 9.1 of the report provides recommendations on how this approach can be adopted to improve the measurement approach through ongoing and consistent data collection in the future. The recommended approach involves collecting quantitative data that indicates how many participants have experienced outcomes as a result of the program.

For the current analysis we developed an understanding of the changes experienced by stakeholders through stakeholder consultations, which was confirmed and refined through regular conversations with the program staff. We identified that young people progress through up to five stages of development as a result of their involvement in the program. Each stage lays the foundation for the next stage of the journey. The five stages of development for the young people are (Figure 5.1):

A. Build trust and meet immediate needs

B. Increase engagement and acceptance of boundaries

C. Increase hope and motivation

D. Exercise own agency

IMPACT. Develop a strong identity*

*This stage is not experienced during engagement with the program.

Figure 5.1: Five stages of development of young people

Graphic showing the five stages of development of young people. Outcomes experienced during engagement in the program are: A. Build trust and meet immediate needs, B. Increase engagement and acceptance of boundaries, C. Increase hope and motivation, D. Exercise own agency. The expected long-term impact of the program is to develop a strong identity.

Given the interconnected nature of change experienced by these young people, it is inevitable that the young people continue to experience change from earlier stages when they move on to later stages.

Existing program participants

To quantify and project the change that is forecast to be experienced by the program participants, information on the past and current program participants was collected. The information contained the names of the program participants, date when they started the program and date when they discontinued / graduated from the program, if applicable. This information helped to identify the stage of development that the young people reached during the time with the program and if they continue their involvement with the program, where they are likely to be in five years' time.

It is expected that some service users will not make any progress during the forecast period. This assumption was based on conversations with the young people and the program manager. For these young people, it is expected that they will continue to be engaged in the program for some time but will eventually disengage. The impact of the program on young people is neutral, and they are not counted in the analysis as stakeholders who experience change.

To quantify and project the change that is forecast to be experienced by the program participants, information on the past and current program participants was collected. The information contained the names of the program participants, date when they started the program and date when they discontinued / graduated from the program, if applicable. This information helped to identify the stage of development that the young people reached during their time with the program and if they continue their involvement with the program, where they are likely to be in five years' time. Information collected is represented in the Figure 5.2 below.

Information collected is summarised in the Figure 5.2 below.

Figure 5.2: Projected change for the current cohort of young people

Bar charts displaying projected changes in stages of development for young people after 5 years. Out of 3 young people starting in Stage A it is expected that 2 will change to C, and 1 will change to D. Out of 9 young people starting in Stage B it is expected that 2 will change to C and 7 will change to D. Out of 15 young people starting in Stage C it is expected that 10 will change to D.

Source: Helping Hand and Linking Youth data; stakeholder engagement and SVA Consulting analysis
*Some young people are not expected to move from one stage to another and therefore, it is assumed that they do not experience material change. These young people are not counted in the analysis.

Future program participants

Over the forecast period new young people will join the program. It is impossible to know exactly what the characteristics of these young people will be and what their experience will be during the program. Therefore, to project the changes experienced by this group of young people, we used information from the past and the current cohorts to estimate the length of time these young people will be in the program, how quickly they will progress through the stages of development and how much change they will experience as a result.

Analysis of the available data revealed that the young people the program works with fall roughly into three levels of need: medium needs, high needs and very high needs. These levels correspond with the time they spent involved with the program (i.e. young people with higher needs stay involved with the program for longer than young people with lower needs). The level of need also determines how much change the young people are likely to experience. Figure 5.3 depicts the classification (low, high and very high needs) and the success rate of the past and existing cohort of the young people engaged in the program. The success rate for a young person indicates the chance of them achieving the final stages of development as a result of the program.

Figure 5.3: Classification and the success rate of the past and the current cohort of young people

Image showing four stacked bar charts. The first displays the past and current cohort’s required support in years: 26% require 5+ years’ support (very high needs), 58% require 3-5 years’ support (high needs), and 16% require 1-2 years’ support (medium needs/early leavers). The other three charts show the success rate in developmental stages for the very high needs, high needs, and medium needs/early leavers groups. The medium needs/early leavers group has the greatest success rate, followed by the high needs group and then the very high needs group.

Source: Helping Hand and Linking Youth data; stakeholder engagement and SVA Consulting analysis

Over the forecast period it was assumed that 13 new young people will join the program each year. This is based on three assumptions: (i) the program is expected to accommodate 13 new young people each year from FY15; (ii) each young person experiences intensive case management for approximately 12-15 months, before transitioning to a situation where they require less support; and (iii) the level of need and success rate of the young people based on the analysis described above. Young people that join the program are expected to be at a stage of development prior to stage A. Figure 5.4 shows the change projected to be experienced by the new young people who will join the program over the next 5 years.

Figure 5.4: projected changes experienced by the new young people who will join the program over the forecast period

Image showing two bar charts. The first shows that 13 new young people are expected to join the program each year from FY15 to FY19. The second projects the change experienced by these new people over 5 years. Most will change to Stage C.

Source: SVA Consulting analysis

We modelled where each of the young people in the program would be along the stages of development at the beginning of the investment period (the beginning of FY15) and the end of the investment period (the end of FY19) based on their length of time in the program and our understanding of how the young people progressed through the program. We then grouped the young people according to how far they had progressed through the stages, and therefore the changes they had experienced. Each of the young people in the program falls into only one of these groups, which means that no double counting occurs.

Modelling the quantity of other stakeholders that will experience change

To understand if the change will occur for other stakeholders we used both the information collected through the stakeholder consultation, as well as the data collected by the organisation as part of their reporting requirements to the Department.

Family and significant others

Based on consultations with the program manager and interviews with family members and significant others, it was identified that approximately 53% of families and significant others of the young people have experienced increased communication as a result of the program and 73% have experienced increased engagement in the lives of the young people. These changes are likely to happen to the families and significant others where the young person has successfully reached Stages C and D of the development journey.

Community

Based on the consultations with the program managers and interviews with the community members, it was identified that all community members that directly engage with the young people during the program experience the change.

It is assumed that the number of the community members who are engaged with the young people will increase in line with the growth in the number of the young people in the program.

Justice System

The amount of change experienced by the justice system is related to the number of young people that change and how they change. Indicators capture the number of young people that experience change when they have just joined the program, during the program and after the program, as well as the expected benefits to the justice system during each of these periods of time.

For example, when the young people join the program they are usually already facing charges for an offence. The program manager supports them in court and in most cases these young people avoid going to juvenile detention. Each of these episodes provides a direct benefit to the justice system and therefore is counted separately.

When estimating the likelihood of young people not offending during or after the program, data collected by the SVA Consulting and anecdotal evidence from the program staff on the offence rates of their clients has been used to inform our quantity estimates.

Indicators of change

Both objective and subjective outcome indicators were identified during stakeholder consultation. An indicator is credible if it can demonstrate that the outcome will be achieved. A mixture of subjective and objective indicators allows the creation of a more robust measurement. This information was used to develop a tool for the program team to collect data to prove that the outcomes are happening.

Tables 5.1 and 5.2 summarise the indicators used to forecast the outcomes for this SROI analysis.

Table 5.1 Indicators for Young people stakeholder group outcomes - 1. Young people
Change in stage of development Indicator
Outcome 1.1 Increased self-esteem 		Indicator
1.2 Increased engagement in meaningful activity 		Indicator
1.3 More positive connections to others 		Indicator
1.4 Reduced likelihood of detention or incarceration in the future 		Quantity
5Y Forecast

# young people that joined the program during the investment period and will reach Stage A

# young people whose self-esteem will increase from extremely low to very low during the investment period

# young people whose engagement in a meaningful activity will increase from not being engaged at all to being engaged to a very small extent during the investment period

# young people whose connections increase from extremely weak to very weak during the investment period

# young people whose likelihood of detention/incarceration will reduce from an extremely large likelihood to a large likelihood during the investment period

3

# young people that joined the program during the investment period and will reach Stage B

# young people whose self-esteem will increase from extremely low to low during the investment period

# of young people whose engagement in a meaningful activity will increase from not being engaged at all to being engaged to a small extent during the investment period

# young people whose connections increase from extremely weak to weak during the investment period

# young people whose likelihood of detention/incarceration will reduce from an extremely large likelihood to a moderate likelihood during the investment period

8

# young people that joined the program during the investment period and will reach Stage C

# young people whose self-esteem will increase from extremely low to below average during the investment period

# of young people whose engagement in a meaningful activity will increase from not being engaged at all to being engaged to a moderate extent during the investment period

# young people whose connections increase from extremely weak to fairly weak during the investment period

# young people whose likelihood of detention/incarceration will reduce from an extremely large likelihood to a small likelihood during the investment period

33

# young people that joined the program during the investment period and will reach Stage D

# young people whose self-esteem will increase from extremely low to average during the investment period

# of young people whose engagement in a meaningful activity will increase from not being engaged at all to being engaged to a large extent during the investment period

# young people whose connections increase from extremely weak to average during the investment period

# young people whose likelihood of detention/incarceration will reduce from an extremely large likelihood to a very small 1likelihood during the investment period

20

# young people that will move from Stage A to Stage B during the investment period

# young people whose self-esteem will increase from very low to low during the investment period

# of young people whose engagement in a meaningful activity will increase from being engaged to a very small extent to being engaged to a small extent during the investment period

# young people whose connections will increase from very weak to weak during the investment period

# young people whose likelihood of detention/incarceration will reduce from a large likelihood to a moderate likelihood during the investment period

0

# young people that will move from Stage A to Stage C during the investment period

# young people whose self-esteem will increase from very low to below average during the investment period

# of young people whose engagement in a meaningful activity will increase from being engaged to a very small extent to being engaged to a moderate extent during the investment period

# young people whose connections will increase from very weak to fairly weak during the investment period

# young people whose likelihood of detention/incarceration will reduce from a large likelihood to a small likelihood during the investment period

2

# young people that will move from Stage A to Stage D during the investment period

# young people whose self-esteem will increase very low to average during the investment period

# of young people whose engagement in a meaningful activity will increase from being engaged to a very small extent to being engaged to a large extent during the investment period

# young people whose connections will increase from very weak to average during the investment period

# young people whose likelihood of detention/incarceration will reduce from a large likelihood to a very small likelihood during the investment period

1

# young people that will move from Stage B to Stage C during the investment period

# young people whose self-esteem will increase from low to below average during the investment period

# of young people whose engagement in a meaningful activity will increase from being engaged to a small extent to being engaged to a moderate extent during the investment period

# young people whose connections will increase from weak to fairly weak during the investment period

# young people whose likelihood of detention/incarceration will reduce from a moderate likelihood to a small likelihood during the investment period

2

# young people that will move from Stage B to Stage D during the investment period

# young people whose self-esteem will increase from low to average during the investment period

# of young people whose engagement in a meaningful activity will increase from being engaged to a small extent to being engaged to a large extent during the investment period

# young people whose connections will increase from weak to average during the investment period

# young people whose likelihood of detention/incarceration will reduce from a moderate likelihood to a very small likelihood during the investment period

7

# young people that will move from Stage C to Stage D during the investment period

# young people whose self-esteem will increase from below average to average during the investment period

# of young people whose engagement in a meaningful activity will increase from being engaged to a moderate extent to being engaged to a large extent during the investment period

# young people whose connections will increase from fairly weak to average during the investment period

# young people whose likelihood of detention/incarceration will reduce from a small likelihood to a very small likelihood during the investment period

10

Table 5.2 - Indicators for other stakeholder group outcomes - 2. Families and Significant Others
Outcomes Indicator Quantity
5Y Forecast

2.1 Improve communication between family members

# of families and significant others which program managers report as having improved communication

46

2.2 Increase engagement in lives of the young people in their care

# of families and significant others where program managers report as having increase engagement in lives of young people in their care

63

Table 5.2 - Indicators for other stakeholder group outcomes - 3. Community
Outcomes Indicator Quantity
5Y Forecast

3.1 Improved perceptions of young people

# of community Elders and other members of the community that report that they are directly engaging with the young people and improving their perception of them

# of community Elders and other members of the community that the program manager reports are directly engaging with the young people and improving their perception of them

38

Table 5.2 - Indicators for other stakeholder group outcomes - 4. Justice system
Outcomes Indicator Quantity
5Y Forecast

4.1 Reduction in anti-social behaviour

# young people that have reduced the frequency of anti-social behaviours they engage in

36

4.1 Reduction in anti-social behaviour

# young people that have exited the program who are not engaging in anti-social behaviours after the program

36

4.2 Decreased number of young people offending

# of young people not offending during the program

20

4.2 Decreased number of young people offending

# of young people not offending after the program

20

4.3 Decreased number of young people in detention

# of young people avoiding detention immediately after joining program

58

4.3 Decreased number of young people in detention

# of young people not in detention during the program

18

4.3 Decreased number of young people in detention

# of young people not in detention after the program

26

This forecast SROI analysis was used to provide guidance to the Department and the Helping Hand and Linking Youth program staff about what outcomes should be measured in the future, and what the indicators of the outcomes experienced by the stakeholder groups should be. A Social Outcomes Measurement and Evaluation Tool has been developed for this purpose. Table 5.2 shows the indicators that will be used as part of the Social Outcomes Measurement and Evaluation Tool. Both subjective and objective indicators are included. This will facilitate more robust evaluation of the program in the future than has been possible in this analysis.

Table 5.3 - Indicators used in this analysis and in the Social Outcomes Measurement and Evaluation Tool - 1. Young people
Outcomes Questions in the Social Outcomes Measurement and Evaluation Tool Response options Respondent

1.1 Increased self-esteem

Rate your level of agreement with the following statements12:

  • I feel that I am a person of worth, at least on an equal plane with others.
  • I feel that I have a number of good qualities.
  • All in all, I am inclined to feel that I am a failure.
  • I am able to do things as well as most other people.
  • I feel I do not have much to be proud of.
  • I take a positive attitude toward myself.
  • On the whole, I am satisfied with myself.
  • I wish I could have more respect for myself.
  • I certainly feel useless at times.
  • At times I think I am no good at all.

Strongly agree

Agree

Disagree

Strongly disagree

Responses automatically tally into the following self-esteem scale:

High

Normal Low

Very Low

Young person

1.2 Increased engagement in meaningful activity

What is the core activity that the young person is engaged in?

At school

Further studying/training

Working

Looking for work

Looking after children

Not currently engaged in a meaningful activity

Other

Young person

1.2 Increased engagement in meaningful activity

How would you rate their level of engagement with the core activity?

Very engaged Somewhat engaged Struggling to engage Not engaged

Program manager

1.3 More positive connections to others

Rate your level of agreement with the following statements13:

  • People don't come to visit me as often as I would like
  • I often need help from other people but can't get it
  • I seem to have a lot of friends
  • I don't have anyone that I can confide in
  • I have no one to lean on in times of trouble
  • There is someone who can always cheer me up when I'm down
  • I often feel very lonely
  • I enjoy the time I spend with the people who are important to me
  • When something's on my mind, just talking with the people I know can make me feel better
  • When I need someone to help me out, I can usually find someone

Strongly agree

Agree

Disagree

Strongly disagree

Responses automatically tally into the following connection scale:

High;

Normal;

Low;

Very Low

Young person

1.4 Reduced likelihood of detention or incarceration in the future

Exhibited anti-social behaviour in the last six months

Never

Very rarely

Occasionally

Frequently

Very frequently

Young person

1.4 Reduced likelihood of detention or incarceration in the future

Committed a criminal offence in the last six months

No

Property offence

Violent offence

Other - minor offence

Other - major offence

Young person

1.4 Reduced likelihood of detention or incarceration in the future

Sentenced to community-based supervision in the last six months

No

Yes

Waiting to find out

Young person

1.4 Reduced likelihood of detention or incarceration in the future

Sentenced to detention in the last six months

No

Yes

Waiting to find out

Young person

1.4 Reduced likelihood of detention or incarceration in the future

Has been in detention / prison in the last six months

No

Yes - most of the time

Yes - some of the time

Young person