9. Recommendations
On this page:
This section suggests recommendations for the Helping Hand and Linking Youth program on how to improve outcomes measurement for the future and other actions that build on the insights from the analysis.
9.1 Recommendations to improve outcomes measurement
At the commencement of this project, there had been little data collected over time to indicate the changes experienced by stakeholders and quantities of stakeholders experiencing those changes. Through stakeholder consultation, and working with the program manager, we developed an understanding of the changes experienced by stakeholders and constructed a dataset of changes experienced by stakeholders that had been involved with the program in the past.
To indicate the quantity of young people that will experience the outcomes we:
- Mapped the experience of past and current clients - start date, end date, status, stage of development reached
- Projected the stage of development reached by the current clients in the next two and five years, based on stakeholder consultation
- Grouped clients into three categories based on the length of time spent in the program
- Modelled the projected stage of development reached by future clients (based on the experience of past and current clients).
We have made the following professional judgements in our modelling of the changes young people will experience:
- If clients do not progress through one or more stages of development, they do not experience change
- If clients do progress through one or more stages of development, they experience all four of the material outcomes
- The outcomes are experienced differently across the stages of development
- 13 new young people join the program per annum
- The movement of future clients through the program will match the movement of past clients. This approach is described in detail in Section 5.1.
The analysis included in this report is reliant on estimates made by the organisation, based on the program manager's extensive knowledge of the young people. We cross-checked the estimates and assumptions with other stakeholders, including our notes from interviews with the young people themselves, and tested them against population data (where relevant). The willingness and ability of young people to engage in the consultation process (for example, due to mental incapacity or a lack of trust for the interviewers) also limited the engagement of young people in this analysis.
To improve the rigour of future analyses, Helping Hand and Linking Youth program staff should collect data on the activity delivered (outputs) and the changes experienced by stakeholders as a result of this activity (outcomes) on an on-going basis. Helping Hand and Linking Youth program staff should focus on answering these five questions:
- Who is changed?
- How do they change?
- How do you know that they have changed?
- How much is as a result of you?
- How important are the changes?
To order to answer these questions, Helping Hand and Linking Youth should track for each young person that they work with:
- Date commenced with the program (start date) and date exited the program (end date)
- Program engagement (e.g. active, inactive)
- Activities that the program provides (e.g. recreation, mentoring)
- Progress towards the achievement of each of the outcomes (increased self-esteem; increased engagement in meaningful activity; more positive connections to others; reduced likelihood of detention or incarceration in the future)
- Number of years that the outcome is experienced for
- Background (e.g. education level; employment history; mental health issues; housing situation; alcohol and drug use; offending history)
- Other support services accessed (i.e. to indicate who else is contributing to change, this will be used to calculate attribution in the future).
A Social Outcomes Measurement and Evaluation Tool has been developed as a part of this project.
Due to the nature of the program and the number of young people in the program, the Social Outcomes Measurement and Evaluation Tool was developed with the following objectives in mind:
- Simple - not over-burdensome on program staff or undermine their ability to deliver the project, including the critical relationship development aspect of the program;
- Meaningful - helping the program staff to deliver optimally against their objectives;
- Timely - allowing for regular collection and monitoring of data to allow for course corrections;
- Transparent - negative findings are identified, acknowledged and addressed;
- Context-aware - clear about how changes in the environment of the project impact on project outcomes (e.g. changes to youth justice approaches in different states and territories); and Provide value - to young people, to program staff, as well as funders.
The Social Outcomes Measurement and Evaluation Tool is designed so that the young people self-select whether they are experiencing change as a result of the program. For Outcome 1.1 (Increased self-esteem), the young people are asked to answer a questionnaire based on Rosenberg's Self-Esteem scale, which has been proven to provide accurate representation of an outcome of self-esteem. Based on these responses, the program manager can record whether the young person's self-esteem is very low, low, normal or high. Similarly, for Outcome 1.3 (Positive connection to others) the young person is asked to answer questions about their connection to the people around them which were designed with reference to The Household, Income and Labour Dynamics in Australia survey conducted by the University of Melbourne, which is widely used by Australian and International researchers and by the Australian Government. The Social Outcomes Measurement and Evaluation Tool will enable objective and subjective indicators to be used. The tool should be used by program staff to evaluate all young people in the program, including those that this analysis found to have experienced no change as a result of the program and therefore were not counted as stakeholders who experience change.
It is recommended that data be collected during the course of each interaction between the case worker and the young person and checked on intake, at six monthly intervals and at exit from the program. Regular monitoring of data could be used to demonstrate to young people the progress that they have Social Outcomes Measurement and Evaluation Tool made and to assist the case worker better plan how each young person is supported and how to balance competing needs of different young people with limited resources.
The Social Outcomes Measurement and Evaluation Tool has been developed to be integrated into other monitoring and reporting that the program manager already undertakes. Figure 9.1 indicates the proposed timing for using the Social Outcomes Measurement and Evaluation Tool (the green crosses) and how it can be used as a resource for the program manager to assist with other information collection.
Figure 9.1: Proposed timing of capturing data through the Social Outcomes Measurement and Evaluation Tool
Helping Hand and Linking Youth staff should also capture the number of families and community members that it interacts with and periodically assess the changes and the value they derive, as a result of the program. This could help to identify ways how these stakeholders should be involved in the program in the future, to generate the most value for both the young people and these stakeholder groups. This could be achieved through a short face-to-face or telephone survey conducted on an annual basis. These steps will aid in communicating the full impact that the program is having across the different stakeholder groups.
In future it may be possible to compare the outcomes achieved by the program with the outcomes achieved by similar programs, and with changes in the population. This will help to understand the relative effectiveness of the program.
9.2 Other recommended actions
As a result of the insights of this analysis, the following actions are also recommended for the Helping Hand and Linking Youth program:
Funding the program
1. Secure funding for the full costs of the program for a five year period in recognition of the time taken to generate changes for the young people involved
The young people require intense, long-term support to avoid offending or re-offending and to develop the sense of identity necessary to maintain a positive path into adulthood. The average length of time that a young person spending as an active member of the program is 12-15 months, however they can remain in contact with the Helping Hand and Linking Youth staff and continue to engage with the program for up to 5 years. To ensure continuity of support for these young people, it is recommended that the program is funded from periods greater than two years at a time, ideally for up to 5 years.
2. Seek funding from the New South Wales Government in recognition of the significant justice system resource reallocations expected to be generated by the program
The program is expected to generate significant resource reallocations for the justice system. The justice system is administered by the New South Wales Government. It is recommended that Helping Hand and Linking Youth program staff share the findings of the analysis with the state government and seek funding to support its activities. As the current primary funder, the Department could play a role in communicating the credentials of the program to the relevant section of the New South Wales Government.
Demonstrating the value of the program
3. Share insights gained from the program with other organisations focused on youth justice diversion
Stakeholder consultations indicated that Helping Hand and Linking Youth program staff are keen to learn more about the effectiveness of the program, share their experiences and learn from the experience of similar programs. As the program has many of the characteristics recognised as best practice in programs for Aboriginal youth at risk, the program staff should seek out opportunities to share the program design, implementation and impact with other IJP youth justice diversion programs. This could take the form of conference presentations, seminars, and/or participation in research studies.