DES Performance Framework Review Working Group



DES Performance Framework Review Working GroupAgenda item:2Date of Meeting:25 September 2012Title of Discussion Paper: Post-Program Monitoring SurveyPurposeTo inform the Working Group about the DES Post-Program Monitoring survey and the collection of JSA participant experience data BackgroundPost-Program Monitoring SurveyThe Post-Program Monitoring (PPM) survey has operated since 1987 and is primarily used to measure the labour market status of job seekers who received assistance through DEEWR-funded employment programs, including DES, with around 400,000 surveys sent to job seekers each year.PPM surveys are sent to DES participants throughout the year with more than 50,000 surveys sent in 2011–12 alone. Two PPM survey types are currently used for DES, Employment Assistance/Post-Placement Support, and Ongoing Support.DES achieves a relatively high 37.5 per cent response rate across both DES PPM survey types (see Attachment A: Post-Program Monitoring survey overview for further details).During the development of the DES PPM surveys, the department engaged an external consultant with experience in the disability services sector to undertake cognitive testing with a range of DES participants to ensure the appropriateness of the questions asked of the DES client base (see Attachment B: Cognitive Testing for DES PPM survey for further details). Job Services Australia Participant Experience Measure (KPI 3)For Job Services Australia (JSA), the established PPM survey was selected as the most appropriate method to collect data for the JSA Quality KPI measure of Participant Experience. The PPM survey underwent cognitive testing and a pilot test to ensure the appropriateness of the additional questions asked and that they were understood by participants. Currently, the scores derived for the JSA KPI3 Participant Experience Measure are based on job seeker responses to questions about the services they’ve received as set out in the JSA Service Guarantee and the job seekers’ extent of satisfaction or dissatisfaction with their Employment Pathway Plan being tailored to their needs. The method for calculating the final scores for the JSA KPI3 Participant Experience Measure were developed by an external consultant, with expertise in developing experience measures (see Attachment C: Job Services Australia KPI3 Participant Experience measure for further details). IssuesThe production of robust DES provider-level PPM results is currently limited. Any increase to the existing sample sizes will increase survey costs. Given that a number of DES Contracts are relatively small, it cannot be guaranteed that all Contracts will be able to get a statistically reliably and robust assessment through the PPM survey. Other attachmentsAttachment D: Sample DES PPM survey cover letterAttachment E: Sample DES Employment Assistance/Post Placement Support PPM surveyAttachment F: Sample DES Ongoing Support PPM surveyFOR NOTING AND DISCUSSIONAttachment A: Post-Program Monitoring survey overviewThe Post-Program Monitoring (PPM) survey has operated since 1987. The survey is used to measure the labour market status of job seekers who received assistance through DEEWR-funded employment programs with around 400,000 surveys sent to job seekers each year.Aggregate survey results are used for:reporting against the effectiveness indicators for employment services in the Portfolio Budget Statements and the department’s Annual Report reporting in the Labour Market Assistance Outcomes reportinput into the KPI 3 (Quality) under the Job Services Australia Performance Frameworkemployment program evaluationpolicy development.The PPM survey is conducted on an ongoing basis and every two weeks a new cohort of participants is selected for inclusion in the survey. This approach makes the large sample easier to administer, and allows seasonal variation to be accounted for. Additionally, it decreases the likelihood of running out of samples as individuals enter and leave the system over the course of the year.The following process has been adopted to achieve the highest possible response rate:A survey is mailed to participants with a reply paid envelope and a letter from the department explaining the purpose of the survey and encouraging response Three weeks following the first mail-out, a reminder letter (including a replacement questionnaire and reply paid envelope) is sent to participants who have not yet responded Three weeks following the second letter, non-responding participants are called and invited to complete the survey over the telephone.Special arrangements are in place for visually impaired participants with a paper-based survey sent, allowing for them or a nominee to complete on their behalf. Reminder notices are not sent to visually impaired participants, but they are called sooner if a survey hasn’t been received. Participants are considered in-scope for inclusion in the survey three months after various events, eg. participation in an activity, exiting a stream, exiting assistance, being placed in a job, etc... though it varies between employment programs. Participants may be included in the PPM survey while they are still receiving assistance from a provider. For DES:A one-in-three sample selection is applied with two survey types used:A job seeker will be recorded as being in-scope for a DES Employment Assistance/Post Placement Support PPM survey if they:exit from DES, including achieving a 26 week outcome and exiting as a fully independent worker;achieve a 26 week outcome and progress into Ongoing Support; orreach 12 months in DES assistance without having progressed into Ongoing Support.A job seeker will be recorded as being in-scope for a DES Ongoing Support PPM survey if they:exit from DES (and were participating in the Ongoing Support phase); orreach 12 months in Ongoing Support.More than 50,000 DES PPM surveys were sent in 2011–12 with relatively high response rate. More than 37 per cent of Employment Assistance/Post Placement support surveys sent were completed by participants. This level of response is largely consistent across most disability types. PPM surveys sent to Ongoing Support participants have a slightly higher response rate of 41 per cent. Tables A.1 and A.2 provide further detail. Table A.1: DES Employment Assistance/Post Placement Support response rates – 2011–12Disability typeResponse rateSurveyedResponsesReturn to senderTotal in-scope populationPsychiatric38.6%25,9149,83540483,110Physical/diverse34.7%18,1266,17234858,371Intellectual36.5%1,998715416,700Learning disability31.6%3,063951569,960Sensory/speech35.9%2,8421,007379,126All participants37.1%45,64416,625790147,075Table A.2: DES Ongoing Support response rates – 2011–12Disability typeResponse rateSurveyedResponsesReturn to senderTotal in-scope populationPsychiatric40.1%2,270898328,413Physical/diverse38.4%1,758665246,361Intellectual39.5%36214151,337Learning disability35.6%481167121,757Sensory/speech38.5%36413931,283All participants41.2%5,5862,2687819,980Attachment C: Job Services Australia KPI3 Participant Experience measureGiven that the PPM survey is a well-established process, it was decided that gauging a measure of JSA participant experience through the PPM survey was the most appropriate method. Development The department engaged an external statistical consultant, with expertise in developing experience measures. The consultant advised on the appropriateness and weighting of the Quality KPI questions to provide the best possible indication of the participant’s experience. To enable calculation of final measures that are equitable to providers and allow the compilation of responses from different streams, statistical methods are used. These sophisticated methods utilise international best practice for these types of measures. The survey questions and calculations of the final scores were developed by the consultant with input from departmental statisticians. Furthermore, as the survey is used to measure participants’ experiences and is included in the Quality Framework data, detailed investigation about the actual sample design, questionnaire design, participant recall and other methodological factors were taken into consideration. This helps to ensure the collection is undertaken in a uniform manner, on a level playing field, and that results obtained from the PPM Survey are statistically robust and consistent for all providers. The additional survey questions that were included in the PPM survey underwent a two stage development phase. The development phase involved cognitive testing of the questionnaire and an eight week pilot test of the entire survey process. AssessmentThe objective of the PPM survey is to be able to give accurate feedback of the receipt of services by participants in accordance with the Service Guarantees and Code of Practice on an ESA level. Providers receive two scores against the Participant Experience measure and are based on survey responses for that Provider, derived from the: Participants surveyed at the ESA level that articulate they have received services as set out in the relevant Service Guarantee; and Participants surveyed who state their EPP was satisfactorily tailored to their needs. The two scores are calculated independently and are represented on a 0 to 100 point scale. It is important to note that the scores are calculated through a statistical process, and are not percentages.The results obtained from the survey are used for comparative purposes allowing providers to benchmark themselves against other providers and the industry as a whole. The scores are also comparable over time, and across Job Services Australia contracts. Specifically for the KPI3 Participant Experience measure, a ‘top up survey’ is conducted for participants that may have not otherwise triggered a PPM survey to ensure that statistically valid results are available for each provider in each ESA. This survey sampling methodology has also been reviewed by an external statistical consultant.The number of responses required for a particular provider at the ESA level varies depending on the number of participants who receive services from that provider within the ESA. The department only assesses providers on this measure where the results obtained through the survey are statistically robust. THE NEXT FEW QUESTIONS ARE ABOUT <PROVIDER NAME>Did <Provider Name> provide or offer to provide you with any of the following? (Tick “No, not needed” if not provided and not needed.)YesNoNo, not neededHelp to prepare or update your resumeSYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"Information about job vacancies in your local areaSYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"Access to computer facilitiesSYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"Advice about how to look for workSYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"Access to counselling or other professional supportSYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"And did staff at <Provider Name>:(Tick “No, not needed” if not provided and not relevant for your situation.)YesNoNo, not neededClearly explain the types of help they will give you?SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"Discuss any jobs you have had in the past?SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"Discuss the types of jobs that you might like in the future?SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"Discuss your skills and education?SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"Discuss extra skills and education that might help you get work?SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"Did you sign a document called an Employment Pathway Plan (EPP) with < Provider Name >? (A sample EPP has been provided on the back of the attached letter.)SYMBOL 168 \f "Wingdings"YesSYMBOL 168 \f "Wingdings"No GO TO F5SYMBOL 168 \f "Wingdings"Don’t know GO TO F5How strongly do you agree or disagree that the Employment Pathway Plan (EPP) you signed with them was tailored to you?Strongly agreeAgreeNeither agree nor disagreeDisagreeStrongly disagreeDon’t knowSYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"Overall, how satisfied or dissatisfied are you with the assistance they gave you in terms of:VerysatisfiedSatisfiedNeither satisfied nor dissatisfiedDissatisfiedVerydissatisfiedProviding help suited to your circumstances?SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"Staff treating you as an individual?SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"Staff treating you with respect?SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"Overall quality of service?SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings"SYMBOL 168 \f "Wingdings" ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download