1. Targeting Labour Market Programs Towards the Disadvantaged An Analysis of Streaming in Job Services Australia
Patrick Hamshere
and
Lixin Cai
jobactive Evaluation Section
Evaluation, Research and Evidence Branch
Australian Government Department of Employment
Disclaimer: the views expressed herein are those of the authors and do not necessarily reflect the views of the Australian Government Department of Employment. The analysis in this presentation is preliminary and subject to change.

2. Overview Background and motivation Policy environment
Methodology and data
Preliminary results
Concluding remarks

3. Background
Active Labour Market Programs (ALMPs) in Australia typically vary a job seekers level of treatment (a “service intensity”) depending on their degree of labour market disadvantage.
A way of targeting resources within a program to where it will be utilised most effectively.
True for the current program (jobactive) and its predecessor model (Job Services Australia; JSA).

4. This Paper
This paper is concerned with measuring the effect of different service intensities (i.e. targeting of services) on labour market outcomes in the second contract period of JSA (2012 to 2015).
Uses Regression Discontinuity Design (RDD) to measure effects as a whole and by subgroup:
income support type
gender
age.

5. Motivation
To measure ‘how much’ labour market outcomes improved with the increase in service intensity.
Implications for value for money / justify spending.
Provides insight on how ALMPs can achieve labour market outcomes more broadly and for whom.
Helps to identify subpopulations where this type of targeting is/is not effective – helps to ‘reconfigure’ our targeting instruments.

6. JSA Policy Environment
Each person who registered as a job seeker was placed into one of four “Streams”.
Streaming was the primary instrument used to target assistance in JSA.
Essentially, a job seekers Stream represented their degree of service intensity.

7. JSA Policy Environment
Job seekers were allocated to Streams based on their difficulty in gaining and maintaining employment.
Allocation to Streams 1-3 depended on their score from Jobseeker Classification Instrument (JSCI).
Allocation to Stream 4 depended on a further assessment to establish they had personal issues (e.g. alcohol or drug problem).

8. JSA Policy Environment
The JSCI questionnaire was used to identify job seekers, at the time of service registration, with the greatest risk of long-term unemployment.
It is based off a logistic regression model that estimated the relative weights of the risk factors that have been identified by policy makers as being associated with long-term unemployment.
By combining the weights and the risk factors, a job seeker gets a JSCI score.

9. JSA Policy Environment
We focus the analysis on the first three Streams.
RDD can only measure the effect for those Streams where allocation was a function of the JSCI score only.
More on this later.

10. JSCI Scores and Streams1 76
Least disadvantaged
Most disadvantaged
Each job seeker received a JSCI score.
Set of integers: {1…76}.
Ordered by degree of labour market disadvantage.

11. JSCI Scores and Streams19 20 28 29 1 76
Least disadvantaged
Most disadvantaged
Allocation to Streams 1-3 depended on the JSCI cut-off scores.

12. Stream Differences Maximum outcome fee ($) Maximum service fee ($)
Employment Pathway Fund ($)
Stream 1
1,258
581 11
Stream 2
2,064
885
550
Stream 3
5,880
1,120
1,100
Outcome fees: paid to providers for an employment outcome.
Service fee: paid to providers to service job seekers.
Employment Pathway Fund: expenditure for training.

13. Average Expenditure per Job Seeker in Their First 12 Months of JSA

14. Methodology
We use a Regression Discontinuity Design (RDD) to estimate the effect of Stream placement on outcomes.
Outcome used is the off income support rate at 12 months from commencement (other outcome rates are also checked – more on this later).

15. Methodology
RDD relies on the fact that Stream placement depends only on the JSCI score and there is a cut-off point in JSCI such that Stream allocation changes.
Underlying idea is that job seekers just below cut-off point (who did not receive treatment) are good comparisons to those just above (who did receive treatment).

16. Methodology Methodology is implemented by:
estimating the function mapping JSCI scores to outcomes
measuring the change in this function at the JSCI boundary score.
RDD controls for jobseeker characteristics because it controls for changes in JSCI score.

17. Distance from JSCI cut-off
Illustrative Example
Outcome
RDD measures difference in the function (solid black line) at boundary.
Stream 1
Stream 2
Distance from JSCI cut-off
(cut-off)
Solid black line is function mapping distance to outcomes.
Dotted black line is prediction using function of other Stream.

18. Methodology
If more services and assistance causes job seekers to achieve better outcomes then there should be a discontinuous changes in the level and/or slope of function at the JSCI cut-off point.
RDD relies on three underlying assumptions.

19. Assumption 1
Assumption 1: there would not have been a “jump” in outcomes at the cut-off score if not for the change in Stream.
In other words, the JSCI questionnaire accurately captured disadvantage.
This assumption was largely satisfied except for a few sub-populations (described later).

20. Outcome Rate By JSCI Score

21. Assumption 2
Assumption 2: job seekers can not perfectly manipulate their JSCI score.
We do not find evidence of manipulation (graph).
Important to note that our study population excludes job seekers who changed Streams during service (more on study population soon).

22. Job Seekers By JSCI
No evidence of a spike in job seeker numbers at boundary scores. 28 29 19 20

23. Functional Form
Assumption 3: proposed functional form is a suitable fit to the JSCI score/outcome relationship.
Most controversial assumption because true functional form is unobservable.
Estimates will be bias if this assumption is violated.

24. Functional Form
To minimize chance of choosing the wrong specification we did not rely on only one functional form.
Instead, tested variety of functional forms.
Results that were stable across equally plausible specifications were regarded as more reliable than those sensitive to minor changes in specification.

25. Functional Form
We test six specifications:

26. Ranking Functional Forms
We follow other researchers in reporting results for the specification with the lowest Akaike Information Criteria.
The AIC is a measure of the functional form’s “goodness of fit”.

27. Data
We use Department of Employment administrative data and Department of Human Services income support data. Specifically:
gender and age
income support type
JSCI scores and Stream.

28. Study Population Second contract period of JSA (2012 to 2015):
first three months of study period are excluded to reduce transition effects
last 12 months are excluded because it would not be possible to assess a jobseekers 12 month outcome rate.

29. Study Population
A number of jobseekers are excluded from analysis in order to isolate the “pure” Stream effect. Includes jobseekers with:
Stream changes during service
Streaming that was not based on JSCI scores (e.g. Stream 4)
Income support types other than Newstart Allowance (NSA) or Youth Allowance (Other) (YA(O)).
Around 400,000 job seekers in total.

30. Preliminary Results
For NSA recipients a moderate Stream effect was found regardless of functional form:
Stream 2: +6.1 ppts (robust)
Stream 3: +1.3 ppts (robust).
For YA(O) recipients the effect is smaller:
Stream 2: +0.7 ppts (robust)
Stream 3: +2.5 ppts (not robust).

31. Preliminary Results
Why was Streaming less effective for YA(O) recipients?
YA(O) recipients may have different behaviour to NSA recipients because of their lower age.
Looking at the off-IS rate may mask the Stream effect for young people. What about study?
Including movements to YA(S) in the outcome rate increased the Stream effect slightly and made it significant.

32. Male and Female (NSA) For males a large Stream effect was found:
Stream 2: +9.3 ppts (robust)
Stream 3: +5.3 ppts (robust).
For females the effect is smaller and less robust:
Stream 2: +2.9 ppts (robust)
Stream 3: -0.2 ppts (not robust)
Stream 3: +2.0 ppts (robust) when partial off-IS rate included in the outcome rate.

33. Male and Female
The results are consistent with the findings of other studies that show females’ drivers to obtain work may be very different to males.
This suggests other issues external to a concept of ‘disadvantage’ are at play when looking for work and finding work that has implications on the way to service different cohorts.

34. Age
Stream effects largest for prime-aged workers (they made up most of the NSA recipients).
For job seekers aged over 55 years old, the JSCI did not capture disadvantage accurately. Hence, assumption (1) that there are no jumps in the outcome rate at non-threshold JSCI scores not satisfied.
Identification strategy questionable for this cohort.

35. Implications
We concluded that differentiating service intensity by level of disadvantage can be important in addressing unemployment amongst the most disadvantaged.
However, such policies may require tailoring to the characteristics of individual subpopulations in order to improve their effectiveness.

36. Implications
For example, the relatively low Stream effect detected for females may be linked to the availability, affordability and quality of formal childcare.
Improvements in childcare policies may help increase the effectiveness of labour market assistance for females.

37. Future Research
Quantifying the Stream effect has implications for value for money but we have not undertaken analysis on this yet.
For example: did the reduction in the income support bill (from increased outcomes) more than offset the marginal cost of the higher Stream placement?

38. Future Research
Placebo testing to check the identification strategy. Involves implementing the RD procedure on non-threshold JSCI scores to check whether it picks up any discontinuities.

39. Questions and comments :)