What Works? A Meta Analysis of Recent Active Labor

RUHR
ECONOMIC PAPERS
David Card
Jochen Kluve
Andrea Weber
What Works?
A Meta Analysis of Recent Active Labor
Market Program Evaluations
#572
Imprint
Ruhr Economic Papers
Published by
Ruhr-Universität Bochum (RUB), Department of Economics
Universitätsstr. 150, 44801 Bochum, Germany
Technische Universität Dortmund, Department of Economic and Social Sciences
Vogelpothsweg 87, 44227 Dortmund, Germany
Universität Duisburg-Essen, Department of Economics
Universitätsstr. 12, 45117 Essen, Germany
Rheinisch-Westfälisches Institut für Wirtschaftsforschung (RWI)
Hohenzollernstr. 1-3, 45128 Essen, Germany
Editors
Prof. Dr. Thomas K. Bauer
RUB, Department of Economics, Empirical Economics
Phone: +49 (0) 234/3 22 83 41, e-mail: [email protected]
Prof. Dr. Wolfgang Leininger
Technische Universität Dortmund, Department of Economic and Social Sciences
Economics – Microeconomics
Phone: +49 (0) 231/7 55-3297, e-mail: [email protected]
Prof. Dr. Volker Clausen
University of Duisburg-Essen, Department of Economics
International Economics
Phone: +49 (0) 201/1 83-3655, e-mail: [email protected]
Prof. Dr. Roland Döhrn, Prof. Dr. Manuel Frondel, Prof. Dr. Jochen Kluve
RWI, Phone: +49 (0) 201/81 49 -213, e-mail: [email protected]
Editorial Oﬃce
Sabine Weiler
RWI, Phone: +49 (0) 201/81 49 -213, e-mail: [email protected]
Ruhr Economic Papers #572
Responsible Editor: Jochen Kluve
All rights reserved. Bochum, Dortmund, Duisburg, Essen, Germany, 2015
ISSN 1864-4872 (online) – ISBN 978-3-86788-658-1
The working papers published in the Series constitute work in progress circulated to
stimulate discussion and critical comments. Views expressed represent exclusively the
authors’ own opinions and do not necessarily reﬂect those of the editors.
Ruhr Economic Papers #572
David Card, Jochen Kluve, and Andrea Weber
What Works?
A Meta Analysis of Recent Active Labor
Market Program Evaluations
Bibliograﬁsche Informationen
der Deutschen Nationalbibliothek
Die Deutsche Bibliothek verzeichnet diese Publikation in der deutschen Nationalbibliograﬁe; detaillierte bibliograﬁsche Daten sind im Internet über:
http://dnb.d-nb.de abrufbar.
http://dx.doi.org/10.4419/86788658
ISSN 1864-4872 (online)
ISBN 978-3-86788-658-1
David Card, Jochen Kluve, and Andrea Weber1
What Works?
A Meta Analysis of Recent Active Labor
Market Program Evaluations
Abstract
We present a meta-analysis of impact estimates from over 200 recent econometric
evaluations of active labor market programs from around the world. We classify
estimates by program type and participant group, and distinguish between three
diﬀerent post-program time horizons. Using meta-analytic models for the eﬀect size of
a given estimate (for studies that model the probability of employment) and for the sign
and signiﬁcance of the estimate (for all the studies in our sample) we conclude that:
(1) average impacts are close to zero in the short run, but become more positive 2-3
years after completion of the program; (2) the time proﬁle of impacts varies by type of
program, with larger gains for programs that emphasize human capital accumulation;
(3) there is systematic heterogeneity across participant groups, with larger impacts for
females and participants who enter from long term unemployment; (4) active labor
market programs are more likely to show positive impacts in a recession.
JEL Classiﬁcation: J00, J68
Keywords: Active labor market policy; program evaluation; meta-analysis
July 2015
1 David Card, UC Berkeley; Jochen Kluve, Humboldt University and RWI; Andrea Weber, University of Mannheim.
– We are extremely grateful to Diana Beyer, Hannah Frings and Jonas Jessen for excellent research assistance.
Financial support from the Fritz Thyssen Foundation and the Leibniz Association is gratefully acknowledged.– All
correspondence to: Jochen Kluve, RWI - Büro Berlin, Invalidenstr. 112, 10115 Berlin, Germany, e-mail: jochen.
[email protected]
I.Introduction
IntheaftermathoftheGreatRecessionthereisrenewedinterestinthe
potentialforactivelabormarketpolicies(ALMP’s)tohelpeaseawiderangeoflabor
marketproblems,includingyouthunemploymentandpersistentjoblessnessamong
displacedadults(e.g.,Martin,2014).Althoughtrainingprograms,employment
subsidies,andother“active”policieshavebeeninuseforwellover50years,credible
evidenceontheircausalimpactshasonlybecomeavailableinrecentdecades(see
Lalonde2003forabriefhistory).Withinarelativelyshortperiodoftimethenumberof
scientificevaluationshasexploded,holdingouttheprospectofbeingabletolearnfrom
paststudieswhattypesofprogramsworkbest,inwhatcircumstances,andforwhom.
InthispaperweundertakeametaanalysisoftherecentALMPevaluation
literature,lookingforsystematicevidenceonthesequestions.1Buildingonanearlier
efforttotrackdowntherelevantliterature(Card,Kluve,Weber,2010;hereafterCKW),
weassembleasampleof207studiesthatprovide857separateestimatesoftheeffect
ofaspecificprogramonaparticularsubgroupofparticipantsatagivenpostͲprogram
timehorizon.Oursampleincludesestimatesfordifferenttypesofprogramsandmany
differentparticipantsubgroups,providingtheopportunitytotestwhethercertain
programstendtoworkbetter(orworse)forspecificgroups.Inmanycaseswealso
observeimpactsforthesameprogramatshorterandlongerhorizons,allowingusto
characterizetheprofileofpostͲprogramimpactsfordifferentALMPpolicies.
Wesummarizetheestimatesfromdifferentstudiesintwocomplementaryways.
Extendingthetraditionalvotecountapproach,weclassify"signandsignificance"based
onwhethertheestimateissignificantlypositive,statisticallyinsignificant,orsignificantly
1
PreviousreviewsincludeHeckman,LalondeandSmith(1999),whosummarize75microeconometric
evaluationsfromtheU.S.andothercountries,Kluve(2010),whoreviewscloseto100studiesfrom
Europe,andFilgesetal.(2015),whoanalyzeanarrowersetof39studies.Greenberg,Michalopoulosand
Robins(2003)reviewU.S.programstargetedtodisadvantagedworkers.IbarraránandRosas(2009)
reviewprogramsinLatinAmericasupportedbytheInterͲAmericanDevelopmentBank.Relatedmeta
analysesfocusingonlabormarketinterventionsinlowandmiddleincomecountriesincludeChoand
Honorati(2014)andGrimmandPaffhausen(2015).
4
negative.Thisclassificationcanbeappliedtoalltheestimatesinoursample,regardless
oftheoutcomevariableoreconometrictechniqueusedintheevaluation.Our
preferredapproach,whichweuseforthelargesubsetofstudiesthatmeasurethe
impactofaprogramontheprobabilityofemployment,modelstheeffectsizeofthe
estimateդtheimpactontheemploymentrateoftheprogramgroup,dividedbythe
standarddeviationofemploymentinthecomparisongroup.Aneffectsizeapproachis
favoredinthemetaanalysisliterature(e.g.,Hedges,1981;HedgesandOlkin,1985),
sinceeffectsizesarenotmechanicallyrelatedtothenumberofobservationsusedinthe
study,whereasstatisticalsignificanceis(inprinciple)sampleͲsizedependent.
Fortunately,thetwoapproachesyieldsimilarconclusionswhenappliedtothesubsetof
studiesforwhicheffectsizesareavailable,givingusconfidencethatourmainfindings
areinvarianttohowwesummarizetheliterature.
Wereachfourmainsubstantiveconclusions.First,onaverageALMP'shave
relativelysmalleffectsintheshortrun(lessthanayearaftertheendoftheprogram),
butlargerpositiveeffectsinthemediumrun(1Ͳ2yearspostprogram)andlongerrun
(2+years).TheaverageshortruneffectsizeforALMP'sthatmeasureimpactson
employmentis0.04ʍ's(standarddeviationunits)andisnotsignificantlydifferentfrom
zero.Theaveragemediumruneffectsize,bycomparison,is0.12ʍ's,whiletheaverage
longerruneffectsizeis0.19ʍ's,andbotharehighlysignificant(t>3).
Second,thetimeprofileofimpactsinthepostͲprogramperiodvarieswiththe
typeofALMP.Jobsearchassistanceandsanctionprogramsthatemphasize"workfirst"
haverelativelylargeshorttermimpacts,onaverage.Trainingandprivatesector
employmentprogramshavesmallershorttermimpactsbutlargereffectsinthe
mediumandlongerruns.Publicsectoremploymentsubsidiestendtohavenegligibleor
evennegativeimpactsatallhorizons.
Third,wefindthattheaverageimpactsofALMP'svaryacrossgroups,withlarger
effectsforfemalesandparticipantsdrawnfromthepooloflongtermunemployed,and
smallereffectsforolderworkersandyouths.Wealsofindsuggestiveevidencethat
5
certaintypesofprogramsworkbetterforspecificsubgroupsofparticipants.Jobsearch
assistanceandsanctionprogramsappeartoberelativelymoresuccessfulfor
disadvantagedparticipants,whereastrainingandprivatesectoremploymentsubsidies
tendtoworkbetterforthelongtermunemployed.Finally,weinvestigatetheroleof
labormarketconditionsontherelativeefficacyofALMP's,andconcludethatprograms
havelargerimpactsinperiodsofslowgrowthandhigherunemployment.
Animportantdevelopmentintherecentevaluationliteratureistheincreasing
useofrandomizedcontrolledtrials(RCT's):oneͲfifthofoursampleisobtainedfrom
randomizeddesigns.Reassuringly,RCTͲbasedprogramestimatesarenotmuch
differentfromthenonͲexperimentalestimatesinoursample,noraretheresignificant
differencesbetweenpublishedandunpublishedstudies,orbetweenhighlyͲcitedand
lessͲcitedstudies.Nevertheless,thechoiceofoutcomevariableandeconometric
methodhassomeinfluenceonthesignandsignificanceoftheimpactestimates,witha
tendencytofindmorepositiveestimatesinstudiesthatmodelendpointslikethetime
toexitfromthebenefitsystemorthetimetoanewjob.
II.SampleConstruction
a.SamplingImpactEvaluationStudies
OursampleofstudiesextendsthesampledevelopedinCKW,usingthesame
criteriatoselectinͲscopestudiesandthesameprotocolstoextractinformationabout
theprogramsandtheirimpacts.TheCKWsamplewasderivedfromresponsestoa2007
surveyofresearchersaffiliatedwiththeInstitutefortheStudyofLabor(IZA)andthe
NationalBureauofEconomicResearch(NBER).Toextendthissamplewebeganby
reviewingtheresearchprofilesandhomepagesofIZAresearchfellowswithadeclared
interestin“programevaluation”,lookingforALMPevaluationswrittensince2007.We
alsosearchedtheNBERworkingpaperdatabaseusingthesearchstrings“training”,
“active”,“publicsectoremployment”,and“searchassistance.”
6
InasecondstepweusedaGoogleScholarsearchtoidentifyallpapersciting
CKWortheearlierreviewbyKluve(2010).WealsosearchedthroughtheInternational
InitiativeforImpactEvaluation's"RepositoryofImpactEvaluationPublishedStudies,"
theonlineprojectlistoftheAbdulLatifJameelPovertyActionLab(JͲPAL),andthelistof
LatinAmericanprogramevaluationsreviewedbyIbarraránandRosas(2009).
Afteridentifyinganinitialsampleofstudiesusingthesesteps,wereviewedthe
citationsinallthestudiestoverifywhethertherewereanyotherstudiesthatwehad
notyetincluded.Finally,weidentifiedfouradditionalpaperspresentedataconference
inearlyfall2014.ThesearchprocesslastedfromApriltoOctober2014andyielded154
newstudiesthatweconsideredforinclusioninourALMPimpactevaluationdatabase.
b.InclusionCriteria
Inordertogenerateaconsistentdatabaseacrossthetwowavesofdata
collection(2007and2014)weimposedthesamerestrictionsadoptedinCKW.First,the
program(s)analyzedintheevaluationhadtobeoneoffollowingfivetypes:
x
classroomoronͲtheͲjobtraining
x
jobsearchassistance
x
sanctionsforfailingtosearch,includingthreatofassignmenttoaprogram
x
subsidizedprivatesectoremployment
x
subsidizedpublicsectoremployment.
Second,welimitedtheemploymentsubsidiestoindividuallytargetedsubsidies,
excludingprogramsthatallowfirmstoselecttheemployeeswhoreceiveasubsidy.
Third,werestrictedattentiontotimeͲlimitedprograms,eliminatingopenͲended
entitlementslikeeducationgrantsorchildcaresubsidies.Fourth,weexcludedpublic
programswithnoexplicitly“active”component,suchasthemanipulationofbenefit
levelsforunemploymentinsurancerecipients.
7
Methodologically,weincludeonlywellͲdocumentedstudiesthatuseindividual
microdataandincorporateacounterfactual/controlgroupdesignorsomeformof
selectioncorrection.
Applyingtheseinclusioncriteria,weretained110ofthe154studiesidentifiedin
thesearchprocess.2Weaddedthesetothe97studiesfromCKW,resultinginatotalof
207ALMPimpactevaluationsinourfinaldatabase.Acompletelistofthesestudiesis
containedinanAppendixavailableonrequestfortheauthors.
c.ExtractingImpactEstimatesandInformationonProgramsandParticipants
Thenextstepinourdatacollectionprocesswastoextractinformationaboutthe
programsandparticipantsanalyzedineachstudy,andthecorrespondingprogram
estimate.UsingthesameclassificationsystemasinCKW,wegatheredinformationon
thetypeofALMP,theageandgenderoftheparticipantpopulation,thetypeof
dependentvariableusedtomeasuretheimpactoftheprogram,andtheeconometric
methodology.3Wealsogatheredinformationonthe(approximate)datesofoperation
oftheprogram;onthetypesofprogramparticipants(longtermunemployed,
disadvantagedworkers,orregularunemploymentinsurancerecipients),onthesource
ofthedatausedintheevaluation(administrativerecordsoraspecializedsurvey),and
ontheapproximatedurationoftheprogram.
Ifastudyreportedseparateimpactestimateseitherbyprogramtypeorby
participantgroup,weidentifiedtheprogram/participantsubgroup(PPS)andcodedthe
impactestimatesseparately.Overall,wehaveinformationon526separatePPS'sfrom
the207studies,withaminimumof1andamaximumof10PPS'sineachstudy.We
alsoidentifieduptothreeimpactestimatesforeachPPS,correspondingtothree
differentpostͲprogramtimehorizons:shortͲterm(approximatelyoneyearafter
2
Themainreasonsforexclusionwere:overlapwithotherpapers(i.e.estimatingimpactsforthesame
program);programoutofscope;andnoexplicitcounterfactualdesign.
3
AsinCKW,weextractedtheinformationfromthestudiesourselves,sincewefoundthatevenadvanced
graduatestudentswereoftenunabletointerpretthestudies.
8
completionoftheprogram);mediumterm(approximatelytwoyearsafter);andlongerͲ
term(approximately3yearsafter).Intotal,wehave857separateprogramestimates
forthe526program/participantsubgroups,withbetweenoneandthreeestimatesof
theeffectoftheprogramatdifferenttimehorizons.4
Weusetwocomplementaryapproachestoquantifytheestimatedprogram
impacts.First,weclassifytheestimatesassignificantlypositive,insignificantlydifferent
fromzero,orsignificantlynegative.Thismeasureofeffectivenessisavailableforevery
estimateinourdatabase.Forthesubsetofstudiesthatmeasureprogrameffectson
theprobabilityofemployment,wealsoextractanestimatedeffectsize,definedasthe
estimatedimpactontheemploymentrateoftheprogramgroup,dividedbythe
standarddeviationoftheemploymentrateofthecomparisongroup.5Asdiscussedin
moredetailinsectionIVa,below,theeffectsizeestimatesareunaffectedbythesizeof
thesampleusedintheestimationprocedure,whereasthesignificancelevelsarein
principaldependentonthesamplesize.Inthestudiesreviewedhere,however,thereis
littleornoassociationbetweensamplesizeandthelikelihoodofasignificantpoint
estimate(eitherpositiveornegative).Asaresultthetwowaysofquantifyingprogram
impactsleadtoverysimilarconclusions.6
Afinalstepinourdataassemblyprocedurewastoaddinformationonlabor
marketconditionsatthetimeofoperationoftheprogram.Specifically,wegathered
countryͲspecificinformationonGDPgrowthratesandunemploymentratesfromthe
OECD,theWorldBank,andtheILO.Weusetheaveragegrowthrateandtheaverage
unemploymentrateduringtheperiodtheprogramgroupparticipatedintheALMPto
proxyforcyclicalconditionsduringtheevaluation.
4
ForaspecificPPSandtimehorizonwetrytoidentifyandcodethemainestimateinthestudy.Wedo
notincludemultipleestimatesforthesamePPSandtimehorizon.
5
Notethattomeasuretheeffectsizeweneedbothanimpactestimateandthemeanemploymentrate
ofthecomparisongroup,whichisnotalwaysavailable.Wealsoextractedeffectsizesforonestudythat
measuredprogramimpactsonthedurationoftimetothestartofthefirstjob.
6
ThesamefindingwasobtainedinCKW.Asdiscussedbelow,webelievethatthisarisesbecauselarger
samplesizestendtobeusedwithmorecomplexresearchdesigns.
9
III.DescriptiveOverview
a.ProgramTypes,ParticipantCharacteristics,EvaluationDesign
Table1presentsanoverviewoftheprogramestimatesinourfinalsample.As
noted,wehaveatotalof857differentimpactestimatesfor526differentPPS's
(programͲtype/participantsubgroupcombinations)extractedfrom207separate
studies.Todealwithpotentialcorrelationsbetweentheprogramestimatesfroma
givenstudy,throughoutthispaperwecalculatestandarderrorsclusteringbystudy.
Column1ofthetablepresentsthecharacteristicsofouroverallsample,while
columns2Ͳ6summarizetheestimatesfromfivesubgroupsofcountries:Austria,
GermanyandSwitzerland(the"Germanic"countries),whichaccountforaboutone
quarterofallstudies;Denmark,Finland,NorwayandSweden(the"Nordic"countries),
whichaccountforanotherquarterofstudies;Australia,Canada,NewZealand,U.K.and
U.S.(the"AngloSaxon"countries),whichaccountforjustover10%ofstudies;andtwo
nonͲmutuallyexclusivegroupsoflowerandmiddleincomecountriesͲͲ"nonͲOECD"
countries(10%ofstudies),andLatinAmericanandCaribbean(LAC)countries(10%of
studies).Forreference,AppendixFigure1showsthenumbersofestimatesbycountry.
ThelargestsourcecountriesareGermany(253estimates),Denmark(115estimates),
Sweden(66estimates),theU.S.(57estimates)andFrance(42estimates).
AsshowninthesecondpanelofTable1weclassifyprogramsintobroadtypes.
TrainingprogramsͲͲincludingclassroomandonͲtheͲjobtrainingͲͲaccountforabout
onehalfoftheprogramestimates,withbiggersharesinthenonͲOECDandLAC
countries,andasmallershareintheNordiccountries.Jobsearchassistance(JSA)
programs,privatesubsidyprograms,andsanction/threatprogramseachaccountfor
aboutoneͲsixthoftheprogramestimates,thoughagainthereisvariabilityacross
countrygroups,withJSAandsanction/threatprogramsbeingparticularlyprevalentin
theNordicandAngloSaxoncountries.Subsidizedpublicsectorjobprogramsare
relativelyrareinallcountygroups.
10
Thenextthreepanelsofthetableshowthecharacteristicsoftheprogram
participantgroups,classifiedbyagegroup,gender,and"type"ofparticipant.About
oneͲhalfoftheestimatesareformixedagegroupsandmixedgendergroups,butwe
alsohaverelativelylargesubsetsofestimatesthatarespecifictoeitheryoungeror
olderworkers,orfemalesormales.Themajorityofprogramestimatesarefor
participantswhoenterfromtheregularunemploymentinsurance(UI)system.Typically
theseparticipantsareassignedtoaprogramandrequiredtoattendasaconditionfor
continuingbenefiteligibility.7Theremainingestimatesaresplitbetweenprogramsthat
servethelongtermunemployed(LTU)andthosethatserve"disadvantaged"participant
groups.Inmanycases,LTUanddisadvantagedparticipantsarerecruitedfromthe
overallpopulationandenrollvoluntarily.Suchvoluntaryprogramsaremorecommonin
theAngloSaxoncountriesandinlessdevelopedcountriesthatlackaformalUIsystem.8
Nextweshowtheoutcomevariablesusedtomeasuretheprogramimpactand
thetimehorizonsoftheestimate.ThemostcommonoutcomeͲͲparticularlyinthe
GermanicandnonͲOECDcountriesͲͲistheprobabilityofemployment,whilethelevelof
earningsisthemostcommonmetricintheAngloSaxoncountries.Aboutonesixthof
theprogramestimatesͲͲbut40%oftheestimatesfromNordiccountriesͲͲmeasurethe
exitratefromthebenefitsystem,typicallyfocusingontherateofexittoanew
(unsubsidized)job.Finally,asmallsubsetofestimatesͲmostlyfromAngloSaxon
countriesͲͲfocusontheprobabilityofunemployment.Aboutonehalfoftheestimates
areforashorttermhorizon(<1year)afterprogramcompletion,35%foramedium
term(1Ͳ2years),and18%foralongerterm(morethan2yearafter).
ThelastrowoftheTableshowsthefractionofprogramestimatesthatarebased
onanexperimentaldesign.Inmostofourcountrygroupsabout30%ofestimatescome
fromrandomizedcontrolledtrials(RCT's)thathavebeenexplicitlydesignedtomeasure
7
ThistypeofprogramrequirementiswidespreadinEuropeͲͲseeSianesi(2004)foradiscussion.
TheU.S,jobtrainingprogramsanalyzedintheseminalpapersofAshenfelter(1978),Ashenfelterand
Card(1985),Lalonde(1986),Heckman,Ichimura,Smith,andTodd(1998)areallofthistype.
8
11
theeffectivenessoftheALMPofinterest.AnimportantexceptionistheGermanic
countries,wherenoexperimentallybasedestimatesareyetavailable.
Thedistributionofprogramestimatesovertime(definingtimebytheearliest
intakeyearoftheprogram)isshowninFigure1,withseparatecountsforthe
experimentalandnonͲexperimentalestimates.Oursampleincludesprogramsfromas
farbackas1980,thoughthemajorityofestimatesarefromthe1990sandearly2000s.
Thetrendtowardincreasinguseofexperimentaldesignsisclearlyevident:amongthe
210estimatesfrom2004andlater,61%arefromrandomizeddesigns.
b.MeasuresofProgramImpactͲOverview
Table2givesanoverviewofourtwomainmeasuresofprogramimpact,
contrastingresultsfortheshortterm,mediumterm,andlongterm.Columnone
summarizesthesignandsignificanceofalltheavailableprogramestimates.Amongthe
415shorttermestimates,40%aresignificantlypositive,42%areinsignificant,and18%
aresignificantlynegative.Thepatternofresultsismorepositiveinthemediumand
longerterms,withamajorityofestimates(52%)beingsignificantlypositiveinthe
mediumterm,and61%beingsignificantlypositiveinthelongerterm.
Column2showsthedistributionofsignandsignificanceforthesubsetofstudies
thatusepostͲprogramemploymentratestoevaluatetheALMPprogram.These111
studiesaccountfor490programestimates(57%ofourtotalsample).Theshortterm
programestimatesfromthissubsetofstudiesaresomewhatlesspositivethaninthe
overallsample:only31%aresignificantlypositive,while22%aresignificantlynegative,
and47%areinsignificantlydifferentfromzero.Inthemediumandlongerterms,
however,thediscrepancydisappears.Indeed,thelongertermestimatesfrom
evaluationsthatstudytheprobabilityofemploymentareslightlymorepositivethan
estimatesfromstudiesthatuseotheroutcomevariables.Asdiscussedbelow,these
patternsarenotexplainedbydifferencesinthetypesofALMPprogramsanalyzedin
differentstudies,orbydifferencesinparticipantcharacteristics.Instead,theyreflecta
12
systematictendencyforstudiesbasedonhazardmodelsforprogramexittoexhibit
morepositiveshorttermimpactsthanstudiesbasedonemploymentorearnings.
Column3ofTable2showsthedistributionsofsignandsignificanceassociated
withtheimpactestimatesthatwecanactuallyconvertintoeffectsizes.9The
distributionsareverysimilartothoseincolumn2,suggestingthatthereisnosystematic
biasassociatedwiththeavailabilityofaneffectsizeestimate,onceweconditiononthe
outcomeusedintheevaluation.
Finally,column4showsthemeaneffectsizesforthesubsampleincolumn3,
alongwithrobuststandarderrorsthattakeaccountofcorrelationamongtheprogram
estimatesfromthesamestudy.Themeanshorttermeffectsizeestimateis0.04ʍ'sand
isatbestonlymarginallysignificant(t=1.65).Bycomparisontheaverageeffectsizesin
themediumandlongertermsare0.12ʍ'sand0.19ʍ's,respectively,andbothare
significantlydifferentfromzerowithahighdegreeofconfidence.Theseestimates
reinforcetheconclusionthatonaverage,ALMP'shavearelativelysmallimpactinthe
shortrun,butmorepositiveeffectsinthemediumandlongerruns.
Distributionsofeffectsizeestimatesintheshort,medium,andlongerrunsare
showninFigures2a,2b,and2c,whichgive"forestplots"oftheeffectsizeestimates,
alongwiththeirassociatedconfidenceintervals,forstudiesfromourmostrecentwave
ofdatacollection.10AssuggestedbythemeansinTable2,theoveralldistributionof
effectsizeestimatesclearlyshiftstotherightasthetimehorizonisextended.Another
interestingfactisthatthewidthoftheconfidenceintervalsisuncorrelatedwiththe
magnitudeoftheeffectsizeestimates.Thereisnoevidencethatmorepositiveeffect
sizeestimatestendtobelesspreciseͲͲasmightbefearedifauthorstendtosearchfor
9
Themainreasonwhywecannotextractaneffectsizeestimatefromastudythatmodelstheprobability
ofemploymentisthatthereisnoinformationontheemploymentrateofthecomparisongroupinthe
relevanttimeframe.
10
InformationtoconstructconfidenceintervalswasnotextractedinCKW.Thus,theestimatesreportedin
Figures2a,2b,and2carefromthelateststudiescollectedinoursecondround.
13
specificationsthatshowapositiveimpact,orifsmallscalestudiesaremorelikelytobe
writtenupiftheresultsarepositive(thesoͲcalled"filedrawer"biasproblem).
ReturningtoTable2,column4alsoshowsthemeaneffectsizesamongprogram
estimatesthatareclassifiedassignificantlypositive,insignificant,orsignificantly
negative.Aswouldbeexpectediftheclassificationofsignandsignificanceismainly
drivenbyvariationinthemagnitudeofaparticulareffectsizeestimateͲͲandnotby
variationinthestandarderrorsoftheestimatesͲͲthemeaneffectsizeforsignificant
positiveestimatesisrelativelylargeandpositive,themeaneffectsizeforsignificant
negativeestimatesisrelativelylargeandnegative,andthemeaneffectsizefor
insignificantestimatesiscloseto0.ThispatternisillustratedinAppendixFigures2a,2b,
and2d,whereweplotthehistogramsofeffectsizeestimatesateachtimehorizon,
highlightingthecontributionsofestimatesineachcategoryofsignandsignificance.At
allthreetimehorizons,thethreesubgroupsofestimatesappeartobedrawnfrom
distributionsthatarecenteredondifferentmidpoints.Thisseparationsuggeststhatthe
signandsignificanceofanestimatecanserveas(noisyindicator)ofitseffectsize.
c.VariationinProgramImpacts
Tables3aand3bprovideafirstlookatthequestionofhowALMPimpactsvary
acrossdifferenttypesofprogramsanddifferentparticipantgroups.Foreachsubsetof
estimatesweshowthemeaneffectsizesateachtimehorizonandthecorresponding
fractionofprogramestimatesthatissignificantlypositive.
Focusingfirstoncomparisonsacrossprogramtypes(Table3a),noticethat
trainingandprivatesectoremploymentprogramstendtohavesmalleffectsintheshort
run,coupledwithmorepositiveimpactsinthemediumandlongerruns.Incontrast,
JSAandsanction/threatprogramshavemorestableordecliningimpactsovertime.
Theseprofilesareconsistentwiththenatureofthetwobroadgroupsofprograms.
Trainingandprivatesubsidyprogramsrequireparticipantstopassupregularjob
opportunitieswhileintheprogram.Thistemporarywithdrawalfromtheregularlabor
14
marketwouldbeexpectedtodepressoutcomesintheperiodimmediatelyafter
completionoftheprogramͲͲasoͲcalled"lockͲin"effect(e.g.,HamandLalonde,1996).
Assumingthatinvestmentsmadeduringtheprogramperiodarevaluable,however,the
outcomesofparticipantswillgraduallycatchupwiththoseofthecomparisongroup.11
Bycomparison,JSAandsanction/threatprogramsaredesignedtopushparticipantsinto
thelabormarketquickly,withlittleornoinvestmentcomponent.Intheabsenceof
largereturnstorecentexperience,itisunlikelythattheseprogramscanhavelargelong
runeffects.12
AnotherclearfindinginTable3aistherelativelypoorperformanceofpublic
sectorprogramsͲͲaresultthathasbeenfoundinotherpreviousanalyses(e.g.,
Heckmanetal.,1999)andinourearlierstudy.Thispatternsuggeststhatprivate
employersplacelittlevalueontheexperiencesgainedinapublicsectorprogramͲͲ
perhapsbecausemanyoftheseprogramshavelittleornoskillͲbuildingcomponent,and
onlyservetoslowdownthetransitionofparticipantstounsubsidizedjobs.
Thelowerpanelofthetablecomparesestimatesfordifferentintakegroups.An
interestingconclusionhereisthatprogramsthatservethelongtermunemployedand
disadvantagedindividualsappeartohavemorepositiveshortrunoutcomesthan
programsforUIrecipients.Inthemediumandlongerruns,UIrecipientsappeartocatch
upsomewhatrelativetotheothertwogroups.
Table3bshowsthecontrastsbyage,genderandexperimentaldesign.The
comparisonsbyageshowamixedpattern,withsomewhatlargeraverageeffectsizes
foryouththantheothergroupsintheshortrun,butworseoutcomesforbothyouth
andolderparticipantgroupsinthemediumandlongerruns.Thecontrastsby
experimentaldesignarealsomixed,withtheeffectsizecomparisonssuggestingmore
11
AsnotedbyMincer(1974)asimilarcrossͲoverpatternisobservedinthecomparisonofearnings
profilesofhighschoolgraduatesandcollegegraduates.
12
Evidenceonthevalueoflabormarketexperienceforlowerskilledworkers(GladdenandTaber,2000;
CardandHyslop,2005)suggeststhatthereturnsaremodestandunlikelytoexceed2or3percentper
yearofwork.
15
positiveresultsforexperimentaldesignsintheshortrunandmorenegativeresultsin
thelongrun.Thecomparisonsbygenderaremoreconsistentandsuggestthatfemale
participantsbenefitmorethanmalesormixedgendergroups.
Wenotethatthesesimplecontrastsmustbeinterpretedcarefullybecausethere
aremultiplesourcesofpotentialheterogeneityintheprogramimpacts.Forexample,
thefractionsoftrainingprogramsevaluatedbyRCT'sisrelativelylow.Themetaanalysis
modelsinSectionIVdirectlyaddressthisissueusingamultivariateregressionapproach.
d.ProfileofPostͲProgramImpacts
Simplecomparisonsacrosstheimpactestimatesinoursamplesuggestthat
ALMP’shavemorepositiveeffectsinthemediumandlongerterms.Toverifythatthisis
actuallytrueforagivenprogramandparticipantsubgroup–andisnotsimplyan
artefactofheterogeneityacrossstudies–weexaminethewithinͲPPSevolutionof
impactestimatesinTable4.
Columns1Ͳ3showthechangesinestimatedeffectsizeforthesubsetofstudies
forwhichweobservebothshortandmediumtermestimates,mediumandlongterm
estimates,andshortandlongtermestimates,respectively.Estimatedeffectsizestend
toincreaseasthetimehorizonisextendedfromtheshortruntothemediumrun.The
averagechangebetweenthemediumandlongerrunsisslightlynegative,butoverall
theshortͲruntolongͲrunchangeisstillpositive.
Comparingacrossprogramtypesitisclearthatthepatternofrisingimpactsis
drivenalmostentirelybytrainingͲbasedprograms,whichshowarelativelylargegainin
effectsizesfromtheshorttermtothemediumtermandonlyasmalldeclinebetween
themediumandlongerruns.Thepatternsfortheothertypesofprogramssuggest
relativelyconstantordecliningeffectsizesoverthepostͲprogramtimehorizon.In
particular,incontrasttothepatternsinTable3a,thereisnoindicationofarisein
impactsforprivatesubsidyprogramsovertime,suggestingthatthegainsinTable3a
maybedrivenbyheterogeneitybetweenstudies.Wereturntothispointbelow.
16
Incolumns4Ͳ6weexaminethewithinͲstudychangesinsignandsignificancefor
abroadersetofstudies.Here,weassignavalueof+1toPPSestimatesthatchange
frominsignificanttosignificantlypositiveorfromsignificantlynegativetoinsignificant;
Ͳ1toestimatesthatchangefromsignificantlypositivetoinsignificantorfrom
insignificanttosignificantlynegative;and0toestimatesthathavethesame
classificationovertime.ThissimplewayofsummarizingthewithinͲstudypatterns
pointstogenerallysimilarconclusionsasthechangesineffectsize,thoughjobsearch
assistanceprogramsshowmoreevidenceofariseinimpactsfromtheshortͲruntothe
mediumrunincolumn4thancolumn1,andprivateemploymentsubsidiesshowamore
positivetrendinimpactsfromtheshorttolongrun.
AppendixTable1presentsfullcrossͲtabulationsofsign/significanceatthe
variouspostͲprogramtimehorizons.Assuggestedbythesimplesummarystatisticsin
Table4,mostprogramestimateseitherremaininthesamecategory,orbecome"more
positive"overtime.
IV.MetaAnalyticModelsofProgramImpacts
a.ConceptualFramework
ConsideranALMPevaluationthatmodelsanoutcomeyobservedformembers
ofbothaparticipantgroupandacomparisongroup.Letbrepresenttheestimated
impactoftheprogramontheoutcomesoftheparticipants,letNrepresentthe
combinedsamplesizeofparticipantsandcomparisons,letɴrepresenttheprobability
limitofb(i.e.,theestimatethatwouldbeobtainedifNwereinfinite)andletʍ
representthestandarddeviationofyinthecomparisongrouppopulation.Assuming
thatbhasastandardasymptoticdistribution,theestimatorobtainedwithasamplesize
ofNisapproximatelynormallydistributedwithmeanɴandvariancev,where
v=K2ʍ2/N
(1)
17
andKisadesignfactorreflectingthedesignfeaturesofthestudy.13Theactualestimate
obtainedinthestudycanbewrittenas:
b=ɴ+KʍNͲ1/2z, wherezisarealizationfromanapproximatelynormaldistribution.
Assumingthatʍisknown(orcanbeestimatedwithhighprecision),the
associatedestimateoftheeffectsizeis:
b/ʍ=ɴ/ʍ+KNͲ1/2z,
(2)
2
whichdiffersfromthelimitingeffectsizeɴ/ʍbyasamplingerrorwithvarianceK /N.
AssumethatthelimitingeffectsizeinagivenstudydependsonavectorXofobserved
characteristicsofthestudy,(includingfeaturesoftheprogram,theparticipants,andthe
evaluationdesign)andonunobservedfactorsɸ:
ɴ/ʍ=Xɲ+ɸ.
(3)
whereɲisavectorofcoefficients.Thisleadstoametaanalysismodelfortheobserved
effectssizeestimatesoftheform:
b/ʍ=Xɲ+u,
wheretheerroru=ɸ+N
(4)
Ͳ1/2
Kzincludesboththesamplingerrorintheestimateband
theunobserveddeterminantsofthelimitingeffectsizeforagivenstudy.
Weusesimpleregressionmodelsbasedonequation(4)toanalyzetheeffect
sizesthatareavailableinoursample.Forthebroadersampleweuseorderedprobit
(OP)modelsforthe3Ͳwayclassificationofsignandsignificance.Tounderstandthe
interpretationoftheOPmodels,notethatequations(1)and(2)implythatthetͲstatistic
associatedwiththeestimatedimpact(b)is:
t=b/v1/2=(ɴ/ʍ)N1/2/K+z.
Usingequation(3):
t=[N1/2/K]Xɲ+z+[N1/2/K]ɸ.
(5)
13
Forexample,inanexperimentwith50%ofthesampleinthetreatmentgroupandnoaddedcovariates,
K2=2(ɷ2+1),whereɷisafactorrepresentingtheratioofthestandarddeviationoftheoutcomeafter
exposuretotreatmenttothestandarddeviationintheabsenceoftreatment.Inmorecomplexdesigns
suchasdifferenceindifferencesorinstrumentalvariablesKwillbebigger.
18
IfN/K2ͲͲthe"effectivesamplesize"ofagivenstudy,takingaccountoftheresearch
designͲͲisconstantacrossstudiesandtherearenounobserveddeterminantsofthe
limitingeffectsize(i.e.,ɸ=0)thetͲstatisticwillbenormallydistributedwithmeanXɲ'
whereɲ'=[N1/2/K]ɲ.14InthiscasethecoefficientsfromanOPmodelforwhetherthet
statisticislessthanͲ2,betweenͲ2and2,orgreaterthan2(i.e.,thesignandsignificance
oftheestimatedprogrameffects)willbestrictlyproportionaltothecoefficients
obtainedfromaregressionmodelofthecorrespondingeffectsizes.Onereasonforthe
effectivesamplesizetoremainroughlyconstantacrossstudiesistheendogenous
selectionofresearchdesigns.Mostanalystswillonlypursueacomplexresearchdesign
ifthereisabigenoughsampletoensuresomeminimumstatisticalpower.
EvenifN/K2variesacrossstudies,orthereareunobserveddeterminantsofthe
limitingeffectsizeindifferentstudies,theimplieddatageneratingprocessmaybe
reasonablyapproximatedbyasimpleorderedprobit.Indeed,asweshowbelow,inour
sampleofstudiesthecoefficientsfromeffectsizemodelsandOPmodelsareverynearly
proportionalsowebelievetheapproximationisreasonable.
b.BasicEffectSizeModels
Table5presentstheestimatesfromaseriesofregressionmodelsfor352effect
sizeestimatesobservedfor200program/participantsubgroupsin83differentstudies.
WepooltheeffectsizesfordifferentpostͲprogramhorizonsandincludedummies
indicatingwhethertheprogramestimateisforthemediumorlongterm(withshort
termestimatesintheomittedgroup).Thebasicmodelincolumn1includesonlythese
controlsandasetofdummiesforthetypeofprogram(withtrainingprogramsinthe
omittedcategory).ConsistentwiththesimplecomparisonsinTable3a,wefindthatthe
14
Ascanbeseenfromequation(2),thesamplingvarianceoftheeffectsizeestimatefromagivenstudyis
proportionaltoK2/N.Fromthesampleofeffectsizesandassociatedsamplingerrorssummarizedin
Figures1a,1band1cwecanestimatethevariabilityofN/K2acrossstudies.Thisexercisesuggeststhat
theeffectivesamplesizevariesfairlywidelyacrossstudies,butisnothighlycorrelatedwithN.
19
effectsizeestimatesarelargerinthemediumandlongrun,andthatpublicsector
employmentprogramsareassociatedwithsmallereffectsizes.
Themodelincolumn2introducesadditionalcontrolsforthetypeofparticipant
(UIrecipientsversuslongtermunemployedordisadvantaged),theirageandgender,
thecountrygroupinwhichtheprogramwasoffered,thedurationoftheprogram,and
fourfeaturesoftheevaluation:whetherithadanexperimentaldesign,thesquareroot
ofthesamplesize,whetherthestudywaspublished,andthestudy'scitationpercentile
relativetoallstudiesinoursamplereleasedinthesameyear.15Thesecontrolsslightly
attenuatethegrowthineffectsizesoverlongerpostͲprogramhorizonsbuthavelittle
effectontheprogramtypedummies.
Columns3and4introduceaparallelsetofmodelsthatallowthetimeprofilesof
postͲprogramimpactstovarywiththetypeofprogram.Inthesespecificationsthe
"maineffects"foreachprogramtypeshowtheshorttermimpactsrelativetotraining
programs(theomittedtype),whiletheinteractionsofprogramtypewithmediumterm
andlongtermdummiesshowhowtheimpactsevolverelativetotheprofilefortraining
programs(whicharesummarizedbythemaineffectsinthefirsttworows).Threekey
conclusionsemergefromthesemoreflexiblespecifications.First,assuggestedbythe
patternsinTable4,theeffectsizesfortrainingprogramstendtoriseovertimewhile
theeffectsforjobsearchassistanceandsanction/threatprogramsarenearlyconstant..
Second,publicsectoremploymentprogramsappeartoberelativelyineffectiveatall
timehorizons.Third,theprofileforprivatesectortrainingprogramsisrelativelysimilar
totheprofilefortrainingprograms.
Theestimatedcoefficientsfortheextracontrolvariablesincludedinthemodels
incolumns2and4ofTable5arereportedinthefirsttwocolumnsofTable7.The
coefficientestimatesfromthetwospecificationsarequitesimilarandsuggestthatthe
15
WeconductedaGoogleScholarsearchforcitationsofallthestudiesinourfinalanalysissamplein
October2014.Weconstructthecitationrankofeachstudyrelativetootherstudieswiththesame
publicationdateinoursampleasourmeasureofcitations.Wealsofitmodelsthatincludethedateof
thestudyandthedatasource.Theseareverysimilartothemodelspresentedinthetable.
20
impactofALMP'svariessystematicallywiththetypeofparticipant(withlargereffects
forthelongtermunemployed),theiragegroup(morenegativeimpactsforolderand
youngerparticipants),andtheirgender(largereffectsforfemales).Ontheotherhand
thereisnoindicationthatthecountrygroup,thedurationoftheprogram,orthe
featuresoftheevaluationmatter.Inparticular,theestimatedcoefficientofthe
experimentaldesigndummyisrelativelysmallinmagnitudeandinsignificantlydifferent
from0(t=0.63incolumn1and0.55incolumn2).
c.BasicModelsforSignandSignificance
Effectsizesareavailableforonly40%ofouroverallsample.Tosupplement
thesemodelsweturntoorderedprobitmodelsforsignandsignificance.Thefirst4
columnsofTable6presentaseriesofOPmodelsthatareparalleltothoseinTable5,
butfittoouroverallsampleofprogramestimates.Thespecificationsincolumns1and
3havenocontrolsotherthandummiesformediumandlongtermhorizonsandthe
typeofALMPͲͲinthelattercaseinteractingthetypeofprogramwiththetimehorizon
dummies.Columns2and4reportexpandedspecificationsthataddthecontrol
variablesreportedincolumns3and4ofTable7.Finally,column5ofTable6repeats
thespecificationfromcolumn4,butfittothesubsampleof352programestimatesfor
whichwehaveaneffectsizeestimate.
TheOPmodelsyieldcoefficientsthatareveryhighlycorrelatedwiththe
correspondingcoefficientsfromtheeffectsizemodels,but4Ͳ5timesbiggerin
magnitude.Forexample,thecorrelationofthe14coefficientsfromthespecificationin
column4ofTable6withcorrespondingcoefficientsfromthespecificationincolumn4
ofTable5is0.84.16Thus,theOPmodelsconfirmthattheimpactsofjobsearch
assistanceandsanction/threatprogramstendtofaderelativetotheimpactsoftraining
16
Theregressionmodelis:OPͲcoefficient=0.00+4.64×EffectͲsizeͲcoefficient;standarderror=0.85.
21
programs,andthatpublicsectoremploymentprogramsarerelativelyineffectiveatall
timehorizons.17
TheOPmodelsalsoconfirmmostofourconclusionsaboutthedifferential
impactsofALMP'sacrossdifferentparticipantgroupsandindifferentcountries.18
ComparingthecoefficientsinTable7,boththeeffectsizemodelsandthe
sign/significancemodelsshowsmallerimpactsofprogramsonyoungparticipantsand
olderparticipants,relativetotheimpactsonmixedagegroups,andlargerimpactsfor
longͲtermunemployedparticipants.Usingtheoverallsampleofprogramestimatesthe
OPmodelsalsopointtoasignificantlypositiverelativeimpactfordisadvantaged
participants.Incontrast,theeffectsizemodels(andtheOPmodelsfittotheeffectsize
sample)yieldaninsignificantcoefficient,arguablyasaconsequenceofthesmall
numberofstudiesthatfocusonthisgroup.
OneimportantdifferencebetweentheeffectsizemodelsandtheOPmodels
concernstherelativeimpactofALMP'sonfemaleparticipants.Intheeffectsizemodels
theestimatedcoefficientsforfemaleparticipantsarearound0.11inmagnitude,and
statisticallysignificantatconventionallevels(withtstatisticsaround2).IntheOP
models,bycomparison,thecorrespondingcoefficientsarerelativelysmallinmagnitude,
andfarfromsignificant.Furtherinvestigationrevealsthatthisdivergenceisdrivenby
theuppertailofeffectsizeestimatesforfemaleparticipants(seeAppendixFigure3),
andinparticularbytherelativelylargeeffectsizeestimatesforprogramsthatshowa
significantlypositiveeffect.19Thisuppertailofeffectsizesdoesnotappeartobedriven
17
Wealsofittwosimplerprobitmodelsfortheeventsofreportingapositiveandsignificantornegative
andsignificantestimate,reportedinAppendixTable2.Aswouldbeexpectediftheorderedprobit
specificationiscorrect,thecoefficientsfromthemodelforasignificantlypositiveeffectarequitecloseto
theOPcoefficients,whilethecoefficientsfromthemodelforasignificantlynegativeeffectareclosein
magnitudebutoppositeinsign.Interestingly,neitheroftheprobitmodelsyieldsasignificanteffectfor
thesquarerootofthesamplesize,confirmingthatvariationinthesignificanceoftheprogramestimates
acrossdifferentstudiesisonlyweaklyrelatedtothesamplesizeusedinthestudy.
18
Thecorrelationbetweenthecoefficientsincolumns2and4ofTable7is0.69.
19
Themedianand75thpercentilesoftheeffectsizedistributionforfemaleparticipantgroups,
conditionalonapositiveimpact,are0.25,and0.46respectively.Bycomparison,thecorresponding
statisticsformaleandmixedgenderparticipantgroupsare0.15,and0.27.
22
byafewoutliers,butinsteadreflectsasystematicallyhigherprobabilityofestimatinga
largepositiveeffectsizewhentheparticipantgroupislimitedtofemales.20
AfinalinterestingaspectoftheOPmodelsisthepatternofcoefficients
associatedwiththechoiceofdependentvariable,reportedinthetoprowsofTable7.
Thesecoefficientsshowthatstudiesmodelingthehazardrateofexitingthebenefit
systemortheprobabilityofunemploymentaresignificantlymorelikelytoreport
positivefindingsthanstudiesmodelingemployment(theomittedcategory)orearnings.
Studiesthatmodelthehazardtoanewjobarealsosomewhatmorelikelytoobtain
positivefindings.21Weinferthatsomecautioniswarrantedininterpretingtheshort
termimpactestimatesfromstudiesthatuseoutcomesotherthanemploymentor
earnings.
d.AreSomeProgramsBetter(orWorse)forDifferentParticipantGroups?
AlongstandingquestionintheALMPliteratureiswhethercertainparticipant
groupswouldhavebetteroutcomesiftheywereassignedtospecifictypesofprograms.
WeaddressthisinTable8,whichpresentsseparatemetaanalysismodelsfortheeffect
sizesfromdifferenttypesofALMP's.
Asabenchmarkcolumn1presentsabaselinespecificationfittoall5program
types,withdummiesfortheprogramtypes(notreported)andcontrolsfortheintake
group,thegendergroup,andtheagegroup.22The(omitted)basegroupiscomprisedof
mixedgenderandagegroupsfromtheregularUIrolls.Inthispooledspecificationthe
estimatedeffectsforfemalesandlongtermunemployedparticipantsaresignificantly
20
WealsoestimatedseparateeffectsizemodelsfordifferenttypesofparticipantsͲͲthosefromthe
regularUIsystemversuslongtermunemployedordisadvantagedgroups.Wefoundasignificantpositive
coefficientforfemaleparticipantsinthemodelsforbothUIrecipientsandthelongtermunemployed.
21
Estimatesfrominteractedmodelsthatallowdifferenteffectsofthedependentvariableatdifferent
timehorizons(notreportedinthetable)showthatthepositivebiasassociatedwiththeuseofexit
hazardsislargelyconfinedtoshorttermimpacts.
22
Thisisasimplifiedversionofthespecificationreportedincolumn2ofTable5andcolumn1ofTable7.
23
positive,whilethecoefficientforolderparticipantsissignificantlynegative,andthe
coefficientforyoungparticipantsisnegativeandmarginallysignificant.
Columns2Ͳ6reportestimatesforthesamespecification(minusthecontrolsfor
thetypeofprogram)fitseparatelytoeffectsizesforeachofthe5programtypes.
ComparisonsacrossthesemodelssuggestthatlongͲtermunemployedparticipants
benefitrelativelymorefrom"humancapital"programs(i.e.,trainingandprivatesector
employment),andrelativelylessfrom"workfirst"programs(i.e.,jobsearchand
sanction/threatprograms).Incontrast,disadvantagedparticipantsappeartobenefit
morefromworkfirstprogramsandlessfromhumancapitalprograms.Female
participantsalsoappeartobenefitrelativelymorefromhumancapitalprograms,while
therelativeeffectsforyouthsandolderparticipantsarenotmuchdifferentacrossthe
programtypes.
Overalltheseresultssuggestthattheremaybepotentialgainstomatching
specificparticipantgroupstospecifictypesofprograms,thoughthesmallsamplesizes
formostoftheprogramtypesmustbenoted.Attemptstoexpandthepowerofthe
analysisbyusingOPmodelsforthesignandsignificanceoftheprogramestimateslead
togenerallysimilarconclusionsastheeffectsizemodelsreportedinTable8withonly
modestgainsinprecision.
e.EffectsofCyclicalConditions
AnotherlongstandingquestionintheALMPliteratureiswhetherprogramsare
more(orless)effectiveindifferentcyclicalenvironments.23Oneviewisthatactive
programswilltendtohavesmallereffectsinadepressedlabormarketbecause
participantshavetocompetewithother,moreadvantagedworkersforalimitedsetof
jobs.AnalternativeviewisthatALMP'saremoreeffectiveinweaklabormarkets
23
Arelatedquestioniswhetherprogramexternalitiesarebiggerorsmallerinweakorstronglabor
markets.ThisisaddressedinaninnovativeexperimentconductedbyCreponetal.(2013).
24
becauseemployersbecomemoreselectiveinaslackmarket,increasingthevalueofan
intervention(particularlythosethatraisehumancapital).
ThreepreviousstudieshaveinvestigatedALMPeffectivenessoverthebusiness
cycle:Kluve(2010)usesbetweenͲcountryvariationinasmallEuropeanmetadataset,
whileLechnerandWunsch(2009)andForslundetal.(2011)specificallyanalyze
programsinGermanyandSweden,respectively.Allthreestudiessuggestapositive
correlationbetweenALMPeffectivenessandtheunemploymentrate.
Toprovidesomenewevidenceweaddedtwoalternativecontextualvariablesto
ouranalysis,representingtheaveragegrowthrateofGDPandtheaverage
unemploymentrateduringtheyearsthetreatmentgroupparticipatedintheprogram.
Sincegrowthratesandunemploymentratesvarywidelyacrosscountries,wealso
introducedasetofcountrydummiesthatabsorbanypermanentdifferencesinlabor
marketconditionsacrosscountries.Theeffectofthesedummiesisinterestinginitsown
rightbecausethesharesofdifferentprogramtypesandparticipantgroupsalsovary
widelyacrosscountries,leadingtothepossibilityofbiasinthemeasuredeffectsof
programtypesandparticipantgroupsifthereareunobservedcountryspecificfactors
thataffecttheaveragesuccessofALMP'sindifferentcountries.
TheresultsofouranalysisaresummarizedinTable9.Forreferencecolumn1
presentsabenchmarkspecificationidenticaltothesimplifiedeffectsizemodelin
column1ofTable8.Column2presentsthesamespecificationwiththeadditionof37
countrydummies.Theadditionofthesedummiesleadstosomemodestbutinteresting
changesintheestimatedcoefficientsinthemetaanalysismodel.Mostnotably,the
coefficientsassociatedwithjobsearchassistance(JSA)andsanction/threatprograms
bothbecomemorenegative,indicatingthat"workfirst"programstendtobemore
widelyusedincountrieswhereallformsofALMP'sarerelativelysuccessful.
Column3presentsamodelthatincludesthecontrolforaverageGPDgrowth
rateduringtheprogramperiod.Thecoefficientisnegativeandmarginallysignificant
(t=1.78)providingsuggestiveevidencethatALMP'sworkbetterinrecessionarymarkets.
25
Amodelthatcontrolsfortheaverageunemploymentrateshowsthesametendency
(coefficient=0.014,standarderror=0.016)thoughtheeffectislessprecise.
Aconcernwiththespecificationincolumn3isthattheaveragenumberof
programestimatespercountryissmall(manycountrieshaveonly2or3estimates)
leadingtopotentialoverͲfitting.Toaddressthis,weestimatedthemodelsincolumns
4Ͳ6,usingonlydatafromthefourcountriesthataccountforthelargestnumbersof
effectsizeestimatesͲDenmark(17estimates),France(20estimates),Germany(147
estimates)andtheU.S.(16estimates).Asshownincolumn4,ourbaselinespecification
yieldscoefficientestimatesthatarequitesimilartotheestimatesfromtheentire
sample,thoughtherelativeimpactsofJSAandsanction/threatprogramsaremore
negativeinthese4countries.
Columns5and6presentmodelsthataddtheaverageGDPgrowthrateandthe
averageunemploymentrate,respectively,tothisbaselinemodel.Thesespecifications
suggestrelativelyimportantcyclicaleffectsonALMPeffectiveness.Forexample,
comparingtwosimilarprogramsoperatinginlabormarketswitha3percentagepoint
gapingrowthrates,theprogramintheslowergrowthenvironmentwouldbeexpected
tohavea0.2largereffectsize.
Whilenotreportedinthetable,wealsoestimatedmetaanalysismodelsfor
thesefourcountriesthatincludeaninteractionbetweenthecyclicalvariableandan
indicatorforhumancapitaltypeprograms(i.e.trainingorprivateemployment
programs).UsingGDPgrowthasthecyclicalindicatortheestimatedcoefficientsare
Ͳ0.048(standarderror=0.023)forthemaineffectandͲ0.028(standarderror=0.017)
fortheinteraction.Usingunemploymentasthecyclicalindicatorthepatternsofthe
maineffectandtheinteractionaresimilarbutlessprecise.24Bothmodelstherefore
suggestthattheimpactsoftrainingandprivatesectoremploymentprogramsaremore
24
Theestimatedmaineffectofaverageunemploymentis0.053(standarderror=0.038),theestimated
interactioneffectis0.0230(standarderror=0.036).
26
countercyclicalthantheimpactsofJSA,sanction/threatandpublicsectorprograms,
thoughtheinteractiontermsarenotsignificantatconventionallevels.
WhiletheevidenceinTable9suggestsacountercyclicalpatternofprogram
effectiveness,itisworthemphasizingthattheexplanationforthispatternislessclear.It
ispossiblethatthevalueofagivenprogramishigherinarecessionaryenvironment.It
isalsopossible,however,thatthecharacteristicsofALMPparticipants,orofthe
programsthemselves,changeinawaythatcontributestoamorepositiveimpactina
slowͲgrowth/highͲunemploymentenvironment.
V.SummaryandConclusions
Wehaveassembledandanalyzedanewsampleofimpactestimatesfrom207
studiesofactivelabormarketpolicies.Buildingonourearlierstudy(CKW),weargue
thatitisimportanttodistinguishbetweenimpactsatvarioustimehorizonssince
completionoftheprogram,andtoconsiderhowthetimeprofileofimpactsvariesby
thetypeofALMP.Wealsostudytheimportanceofparticipantheterogeneity,andlook
forevidencethatspecificsubgroupsmaybenefitmoreorlessfromparticulartypesof
programs.Finally,westudyhowthestateofthelabormarketaffectsthemeasured
effectivenessofALMP's.
WithregardtotheimpactsofdifferenttypesofALMP's,wefindthatthetime
profilesof"workfirst"stylejobsearchassistanceandsanction/threatprogramsdiffer
fromtheprofilesof"humancapital"styletrainingandprivatesectoremployment
subsidies.Workfirstprogramstendtohavelargershorttermeffects,whereashuman
capitalprogramshavesmall(orinsomecasesevennegative)shorttermimpacts,
coupledwithlargerimpactsinthemediumorlongerrun(2Ͳ3yearsaftercompletionof
theprogram).Wealsoconfirmthatpublicsectoremploymentprogramshave
negligible,orevennegativeprogramimpactsatalltimehorizons.
Withregardtodifferentparticipantgroups,wefindthatfemaleparticipantsand
thosedrawnfromthepooloflongtermunemployedtendtohavelargerprogram
27
effectsthanothergroups.Incontrast,theprogramestimatesforyouthsandolder
workersaretypicallylesspositivethanforothergroups.Wealsofindindicationsof
potentialgainstomatchingdifferentparticipantgroupstospecificprograms,with
evidencethatworkfirstprogramsarerelativelymoresuccessfulfordisadvantaged
participants,whereashumancapitalprogramsaremoresuccessfulforthelongterm
unemployed.
Withregardtothestateofthelabormarket,wefindthatALMP'stendtohave
largerimpactsinperiodsofslowgrowthandhigherunemployment.Inparticular,we
findarelativelylargecyclicalcomponentintheprogramestimatesfromfourcountries
thataccountforoneͲhalfofoursample.Wealsofindsuggestiveevidencethathuman
capitalprogramsaremorecyclicallysensitivethanworkfirstprograms.
Ourfindingsontherelativeefficacyofhumancapitalprogramsforlongterm
unemployed,andonthelargerimpactsoftheseprogramsinrecessionary
environments,pointtoapotentiallyimportantpolicylesson.AsnotedbyKrueger,Judd
andCho(2014)andKroftetal.(forthcoming),thenumberoflongtermunemployed
risesrapidlyasarecessionpersists.Thisgrouphasahighprobabilityofleavingthe
laborforce,riskingpermanentlossesintheproductivecapacityoftheeconomy.One
policyresponseiscountercyclicaljobtrainingprogramsandprivateemployment
subsidies,whichareparticularlyeffectiveforthelongerͲtermunemployedina
recessionaryclimate.
Methodologically,wefindanumberofinterestingpatternsintherecentALMP
literature.Mostimportantly,wefindthattheestimatedimpactsderivedfrom
randomizedcontrolledtrials,whichaccountforoneͲfifthofoursample,arenotmuch
differentonaveragefromthenonͲexperimentalestimates.Wealsofindnoevidenceof
"publicationbias"intherelationshipbetweenthemagnitudeofthepointestimates
fromdifferentstudiesandtheircorrespondingprecision.Theestimatedimpactsare
alsoverysimilarfrompublishedandunpublishedstudies,andfrommoreandlesscited
studies.Wedofindthatthechoiceofoutcomevariableusedintheevaluationmatters,
28
withatendencytowardmorepositiveshorttermimpactestimatesfromstudiesthat
modelthetimetofirstjobthanfromstudiesthatmodeltheprobabilityofemployment
orthelevelofearnings.
Finally,weconcludethatmetaanalyticmodelsbasedonthesignand
significanceoftheprogramimpactsleadtogenerallysimilarconclusionsasmodels
basedoneffectsizes.Thisarisesbecausemuchofthevariationinthesignand
significanceofestimatedimpactsacrossstudiesintheALMPliteratureisdrivenby
variationinestimatedeffectsizes,ratherthanbyvariationinthecorresponding
samplingerrors.
29
References
Ashenfelter,Orley(1978).“EstimatingtheEffectofTrainingProgramsonEarnings”
ReviewofEconomicsandStatistics60(1):47Ͳ57.
Ashenfelter,OrleyandDavidCard(1985).“UsingtheLongitudinalStructureofEarnings
toEstimatetheEffectofTrainingPrograms.”ReviewofEconomicsandStatistics67(4):
648Ͳ660.
Card,DavidandDeanR.Hyslop(2005)."EstimatingtheEffectofaTimeͲLimitedEarnings
SubsidyforWelfareRecipients."Econometrica73(6):1723Ͳ1770.
Card,D.,J.KluveandA.Weber(2010)."ActiveLabourMarketPolicyEvaluations:A
MetaͲanalysis."EconomicJournal120(548):F452ͲF477.
Cho,Y.andM.Honorati(2014).“EntrepreneurshipProgramsinDevelopingCountries:A
MetaRegressionAnalysis.”LabourEconomics28(C):110Ͳ130.
Crépon,Bruno,EstherDuflo,MarcGurgand,RolandRathelot,andPhilippeZamora
(2013)."DoLaborMarketPolicieshaveDisplacementEffects?EvidencefromaClustered
RandomizedExperiment."QuarterlyJournalofEconomics128(2):531Ͳ580.
Filges,Trine,GeirSmedslund,AnneͲSofieDueKnudsen,andAnneͲMarieKlintJørgensen
(2015)."ActiveLabourMarketProgrammeParticipationforUnemploymentRecipients:
ASystematicReview."CampbellSystematicReviews2015(2).
Forslund,A.,P.FredrikssonandJ.Vikström(2011).“WhatActiveLaborMarketPolicy
WorksInaRecession?”.NordicEconomicPolicyReview1:171Ͳ207.
Gladden,TriciaandChristopherTaber(2000)."WageProgressionAmongLessSkilled
Workers."InRebeccaM.BlankandDavidCard,editors,FindingWork:JobsandWelfare
Reform.NewYork:RussellSageFoundation.
Greenberg,DavidH.,CharlesMichalopoulosandPhilipK.Robins(2003).“AMetaͲ
AnalysisofGovernmentͲSponsoredTrainingPrograms.”IndustrialandLaborRelations
Review57():31Ͳ53.
Grimm,M.andA.L.Paffhausen(2015).“DoInterventionsTargetedatMicroͲ
EntrepreneursandSmallandMediumͲsizedFirmsCreateJobs?ASystematicReviewof
theEvidenceforLowandMiddleIncomeCountries”.LabourEconomics32:67Ͳ85.
30
Ham,JohnC.andRobertJ.Lalonde(1996)."TheEffectofSampleSelectionandInitial
ConditionsinDurationModels:EvidencefromExperimentalDataonTraining."
Econometrica64(1):175Ͳ205.
Heckman,JamesJ.,HidehikoIchimura,JeffreyA.Smith,andPetraTodd(1998).
“CharacterizingSelectionBiasUsingExperimentalData.”Econometrica66(5):1017Ͳ
1098.
Hedges,LarryV.(1981)."DistributiontheoryforGlass'estimatorofeffectsizeand
relatedestimators".JournalofEducationalStatistics6(2):107–128.
Hedges,LarryV.andIngramOlkin(1985).StatisticalMethodsforMetaͲAnalysis.
Orlando:AcademicPress.
Ibarrarán,P.andD.Rosas(2009),EvaluatingtheImpactofJobTrainingProgramsin
LatinAmerica:EvidencefromIDBfundedoperations,JournalofDevelopment
Effectiveness1:195Ͳ216.
Kluve,J.(2010)."TheEffectivenessofEuropeanActiveLaborMarketPrograms."Labour
Economics17:904Ͳ918.
Kroft,Kory,FabienLange,MatthewJ.Notowidigdo,andLawrenceF.Katz.
(forthcoming)."LongͲtermUnemploymentandtheGreatRecession:TheRoleof
Composition,DurationDependence,andNonͲParticipation.JournalofLaborEconomics.
Krueger,AlanB.,JuddCramer,andDavidCho(2014)."AretheLongͲTermUnemployed
ontheMarginsoftheLaborMarket?"BrookingsPapersonEconomicsActivitySpring:
229Ͳ280.
Lalonde,RobertJ(1986).“EvaluatingtheEconometricEvaluationsofTrainingPrograms
withExperimentalData.”AmericanEconomicReview76(4):604Ͳ620.
Lalonde,RobertJ(2003)."EmploymentandTrainingPrograms."InRobertA.Moffit,
editor,MeansTestedTransferProgramsintheUnitedStates.Chicago:Universityof
ChicagoPress.
Lechner,MichaelandConnieWunsch(2009).“AreTrainingProgramsMoreEffective
whenUnemploymentisHigh?”JournalofLaborEconomics27(4):653Ͳ692.
Martin,JohnP.(2014)"ActivationandActiveLabourMarketPoliciesinOECDCountries:
StylizedFactsandEvidenceonTheirEffectiveness."IZAPolicyPaperNo.84.
31
Mincer,Jacob.(1974).Schooling,ExperienceandEarnings.NewYork:NationalBureauof
EconomicResearch.
32
NumberofProgramEstimates
0
10
20
30
40
50
60
70
80
90
100
1980
1989
1994
YearofProgramStart
1999
2004
2009
NonͲexperimentalDesign
ExperimentalDesign
Figure1:NumberofProgramEstimates,ByYearofProgramStart
Ͳ0.8
Ͳ0.6
Ͳ0.4
0
0.2
ShortTermEffectSize
Ͳ0.2
Note:3largepositiveestimatedeffectsizesnotshown.
Ͳ1
0.4
0.6
Figure2a:ShortTermEffectSizesandConfidenceIntervals
0.8
1
Ͳ1
Ͳ0.8
Ͳ0.6
Ͳ0.4
0
0.2
MediumTermEffectSize
Ͳ0.2
0.4
0.6
Figure2b:MediumTermEffectSizesandConfidenceIntervals
0.8
1
Ͳ1
Ͳ0.8
Ͳ0.6
Ͳ0.4
Ͳ0.2
0
0.2
LongTermEffectSize
0.4
0.6
Figure2c:LongTermEffectSizesandConfidenceIntervals
0.8
1
Table1:DescriptionofSampleofProgramEstimates
Austria,
Germany,
Fullsample Switzerland
(1)
(2)
CountryofStudy:
U.S.,U.K,
Nordic
Aust.,N.Z.,
Countries
Canada
(3)
(4)
LatinAmer.
and
NonͲOECD Carribean
(5)
(6)
Numberofestimates
NumberofPPS's
Numberofstudies
857
526
207
290
163
52
212
127
48
87
45
24
132
86
33
72
54
19
Typeofprogram(%):
Training
JobSearchAssistance
PrivateSubsidy
PublicEmployment
Sanctions/Threat
49
15
13
8
16
62
8
17
9
5
17
26
15
10
32
45
22
5
3
25
79
2
11
6
2
97
0
3
0
0
Ageofprogramgroup(%):
Mixed
Youth(<25years)
Older(ш25years)
56
21
23
54
12
33
61
20
19
72
15
13
40
53
8
25
69
6
GenderofProgramgroup(%):
Mixed
51
Malesonly
24
Femalesonly
24
53
24
23
67
18
16
43
25
32
43
23
31
11
44
44
TypeofProgramParticipants(%):
Registeredunemployed
64
LongͲtermunemployed
10
Disadvantaged
26
86
8
6
67
10
23
33
25
41
24
7
69
0
0
100
OutcomeofInterest(%):
Employmentstatus
Earnings
Hazardtonewjob
Otherhazard
Unemploymentstatus
56
23
11
6
5
83
8
7
0
2
31
25
25
16
4
26
47
3
2
21
63
36
0
0
1
54
43
0
3
0
EffectMeasuredat(%):
ShortTerm
MediumTerm
LongTerm
47
35
18
42
34
23
54
31
16
37
40
23
47
45
8
57
42
1
ExperimentalDesign(%)
19
0
39
31
28
26
Note:seetextfordescriptionofsample.Studyreferstoanarticleorunpublishedpaper.PPSreferstoa
programandparticipantsubgroup(e.g.,ajobsearchassistanceprogramformixedgenderyouths).
EstimatereferstoanestimateoftheeffectoftheprogramontheparticipantsubgroupateitherashortͲ
term(<1yearaftercompletionoftheprogram),mediumterm(1Ͳ2yearspostcompletion)orlongterm(2+
yearspostcompletion)timehorizon.
207
NumberofPPS's
Numberofstudies
40
42
18
SignificantpositiveSTestimateͲͲpct.ofSTsample
InsignificantSTestimateͲͲpct.ofSTsample
SignificantnegativeSTestimateͲͲpct.ofSTsample
40
8
InsignificantMTestimateͲͲpct.ofMTsample
SignificantnegativeMTestimateͲͲpct.ofMTsample
35
4
InsignificantLTestimateͲͲpct.ofsample
SignificantnegativeLTestimateͲͲpct.ofsample
3
32
65
91[19]
9
41
50
194[40]
22
47
31
205[42]
111
274
490
Potentially
HaveEffectSize
(2)
3
32
65
68[19]
10
43
47
143[41]
23
44
33
141[40]
83
200
352
Have
EffectSize
(3)
Ͳ9.7(0.1)
2.7(1.2)
27.6(6.1)
18.5(5.0)
Ͳ10.9(2.3)
2.5(0.8)
24.2(4.3)
11.5(2.8)
Ͳ14.4(2.4)
0.5(1.2)
21.4(3.3)
3.8(2.3)
ͲͲ
ͲͲ
ͲͲ
MeanEffectSize
forCol.3Sample
(robuststd.error)a
(4)
a
Entriesinthiscolumnaremeaneffectsizesforthesubsetindicatedbyrowheading.Standarderrorofmeanisclusteredbystudy.
Notes:seenotetoTable1.Shorttermprogramestimatesarefortheperiodupto1yearafterthecompletionoftheprogram.Mediumterm
estimatesarefortheperiodfrom1to2yearsaftercompletionoftheprogram.Longtermestimatesarefortheperiod2ormoreyearsafter
completionoftheprogram.Effectsizesareonlyavailableforstudiesthatmodeltheprobabilityofemploymentastheoutcomeofinterest,and
provideinformationonmeanemploymentrateofcomparisongroup.
61
SignificantpositiveLTestimateͲͲpct.ofsample
AllLTestimatesͲͲnumber[pct.oftotalestimates]
141[16]
52
SignificantpositiveMTestimateͲͲpct.ofMTsample
LongTerm(LT)Estimates
301[35]
AllMTestimatesͲͲnumber[pct.oftotalestimates]
MediumTerm(MT)Estimates
415[48]
AllSTestimatesͲͲnumber[pct.oftotalestimates]
ShortTermEstimates
857
526
Numberofestimates
Fullsample
(1)
Table2:SummaryofProgramEstimatesbyAvailabilityofEffectSize
418
129
118
76
116
554
106
197
ByProgramType:
Training
JobSearchAssist.
PrivateSubsidy
PublicSectorEmp.
Sanction/Threat
ByIntakeGroup:
UIRecipients
LongTermUnem.
Disadvantaged
7,027
8,900
11,000
17,391
17,084
10,000
4,648
7,700
10,709
Median
Sample
Size
(2)
27.4
16.0
17.1
31.0
0.0
8.5
51.2
12.9
19.4
Percent
RCT's
(3)
17.1
(17)
Ͳ0.6
(93)
17.8
(8)
Ͳ4.2
(14)
3.2
(13)
4.1
(16)
3.9
(90)
3.8
(141)
29.9
(16)
8.7
(101)
14.6
(9)
Ͳ2.2
(12)
11.7
(17)
4.4
(13)
14.0
(92)
11.5
(143)
30.7
(10)
17.3
(50)
3.6
(4)
2.5
(6)
45.9
(16)
2.3
(7)
13.6
(35)
18.5
(68)
MeanEffectSizes(×100)
Short
Medium
Longer
Term
Term
Term
(4)
(5)
(6)
50
(50)
34
(258)
52
(56)
32
(41)
37
(49)
53
(68)
35
(201)
40
(415)
65
(40)
47
(193)
38
(37)
25
(24)
65
(37)
63
(40)
54
(163)
52
(301)
63
(16)
59
(103)
43
(23)
27
(11)
88
(32)
43
(21)
67
(54)
61
(141)
Pct.withSig.PositiveImpact
Short
Medium
Longer
Term
Term
Term
(7)
(8)
(9)
9.8
11.1
10.4
50
59
68
(31)
(26)
(8)
(107)
(68)
(22)
Notes:seeTables1and2.Numberofprogramestimatesassociatedwitheachtableentryisreportedinparentheses.Effectsizesareonly
availableforstudiesthatmodeltheprobabilityofemploymentastheoutcomeofinterest.
857
All
Number
Est's.
(1)
Table3a:ComparisonofImpactEstimatesbyProgramTypeandParticipantGroup
505
180
172
466
191
200
166
691
ByAge:
MixedAge
Youth(<25)
NonͲYouth
ByGender:
MixedGender
MalesOnly
FemalesOnly
ByEvaluationDesign:
Experimental
NonͲexperimental
16,000
1,471
8,345
10,000
11,000
25,850
3,000
10,000
10,709
Median
Sample
Size
(2)
0.0
100.0
22.5
15.2
19.7
12.8
33.3
16.6
19.4
Percent
RCT's
(3)
2.1
(113)
10.6
(28)
9.4
(28)
Ͳ2.8
(24)
3.9
(89)
0.3
(36)
7.5
(34)
3.8
(71)
3.8
(141)
12.7
(118)
5.6
(25)
16.5
(30)
12.4
(28)
9.4
(85)
8.9
(30)
5.9
(29)
14.4
(84)
11.5
(143)
23.5
(53)
0.8
(15)
34.5
(14)
29.2
(9)
11.3
(45)
9.6
(12)
0.1
(5)
22.4
(51)
18.5
(68)
MeanEffectSizes(×100)
Short
Medium
Longer
Term
Term
Term
(4)
(5)
(6)
40
(337)
40
(78)
41
(96)
41
(95)
39
(224)
31
(85)
32
(92)
47
(238)
40
(415)
55
(243)
41
(58)
55
(74)
50
(72)
52
(155)
51
(59)
41
(64)
57
(178)
52
(301)
68
(111)
37
(30)
70
(30)
58
(24)
59
(87)
43
(28)
67
(24)
65
(89)
61
(141)
Pct.withSig.PositiveImpact
Short
Medium
Longer
Term
Term
Term
(7)
(8)
(9)
Notes:seeTables1and2.Numberofprogramestimatesassociatedwitheachtableentryisreportedinparentheses.Effectsizesareonly
availableforstudiesthatmodeltheprobabilityofemploymentastheoutcomeofinterest.
857
All
Number
Est's.
(1)
Table3b:AdditionalComparisonsofImpactEstimatesbyParticipantGroupsandDesign
Ͳ0.055
(0.126)
9
PrivateSubsidy
0.013
(0.035)
6
Sanction/Threat
Ͳ0.048
(0.021)
4
Ͳ0.299
(0.299)
2
Ͳ0.006
(0.156)
2
Ͳ0.005
(0.003)
7
0.087
(0.035)
28
Ͳ0.029
(0.012)
4
Ͳ0.039
(0.039)
2
Ͳ0.005
(0.031)
6
Ͳ0.004
(0.006)
7
Ͳ0.010
(0.011)
28
0.000
(0.108)
27
0.158
(0.170)
19
0.083
(0.150)
24
0.265
(0.095)
34
0.314
(0.072)
121
0.158
(0.182)
19
Ͳ0.143
(0.494)
7
0.167
(0.267)
12
0.143
(0.167)
21
0.439
(0.085)
41
0.211
(0.092)
19
Ͳ0.143
(0.285)
7
Ͳ0.062
(0.068)
16
Ͳ0.111
(0.144)
18
0.048
(0.049)
42
ChangeinSign/Significance
shorttermto shorttermto mediumterm
mediumterm
longterm
tolongterm
(4)
(5)
(6)
0.231
0.250
0.020
(0.055)
(0.103)
(0.052)
225
100
102
Notes:Changeineffectsizeincolumn1representsthedifferencebetweenthemediumtermandshorttermeffectsizesforagivenprogramand
participantsubgroup(PPS).Changesincolumns2and3aredefinedanalogously.Changeinsign/significanceincolumn4isdefinedas+1iftheshortterm
estimateissignificantlynegativeandthemediumtermestimateisinsignificant,oriftheshorttermestimateisinsignificantandthemediumtermestimate
issignificantlypositive;0ifthesignandsignificanceoftheshorttermandmediumtermestimatesisthesame;andͲ1iftheshorttermestimateis
significantlypositiveandthemediumtermestimateisinsignificant,oriftheshorttermestimateisinsignificantandthemediumtermestimateis
significantlynegative.Changesincolumns5and6aredefinedanalogously.Standarddeviations(clusteredbystudynumber)inparenthesis.
NumberStudies
NumberStudies
Ͳ0.007
(0.070)
10
PublicSectorEmp.
NumberStudies
NumberStudies
0.009
(0.019)
10
0.070
(0.018)
70
JobSearchAssist.
NumberStudies
ByProgramType
Training
NumberStudies
All
ChangeinEffectSize
shorttermto shorttermto mediumterm
mediumterm
longterm
tolongterm
(1)
(2)
(3)
0.043
0.037
Ͳ0.012
(0.020)
(0.035)
(0.007)
105
43
47
Table4:TransitionsinProgramImpactsforaGivenProgramandParticipantSubgroup
Table5:EstimatedEffectSizeModels
(1)
EffectTerm(Omitted=ShortTerm)
MediumTerm
0.071
(0.027)
LongTerm
0.131
(0.044)
ProgramType(Omitted=Training)
JobsearchAssist.
Ͳ0.059
(0.027)
DependentVariable=EstimatedEffectSize
(2)
(3)
(4)
0.056
(0.021)
0.091
(0.038)
0.101
(0.037)
0.097
(0.040)
0.088
(0.025)
0.099
(0.040)
Ͳ0.012
(0.043)
0.002
(0.026)
0.029
(0.044)
PrivateSubsidy
0.094
(0.068)
0.086
(0.057)
Ͳ0.007
(0.091)
0.044
(0.099)
PublicSectorEmp.
Ͳ0.120
(0.034)
Ͳ0.152
(0.044)
Ͳ0.081
(0.055)
Ͳ0.084
(0.062)
0.036
(0.071)
InteractionwithMediumTerm:
JobsearchAssist.
0.007
(0.094)
0.139
(0.068)
0.108
(0.098)
Ͳ0.098
(0.043)
Ͳ0.092
(0.041)
PrivateSubsidy
Ͳ0.016
(0.102)
Ͳ0.055
(0.104)
PublicSectorEmp.
Ͳ0.081
(0.070)
Ͳ0.09
(0.073)
Sanction/Threat
Ͳ0.133
(0.048)
Ͳ0.105
(0.045)
Ͳ0.115
(0.041)
Ͳ0.083
(0.052)
0.329
(0.142)
0.182
(0.127)
Sanction/Threat
InteractionwithLongTerm:
JobsearchAssist.
PrivateSubsidy
PublicSectorEmp.
Ͳ0.030
Ͳ0.156
(0.081)
(0.108)
Sanction/Threat
Ͳ0.239
Ͳ0.273
(0.073)
(0.092)
AdditionalControls
No
Yes
No
Yes
RSquared
0.13
0.33
0.21
0.37
Notes:Standarderrors(clusteredbystudy)inparentheses.Modelsarelinearregressions
withtheeffectsizeasdependentvariable.Coefficientsofadditonalcontrolvariables
includedinmodelsincolumns2and4reportedinTable7.Samplesizeis352.
Table6:OrderedProbitModelsforSign/SignificanceofEstimatedProgramImpacts
DependentVariable=OrdinalIndicatorforSign/Significance
(1)
(2)
(3)
(4)
(5)
EffectTerm(Omitted=ShortTerm)
MediumTerm
0.372
0.483
0.563
0.639
0.491
(0.088)
(0.099)
(0.130)
(0.138)
(0.145)
LongTerm
0.597
0.742
0.901
1.053
1.030
(0.157)
(0.167)
(0.175)
(0.171)
(0.206)
ProgramType(Omitted=Training)
JobsearchAssist.
0.274
(0.156)
0.286
(0.168)
0.531
(0.180)
0.532
(0.197)
0.569
(0.459)
PrivateSubsidy
0.139
(0.189)
0.076
(0.210)
Ͳ0.04
(0.224)
Ͳ0.132
(0.263)
Ͳ0.166
(0.438)
PublicSectorEmp.
Ͳ0.677
(0.219)
Ͳ0.758
(0.228)
Ͳ0.383
(0.276)
Ͳ0.489
(0.279)
Ͳ1.399
(0.496)
Sanction/Threat
Ͳ0.110
(0.172)
Ͳ0.205
(0.184)
0.318
(0.206)
0.202
(0.236)
1.148
(0.653)
Ͳ0.289
(0.235)
Ͳ0.283
(0.249)
Ͳ0.004
(0.343)
PrivateSubsidy
0.138
(0.289)
0.226
(0.311)
0.353
(0.486)
PublicSectorEmp.
Ͳ0.645
(0.285)
Ͳ0.573
(0.288)
0.051
(0.477)
Sanction/Threat
Ͳ0.764
(0.226)
Ͳ0.705
(0.245)
Ͳ0.662
(0.278)
Ͳ1.017
(0.313)
Ͳ1.022
(0.294)
Ͳ0.832
(0.313)
PrivateSubsidy
0.611
(0.375)
0.58
(0.387)
1.274
(0.798)
PublicSectorEmp.
Ͳ0.643
(0.490)
Ͳ0.675
(0.497)
0.131
(0.832)
Sanction/Threat
Ͳ0.999
(0.353)
Ͳ1.021
(0.375)
Ͳ1.638
(0.430)
InteractionwithMediumTerm:
JobsearchAssist.
InteractionwithLongTerm :
JobsearchAssist.
AdditionalControls
No
Yes
No
Yes
Yes
NumberofObservations
857
857
857
857
352
LogLikelihood
Ͳ801
Ͳ765
Ͳ786
Ͳ752
Ͳ283
Notes:Standarderrors(clusteredbystudy)inparentheses.Modelsareorderedprobits,fittoordinaldata
withvalueof+1forsignificantlypositive,0forinsignificant,Ͳ1forsignificantlynegativeestimate.Estimated
cutpoints(2permodel)arenotreportedintheTable.Modelincolumn5isfittosubsampleofestimatesfor
whichaneffectsizeestimateisavailable.Coefficientsforadditionalcontrolvariablesformodelsincolumns
2,4and5arereportedinTable7.
Table7:EstimatedCoefficientsforAdditionalControlVariablesforModelsReportedinTables5and6
EffectSizeOLSModels
(1)
(2)
OutcomeofInterest(Omitted=ProbabilityofEmployment)
Earnings
OrderedProbitModelsforSign/Significance
(3)
(4)
(5)
Ͳ0.003
(0.130)
Ͳ0.01
(0.132)
HazardtoNewJob
0.275
(0.211)
0.264
(0.212)
OtherHazard
0.613
(0.275)
0.598
(0.293)
0.547
(0.263)
0.591
(0.285)
UnemploymentStatus
AgeofProgramGroup(Omitted=Mixed)
Youths(<25)
Ͳ0.062
(0.045)
Older(>=25)
Ͳ0.151
(0.044)
Ͳ0.05
(0.045)
Ͳ0.135
(0.045)
Ͳ0.368
(0.151)
Ͳ0.423
(0.157)
Ͳ0.348
(0.153)
Ͳ0.425
(0.160)
Ͳ0.518
(0.287)
Ͳ0.671
(0.297)
GenderofProgramGroup(Omitted=Mixed)
Malesonly
0.029
(0.049)
0.020
(0.049)
Ͳ0.007
(0.149)
Ͳ0.006
(0.149)
Ͳ0.328
(0.266)
0.107
(0.052)
0.094
(0.051)
0.064
(0.144)
0.053
(0.146)
0.000
(0.250)
0.107
(0.073)
0.082
(0.072)
0.250
(0.192)
0.176
(0.196)
0.910
(0.488)
Ͳ0.07
(0.081)
0.04
(0.072)
Ͳ0.078
(0.076)
0.055
(0.065)
0.177
(0.241)
0.131
(0.201)
0.14
(0.236)
0.096
(0.202)
1.231
(0.579)
0.618
(0.378)
0.019
(0.060)
Ͳ0.01
(0.129)
0.009
(0.059)
Ͳ0.003
(0.130)
0.125
(0.187)
0.108
(0.338)
0.088
(0.189)
0.1
(0.338)
0.738
(0.483)
1.012
(0.826)
Ͳ0.063
(0.281)
Ͳ0.064
(0.286)
1.124
(0.529)
0.542
(0.228)
0.388
(0.181)
0.527
(0.228)
0.404
(0.179)
0.356
(0.623)
0.392
(0.332)
Ͳ0.135
(0.179)
Ͳ0.122
(0.177)
Ͳ0.55
(0.232)
Femalesonly
CountryGroup(Omitted=Nordic)
Germanic
Anglo
EastEurope
RestofEurope
LatinAmerica
RemainingCountries
0.084
0.089
(0.091)
(0.092)
TypeofProgramParticipant(Omitted=RegisteredUnemployed)
Disadvantaged
0.06
0.049
(0.089)
(0.088)
LongͲtermUnemployed
0.217
0.212
(0.076)
(0.074)
OtherControls:
Progam>9Months
Ͳ0.056
Ͳ0.043
(0.041)
(0.042)
Experiment
Ͳ0.031
(0.049)
Ͳ0.027
(0.049)
Ͳ0.065
(0.170)
Ͳ0.095
(0.170)
Ͳ0.314
(0.332)
SquareRootofSamplesize
Ͳ0.039
(0.086)
Ͳ0.028
(0.077)
0.159
(0.184)
0.098
(0.191)
0.484
(0.706)
PublishedArticle
Ͳ0.056
(0.043)
Ͳ0.063
(0.043)
Ͳ0.203
(0.133)
Ͳ0.213
(0.132)
Ͳ0.41
(0.254)
CitationsRankIndex
Ͳ0.003
(0.004)
Ͳ0.002
(0.003)
0.007
(0.012)
0.005
(0.012)
Ͳ0.005
(0.024)
NumberofObservations
352
352
857
857
352
Notes:Standarderrors(clusteredbystudy)inparentheses.Tableentriesarecoefficientestimatesforadditionalcontrols
includedinmodelsinTables5and6.Modelsincolumns1Ͳ2correspondtomodelsincolumns2and4ofTable5.Models
incolumns3Ͳ5correspondtomodelsincolumns2,4,and5ofTable7.
0.046
(0.040)
0.063
(0.022)
0.110
(0.040)
352
83
9.77
0.026
(0.040)
0.092
(0.023)
0.123
(0.039)
217
51
9.73
Training
(2)
0.000
(0.034)
0.030
(0.029)
0.006
(0.025)
36
15
3.82
JobSearch
Assistance
(3)
Ͳ0.117
(0.102)
0.037
(0.108)
0.274
(0.148)
46
19
21.17
PrivateSector
Job/Subsidy
(4)
0.039
(0.073)
0.010
(0.065)
Ͳ0.050
(0.089)
32
14
Ͳ2.23
PublicSector
Employment
(5)
0.213
(0.058)
0.042
(0.067)
Ͳ0.019
(0.037)
21
8
13.70
Theat/Sanctions
(6)
a
Fourdummiesfordifferenttypesofprogramsincluded.
IntakeGroup(Base=RegularUIRecipients)
Disadvantaged
0.003
Ͳ0.046
0.200
0.124
ͲͲ
0.157
(0.043)
(0.043)
(0.034)
(0.068)
(0.114)
LongTermUnemployment
0.190
0.306
0.060
0.274
0.143
Ͳ0.239
(0.077)
(0.138)
(0.039)
(0.077)
(0.049)
(0.045)
GenderGroup(Base=Mixed)
Male
0.040
0.060
(omitted)
0.234
Ͳ0.132
Ͳ0.075
(0.043)
(0.054)
(0.147)
(0.062)
(0.063)
Female
0.119
0.126
(omitted)
0.365
Ͳ0.081
Ͳ0.312
(0.050)
(0.063)
(0.113)
(0.061)
(0.099)
AgeGroup(Base=Mixed)
ͲͲ
Youth
Ͳ0.076
Ͳ0.047
0.016
0.100
Ͳ0.155
(0.049)
(0.022)
(0.092)
(0.066)
(0.045)
OlderParticipants
Ͳ0.103
Ͳ0.113
0.029
Ͳ0.244
Ͳ0.134
0.114
(0.041)
(0.053)
(0.041)
(0.119)
(0.081)
(0.006)
ControlsforProgramTypea
Yes
No
No
No
No
No
Notes:standarderrors,clusteredbystudy,inparenthesis.SeenotetoTable5.Dependentvariableinallmodelsisestimatedeffectsize.
Modelsincolumn1arefittoallavailableeffectsizes.Modelsincolumns2Ͳ6areestimatedonsubsetsofeffectsizeestimatesforprogram
typesindicatedincolumnheading.
LongTerm
MediumTerm
Constant
NumberofEstimates
NumberofStudies
MeanEffectSize(×100)
All
ProgramTypes
(1)
Table8:ComparisonofRelativeImpactsforDifferentParticipantGroupsbyTypeofProgram
Ͳ0.110
(0.055)
0.043
(0.062)
Ͳ0.137
(0.053)
Ͳ0.141
(0.084)
0.024
(0.073)
0.237
(0.076)
0.117
(0.055)
0.201
(0.067)
Ͳ0.047
(0.050)
Ͳ0.130
(0.054)
Ͳ0.105
(0.057)
0.064
(0.062)
Ͳ0.149
(0.057)
Ͳ0.093
(0.070)
0.001
(0.074)
0.245
(0.081)
0.103
(0.056)
0.186
(0.067)
Ͳ0.047
(0.048)
Ͳ0.115
(0.054)
Ͳ0.030
(0.064)
Ͳ0.150
(0.065)
0.148
(0.058)
0.212
(0.073)
0.132
(0.068)
0.284
(0.080)
Ͳ0.149
(0.074)
0.038
(0.063)
Ͳ0.199
(0.065)
Ͳ0.250
(0.088)
Ͳ0.100
(0.056)
Ͳ0.202
(0.053)
0.180
(0.051)
0.249
(0.068)
0.262
(0.086)
0.279
(0.068)
Ͳ0.207
(0.062)
Ͳ0.010
(0.054)
Ͳ0.149
(0.053)
Ͳ0.448
(0.117)
Ͳ0.054
(0.127)
Ͳ0.131
(0.054)
0.115
(0.049)
0.180
(0.064)
0.140
(0.078)
0.242
(0.080)
0.051
(0.131)
0.024
(0.065)
Ͳ0.208
(0.056)
Ͳ0.286
(0.136)
Denmark,France,Germany,andU.S.Only
Baseline
+GDPGrowth
+Unemp.Rate
(4)
(5)
(6)
0.095
0.078
0.090
(0.023)
(0.020)
(0.025)
0.128
0.073
0.109
(0.053)
(0.040)
(0.054)
ͲͲ
Ͳ0.070
0.078
(0.019)
(0.024)
CountryDummies
No
Yes
Yes
Yes
Yes
Yes
Notes:standarderrors,clusteredbystudy,inparenthesis.Modelsincolumns1Ͳ3arefitto352programestimatesfrom83studies,withmeanof
dependentvariable=0.098.Modelsincolumns4Ͳ5arefitto200studiesfromDenmark,France,Germany,andtheU.S.from38studies,withmeanof
dependentvariable=0.089.Modelincolumns6isfitto181studiesfromsamefourcountriesfrom34studies,withmeanofdependentvariable=0.093.
Ͳ0.065
(0.039)
PrivateSectorJob/Subsidy
0.066
(0.055)
PublicSectorEmployment
Ͳ0.151
(0.041)
Sanctions/Threat
Ͳ0.024
(0.088)
IntakeGroup(Base=RegularUIRecipients)
Disadvantaged
0.003
(0.043)
LongTermUnemployed
0.190
(0.077)
GenderGroup(Base=Mixed)
Male
0.040
(0.043)
Female
0.119
(0.050)
AgeGroup(Base=Mixed)
Youth
Ͳ0.076
(0.045)
OlderParticipants
Ͳ0.103
(0.041)
ProgramType(Base=Training)
JobSearchAssistance
GDPGrowthRate(%)
(Unemp.Rateincol.6)
LongTerm
MediumTerm
AllAvailableEffectSizeEstimates
Baseline
+CountryEffects
+GDPGrowth
(1)
(2)
(3)
0.063
0.062
0.057
(0.022)
(0.023)
(0.022)
0.110
0.097
0.084
(0.040)
(0.044)
(0.040)
ͲͲ
ͲͲ
Ͳ0.022
(0.013)
Table9:ImpactsofMacroConditionsontheEffectivenessofActiveLaborMarketPolicies
AppendixFigure1:DistributionofProgramEstimatesbyCountry
Argentina
Australia
Austria
Belgium
Brazil
Bulgaria
Canada
China
Colombia
Czech
Denmark
Dominican
Estonia
Finland
France
Germany
Hungary
India
Ireland
Israel
Italy
Jordan
Korea
Latvia
Malawi
Mexico
Netherlands
NZ
Nicaragua
Norway
Panama
Peru
Poland
Portugal
Romania
Russia
Serbia
Slovakia
Slovenia
SouthAfrica
Spain
SriLanka
Sweden
Switzerland
Turkey
UK
US
22
4
25
8
3
1
3
2
4
1
115
8
2
6
42
253
11
2
1
2
4
4
3
19
2
6
5
9
1
25
4
24
12
3
8
8
2
13
8
4
16
6
66
12
8
13
57
0
25
50
75
100
125
150
175
200
NumberofProgramEstimates
225
250
275
NumberofEstimates
Ͳ0.375
0
5
10
15
20
25
30
Ͳ0.275
Ͳ0.175
0.025
0.125
0.225
0.325
0.425
Effectsize:MidpointofInterval(dashedline=0)
Ͳ0.075
SignificantlyPositive
Insignificant
SignificantlyNegative
0.525
AppendixFigure2a:HistogramofShortTermEffectSizeEstimates
0.625
0.725
NumberofEstimates
Ͳ0.375
0
5
10
15
20
25
30
Ͳ0.275
Ͳ0.175
0.025
0.125
0.225
0.325
0.425
Effectsize:MidpointofInterval(dashedline=0)
Ͳ0.075
SignificantlyPositive
Insignificant
SignificantlyNegative
0.525
AppendixFigure2b:HistogramofMediumTermEffectSizeEstimates
0.625
0.725
NumberofEstimates
Ͳ0.375
0
2
4
6
8
10
12
Ͳ0.275
Ͳ0.175
0.025
0.125
0.225
0.325
0.425
Effectsize:MidpointofInterval(dashedline=0)
Ͳ0.075
SignificantlyPositive
Insignificant
SignificantlyNegative
0.525
AppendixFigure2c:HistogramofLongerTermEffectSizeEstimates
0.625
0.725
Fraction
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
EffectsizeRange
0.05/ 0.10/ 0.15/ 0.20/ 0.25/ 0.30/ 0.35/ 0.40/ 0.45/ 0.50+
0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50
FemalesͲSign.Positive
MalesandMixedGenderͲSign.Positive
<Ͳ0.25 Ͳ0.25/ Ͳ0.20/ Ͳ0.15/ Ͳ0.10/ Ͳ0.05/ 0/
Ͳ0.20 Ͳ0.15 Ͳ0.10 Ͳ0.05
0
0.05
FemalesͲInsignificant
FemalesͲSign.Negative
MaleandMixedGenderͲInsignficant
MalesandMixedGenderͲSign.Negative
AppendixFigure3:EffectsizeDistributionsConditionalonSign/SignificanceͲͲ
Femalesvs.OtherParticipantGroups
'(
*
*&
(
**
&)
,'
)+
('
-&)
''
!/(."
!/(."
!/''"
!/*-"
!/(*"
!/."
&%
'
-(
(
((
))
-)
&+
%
(,
+%
(
'%
)+
()
!/-%"
!/.*"
!/*%"
&
0/
)+
(
+)
./
*
)*
,*
,.
0(
)+
/
+/
.+
(
**
./
))
01
))
(
.
00
.
(
./
++
$2+)%
$2,0%
$2*.%
$2)-%
$2)1%
$21%
$2*/%
$2)/%
$2*/%
)"
#
!
AppendixTable2:ComparisonsofEffectSize,OrderedProbit,andProbitModels
MediumTerm
LongTerm
EffectSize
(OLS)
(1)
0.056
(0.021)
0.091
(0.038)
OrderedProbit
Sign/Significance
(2)
0.489
(0.114)
0.969
(0.221)
ProgramType(Omitted=Training)
JobsearchAssist.
Ͳ0.012
0.443
(0.043)
(0.436)
PrivateSubsidy
0.086
0.252
(0.057)
(0.300)
PublicSectorEmp.
Ͳ0.152
Ͳ1.356
(0.044)
(0.287)
Other
0.007
0.550
(0.094)
(0.542)
AgeofProgramGroup(Omitted=Mixed)
Ͳ0.062
Ͳ0.614
Youths(<25)
(0.045)
(0.282)
Older(>=25)
Ͳ0.151
Ͳ0.735
(0.044)
(0.280)
GenderofProgramGroup(Omitted=Mixed)
Malesonly
0.029
Ͳ0.289
(0.049)
(0.273)
Femalesonly
0.107
0.043
(0.052)
(0.251)
CountryGroup(Omitted=Nordic)
Germanic
0.107
1.033
(0.073)
(0.460)
Anglo
Ͳ0.07
1.265
(0.081)
(0.577)
EastEurope
0.04
0.615
(0.072)
(0.358)
RestofEurope
0.019
0.825
(0.060)
(0.469)
LatinAmerica
Ͳ0.01
1.017
(0.129)
(0.826)
RemainingCountries
0.084
1.161
(0.091)
(0.521)
TypeofProgramParticipant(Omitted=RegisteredUnemployed)
0.060
0.428
Disadvantaged
(0.089)
(0.618)
LongͲtermUnemployed
0.217
0.436
(0.076)
(0.311)
OtherControls:
Program>9months
Ͳ0.056
Ͳ0.599
(0.042)
(0.234)
Experiment
Ͳ0.031
Ͳ0.312
(0.049)
(0.330)
SquareRootofSamplesize
Ͳ0.039
0.471
(0.086)
(0.689)
PublishedArticle
Ͳ0.056
Ͳ0.374
(0.043)
(0.252)
CitationsRankIndex
Ͳ0.003
Ͳ0.009
(0.004)
(0.023)
NumberofObservations
RSquared/LogLikelihood
352
0.33
352
Ͳ288
ProbitModels
Sign.Positive Sign.Negative
(3)
(4)
0.429
Ͳ0.719
(0.137)
(0.173)
0.851
Ͳ1.365
(0.218)
(0.437)
0.507
(0.434)
0.733
(0.314)
Ͳ1.239
(0.330)
0.987
(0.492)
Ͳ0.531
(0.528)
0.475
(0.315)
1.349
(0.335)
0.595
(0.628)
Ͳ0.643
(0.322)
Ͳ0.657
(0.272)
1.052
(0.499)
0.845
(0.428)
Ͳ0.591
(0.310)
Ͳ0.196
(0.272)
Ͳ0.101
(0.279)
Ͳ0.476
(0.315)
1.043
(0.498)
1.312
(0.625)
0.644
(0.447)
0.909
(0.560)
1.334
(0.879)
1.138
(0.618)
Ͳ1.351
(0.454)
ͲͲ
0.294
(0.588)
0.481
(0.325)
Ͳ0.990
(1.022)
Ͳ0.512
(0.332)
Ͳ0.526
(0.265)
Ͳ0.677
(0.395)
0.796
(0.851)
Ͳ0.41
(0.277)
0.003
(0.024)
0.563
(0.326)
Ͳ0.950
(0.461)
0.817
(0.719)
0.328
(0.298)
0.057
(0.034)
352
Ͳ190
315
Ͳ92
Ͳ0.579
(0.437)
Ͳ0.984
(0.456)
Ͳ1.075
(1.170)
ͲͲ
Notes:Standarderrors(clusteredbystudy)inparentheses.Dependentvariablesareasfollows:column(1)=estimatedeffect
size;column(2)=classificationofsignandsignficanceoftheprogramestimate;column(3)=indicatorforsignificantlypositive
programestimate;column(4)=indicatorforsignificantlynegativeprogramestimate.Modelincolumn(4)isestimatedon
subsetof315observationsbecauseofperfectpredictabilityofresponsesforsomeobservations.

Download Report