Evaluating biodiversity offset policies — an interim outline by Megan Evans on her PhD research

I have been fol­low­ing the work of Megan Evans (who is PhD­ing at at the Aus­tralian National Uni­ver­sity in Can­berra, Aus­tralia) for quite a while, now. Not only in per­sonal com­mu­ni­ca­tion or via her pub­li­ca­tions (by the way, Megan, I should tell you, I admire you for your dili­gence — you’ve been very busy pub­lish­ing those last months) — but also via her blog, enti­tled “Rav­ings and research on envi­ron­men­tal pol­icy”. This has given me some inter­est­ing views on off­sets the “Aussie” way (sorry, if that sounds weird from a non-Australian).

Now, some days ago Megan has launched a four part series of posts out­lin­ing the research jour­ney of her PhD so far. Because that is par­tic­u­larly dense infor­ma­tion and an inter­est­ing read, I’ll share some of that here. She describes the line of her rea­son­ing from the ini­tial inter­est in the out­comes of bio­di­ver­sity off­set pol­icy over the tricky path of how to do pol­icy analy­sis and eval­u­a­tion back to the ques­tion why bio­di­ver­sity off­set poli­cies are hardly ever eval­u­ated despite their expansion.

Post 1: Seek­ing evi­dence to inform the bio­di­ver­sity off­sets pol­icy debate

evidenceWhen I first embarked on my PhD research, my main inter­est was in try­ing to learn about what out­comes had been deliv­ered by bio­di­ver­sity off­set­ting poli­cies. There were a few rea­sons for this. First, I knew that bio­di­ver­sity off­set­ting was becom­ing increas­ingly pop­u­lar. At last esti­mate, 45 coun­tries have some form of off­set­ting or ‘no net loss’ pol­icy, and another 27 coun­tries have poli­cies in devel­op­ment (Mad­sen et al. 2011). I’d also noticed that off­set­ting was being pro­posed as a mech­a­nism for expand­ing the net­work of pro­tected areas in my home state of Queens­land – a pol­icy which I’d expressed some con­cerns about at the time. But prob­a­bly the main moti­va­tion for my research ques­tion came from the often polar­ized dis­course that sur­rounds the use of bio­di­ver­sity offsetting. […]

I think that much of the dis­cus­sion around bio­di­ver­sity off­set­ting, and indeed the use of eco­nomic or mar­ket based instru­ments in bio­di­ver­sity con­ser­va­tion more gen­er­ally, falls into two fairly diver­gent nar­ra­tives. One is pri­mar­ily con­cerned by oppor­tu­ni­ties; whereby off­sets could pro­vide more effi­cient and effec­tive pol­icy out­comes than tra­di­tional reg­u­la­tion, deliver “win-win” out­comes for con­ser­va­tion and the econ­omy, and may deliver social and cul­tural co-benefits. The other is chiefly con­cerned with risks; that off­set cal­cu­la­tions will fail to cap­ture all the val­ues we care about, that off­sets are merely “sym­bolic poli­cies”, and more gen­eral eth­i­cal con­cerns over the com­mod­i­fi­ca­tion of nature.

The prob­lem with hav­ing a pol­icy debate which focuses on such diver­gent and often irrec­on­cil­able views, is that it pro­vides no clear path for­ward in a polit­i­cal envi­ron­ment where off­set pol­icy is, and has been for some time, the pol­icy of choice for demo­c­ra­t­i­cally elected gov­ern­ments. Given this real­ity, how does one prag­mat­i­cally move forward?

This is why I was inter­ested in exam­in­ing the evi­dence. […] As I looked fur­ther into the issues, how­ever, I learned that answer­ing my research ques­tion might not be so fea­si­ble after all (or, indeed, the most inter­est­ing ques­tion to ask).

Read more here.

Post 2: Eval­u­at­ing envi­ron­men­tal poli­cies: what, how and why

Megan Evans_2015_Research areas for policy evaluationA key crit­i­cism of bio­di­ver­sity off­set­ting is that the mea­sures imple­mented on-ground to com­pen­sate for impacts else­where often fail to deliver their intended envi­ron­men­tal out­comes – or in fact, are never actu­ally imple­mented as promised. Clearly, there’s a need to know what, if any, out­comes have been deliv­ered by bio­di­ver­sity off­set poli­cies, and ide­ally to under­stand some of the rea­sons as to why a par­tic­u­lar result is being deliv­ered. This is where pol­icy eval­u­a­tion comes in.

Now, I want to point out right now that there is a pos­i­tively e n o r m o u s lit­er­a­ture on pol­icy eval­u­a­tion, span­ning many dis­ci­plines and schools of thought.[…]

First, I’ll clar­ify that in my research I’m inter­ested in on ex post or ret­ro­spec­tive eval­u­a­tion, where the focus is on deter­min­ing the mer­its of a pol­icy that has already been imple­mented, whereas ex ante or prospec­tive eval­u­a­tion is an analy­sis of the merit a pol­icy prior to implementation.[…]

The pur­pose of an eval­u­a­tion may not sim­ply be to deter­mine what out­comes (intended and unin­tended) have been deliv­ered by a pol­icy, but could also want to exam­ine the pol­icy design, the process of imple­men­ta­tion, or to attempt to estab­lish the impact of the pol­icy rel­a­tive to what would have hap­pened in its absence. […]

A key con­cern is estab­lish­ing the coun­ter­fac­tual – that is, what would have hap­pened in absence of the pol­icy inter­ven­tion? This means try­ing to elim­i­nate all pos­si­ble con­found­ing vari­ables. Nice exam­ples are this paper by Green­stone (2004) which asked whether the US Clean Air Act actu­ally did lead to the mas­sive observed reduc­tion in sul­phur diox­ide emis­sions (spoiler alert: prob­a­bly not), and Bot­trill et al (2011) which found that recov­ery plans didn’t actu­ally lead to threat­ened species recov­ery in Australia.

Read more here.

Post 3: Eval­u­a­tion is impor­tant, so why don’t we do it more often?

I think it’s pretty well estab­lished that eval­u­a­tion is an incred­i­bly impor­tant activ­ity, as it’s really the only way to find out whether or not a pol­icy is meet­ing it’s intended goals. Not only whether the these goals are being met (effec­tive­ness), but whether or not the pol­icy is effi­cient, if any unin­tended out­comes have occurred, and whether there are any oppor­tu­ni­ties for pol­icy improvement.

A num­ber of keys papers have exam­ined the use of eval­u­a­tion in the envi­ron­men­tal and con­ser­va­tion pol­icy spaces in the last 10 years, and many have pro­vided guid­ance on how to con­duct eval­u­a­tions in dif­fer­ent cir­cum­stances. Pos­si­bly the most promi­nent exam­ple is Fer­raro and Pattanayak’s 2006 “Money for Noth­ing” paper, which intro­duced key con­cepts and terms from the eval­u­a­tion lit­er­a­ture to a broader audi­ence, and called for a greater focus on eval­u­at­ing the impact of con­ser­va­tion poli­cies. More recently, Keene and Pullin (2011) argued for an “effec­tive­ness rev­o­lu­tion” in envi­ron­men­tal man­age­ment, and Miteva and oth­ers (2012) called for what they termed “Con­ser­va­tion Eval­u­a­tion 2.0″ – more urgent efforts to eval­u­ate con­ser­va­tion poli­cies in more locations. […]

So, we have the know-how, and the need to know whether the huge invest­ment into con­ser­va­tion glob­ally is achiev­ing results – surely we should be doing eval­u­a­tion all the time, right?

Unfor­tu­nately, this is not really the case. In their review of stud­ies which have eval­u­ated the per­for­mance of con­ser­va­tion poli­cies, Miteva and oth­ers (2012) found that “…cred­i­ble eval­u­a­tions of com­mon con­ser­va­tion instru­ments con­tinue to be rare”. […]

So, given all that, why on earth are we not eval­u­at­ing more often?

1) It’s (method­olog­i­cally) hard

Let’s face it, it’s pretty hard to do an eval­u­a­tion if you don’t have any data avail­able to do such analy­sis. And when we’re work­ing in the envi­ron­men­tal space, col­lect­ing suit­able data and doing an eval­u­a­tion just becomes even harder. Eco­log­i­cal sys­tems are incred­i­bly com­plex and dif­fi­cult to mea­sure – there are long time-lags, non-linear responses, and very large spa­tial scales to con­sider – to name just a few issues.[…]

2) Shouldn’t we be spend­ing money on man­age­ment, rather than mon­i­tor­ing (or evaluating)?

In the face of scarce data, it’s often easy to rec­om­mend that we just need to make a greater effort to col­lect it. In some cases this may be true, but I think it’s often more com­plex than this, and we need to think care­fully about the time and resources required to col­lect data and whether those resources could be bet­ter used for other activ­i­ties (see our recent paper in Con­ser­va­tion Biol­ogy for a dis­cus­sion of this).[…]

3) Many actors, and many dif­fer­ent objectives 

Envi­ron­men­tal pol­icy doesn’t exist in a ratio­nal, value-free vac­uum where the only con­sid­er­a­tions are the envi­ron­men­tal assets we’re try­ing to pro­tect or man­age. It oper­ates within a com­plex socio-political sys­tem, con­tain­ing a range of indi­vid­u­als and organ­i­sa­tions with vary­ing moti­va­tions and objectives.

Organ­i­sa­tions who are respon­si­ble for design­ing and imple­ment­ing poli­cies, whether they be gov­ern­ment or non-government, can be reluc­tant to eval­u­ate how suc­cess­ful those poli­cies were, as it opens them up to crit­i­cism if the pol­icy was not suc­cess­ful. There often needs to be a lot of polit­i­cal will and a strong com­mit­ment to account­abil­ity for eval­u­a­tion to occur.[…]

So, these are some issues which are con­sid­ered as pos­si­ble bar­ri­ers to eval­u­a­tion of envi­ron­men­tal pol­icy in gen­eral. What about bio­di­ver­sity off­set­ting specif­i­cally? Are there any unique chal­lenges to consider?

Read more here.

Post 4: A frame­work to analyse the effec­tive­ness of bio­di­ver­sity off­set policy

domains of biodiversity offset market designI began my PhD by ask­ing the ques­tion, “What envi­ron­men­tal out­comes are being deliv­ered by bio­di­ver­sity off­set policies?”.[…]

In find­ing that answer­ing the “What…?” this ques­tion was actu­ally going to be extremely dif­fi­cult, I asked “Why is there a lack of infor­ma­tion on what out­comes are being deliv­ered by bio­di­ver­sity off­set poli­cies?”. Per­haps there are par­tic­u­lar issues act­ing as bar­ri­ers to eval­u­a­tion. If so, what could these be?

Indeed, if we step back a bit, and con­sider that mon­i­tor­ing and eval­u­a­tion is a key com­po­nent of good pub­lic pol­icy (Dovers and Hussey, 2013), then this could pro­vide a use­ful frame­work to iden­tify a range of issues which ulti­mately influ­ence what envi­ron­men­tal out­comes are deliv­ered by bio­di­ver­sity off­set­ting policy.

If we think about what makes a “good” bio­di­ver­sity off­set pol­icy, there has been a lot of dis­cus­sion in the lit­er­a­ture about what should be the some of the key pol­icy prin­ci­ples (e.g, no-net-loss, like-for-like). This research speaks to how the pol­icy is framed and what the goals could be (item 2 in the fig­ure), but not nec­es­sar­ily how it should be imple­mented, mon­i­tored or evaluated.

There’s also been a lot of research which has been focused on devel­op­ing and refin­ing off­set met­rics, which can most accu­rately mea­sure how much and what kind of bio­di­ver­sity val­ues are to be lost from an impact site, and what would be an ade­quate com­pen­sa­tion at the off­set site. Con­sid­er­ing things like time lags, uncer­tainty and the spe­cific bio­di­ver­sity fea­tures being impacted is really impor­tant (see our recent paper on the EPBC Act off­set met­ric, and a related blog post).

Although the avail­abil­ity of cred­i­ble and sci­en­tif­i­cally robust off­set met­rics is obvi­ously an essen­tial com­po­nent of for­mu­lat­ing bio­di­ver­sity off­set pol­icy, there are a range of issues (which are often non-ecological) which influ­ence the envi­ron­men­tal out­comes likely to arise from off­set pol­icy. We can con­cep­tu­alise these issues as falling into one of three domains – mea­sure­ment, insti­tu­tional and organizational. […]

Read more here.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>