2013 NYC Benchmarking Raises Questions about EPA’s new Multifamily Housing Model

A few  weeks ago NYC released Energy Benchmarking data for something like 15,000 buildings for 2013.  9500 of these buildings are classified as “Multifamily Housing” — the dominant property type for commercial buildings in NYC. While data from Multifamily Housing buildings were released by NYC last year, none included an ENERGY STAR building rating as the EPA had not yet developed a model for this type of building.

But a few months ago the EPA rolled-out its ENERGY STAR building score for Multifamily Housing.  So this latest benchmarking disclosure from NYC includes ENERGY STAR scores for 876 buildings of this type.  (Apparently the vast majority of NYC’s multifamily buildings did not qualify to receive an ENERGY STAR score — probably because the appropriate parameters were not entered into Portfolio Manager.)  Scores span the full range, some being as low as 1 and others as high as 100.  But are these scores meaningful?

Earlier this year I published a paper summarizing my analysis of the science behind 10 of the EPA’s ENERGY STAR models for conventional building types including: Offices, K-12 Schools, Hotels, Supermarkets, Medical Offices, Residence Halls, Worship Facilities, Senior Care Facilities, Retail Stores, and Warehouses.  What I found was that these scores were nothing more than placebos — numbers issued in a voluntary game invented by the EPA to encourage building managers to pursue energy efficient practices.  The problem with all 10 of these models is that the data on which they are based are simply inadequate for characterizing the parameters that determine building energy consumption.  If this were not enough the EPA compounded the problem by making additional mathematical errors in most of its models.  The entire system is built on a “house of cards.”  The EPA ignores this reality and uses these data to generate a score anyway.  But the scores carry no scientific significance.  ENERGY STAR certification plaques are as useful as “pet rocks.”

Most of the above 10 models I analyzed were based on public data obtained from the EIA’s Commercial Building Energy Consumption Survey (CBECS).  Because these data were publicly available these models could be replicated.  One of the models (Senior Care Facilities) was based on voluntary data gathered by a private trade organization — data that were not publicly available. I was able to obtain these data through a Freedom of Information Act (FOIA) request and, once obtained, confirmed that this model was also not based on good science.

Like the Senior Care Facility model, the EPA’s Multifamily Housing ENERGY STAR model is constructed on private data not open to public scrutiny.  These data were gathered by Fannie Mae.  It is my understanding that a public version of these data will become available in January 2015.  Perhaps then I will be able to replicate the EPA’s model and check its veracity.  Based on information the EPA has released regarding the Multifamily ENERGY STAR model I fully expect to find it has no more scientific content than any of the other building models I have investigated.

One of the problems encountered when building an ENERGY STAR score on data that are “volunteered” is that they are necessarily skewed.  Put more simply, there is no reason to believe that the data submitted voluntarily are representative of the larger building stock.  ENERGY STAR scores are supposed to reflect a building’s energy efficiency percentile ranking as compared with similar buildings, nationally.  When properly defined, one expects these scores to be uniformly distributed in the national building stock.  In other words, if you were to calculate ENERGY STAR scores for thousands of Multifamily Housing Buildings across  the nation, you expect 10% of them to be in the top 10% (i.e., scores 91-100), 10% in the lowest 10% (i.e., scores 1-10), and so on.  If this is not the case then clearly the scores do not mean what we are told they mean.

Meanwhile, it is interesting to look at the distribution of ENERGY STAR scores that were issued for the 900-or-so Multifamily Housing facilities in NYC’s 2013 benchmarking data.  A histogram of these scores is shown below.  The dashed line shows the expected result — a uniform distribution of ENERGY STAR scores.  Instead we see that NYC has far more low and high scores than expected, and relatively fewer scores in the mid-range.  24% of NYC buildings have ENERGY STAR scores ranging from 91-100, more than twice the expected number.  And 31% of its buildings have scores 1-10, more than 3X the expected number.  Meanwhile only 12% have scores ranging from 41 to 90.  We expect 50% of the buildings to have scores in this range.

histogram of 2013 MFH NYC ES scores

Of course it is possible that New York City just doesn’t have many “average” Multifamily Housing buildings.  After all, this is a city of extremes — maybe it has lots of bad buildings and lots of great buildings but relatively few just so-so buildings.  Maybe all the “so-so” buildings are found in the “fly-over states.”

I ascribe to the scientific principal known as Occam’s Razor.  This principal basically says that when faced with several competing explanations for the same phenomenon, choose the simplest explanation rather than more complicated ones.  The simplest explanation for the above histogram is that these ENERGY STAR scores do not, in fact, represent national percentile rankings at all.  The EPA did not have a nationally representative sample of Multifamily Housing buildings on which to build its model, and its attempt to compensate for this failed.  Until the EPA provides evidence to the contrary — this is the simplest explanation.

 

LEED Certification: intent, implementation, and results

Last week I had the opportunity to deliver the keynote address at the annual conference of the Ohio Public Facilities Maintenance Association (OPFMA) held in Columbus, OH.  Here is a link to the slides used for my presentation, LEED Certification: intent, implemenation, and results.

The thrust of my presenation was to discuss what we know about primary energy savings reduction in green house gas emission for LEED-certified buildings.  Despite the fact that there are roughly 11,000 U.S. commercial buildings certified before Jan. 1, 2013 under LEED New Construction (NC), Core and Shell (CS), Existing Buildings (EB:OM), and LEED for Schools — all LEED programs that address whole building energy use — we have published data from just 2% of these buildings.  This paltry amount of data is mostly gathered by voluntary submissions by building owners willing to share their energy data.  You can bet that such data are skewed towards the better performing buildings.

And even so, the data available show that, on average, LEED-certified buildings show no significant source energy savings or reduction in GHG emission relative to comparable, non-LEED buildings.  That was the thrust of my presentation.

Note that promoters of LEED certification continue to claim energy savings — but these claims are based on design projections not actual performance measurements.  For instance, promoters of Ohio’s Green schools claim 33% reduction in energy use.  But there has never been a study of energy used by Ohio’s LEED-certified schools to demonstrate this assumed savings.  Such claims of energy savings are based on “faith” not “fact.”

 

Power point Slides for ACEEE Talk posted

A video of the power point presentation with audio for my ACEEE talk on building ENERGY STAR scores is now available on the web.  The audio was recorded during a practice  presentation.  The presentation is accompanied by a 16-page paper that may be downloaded from the ACEEE web site (see previous post).

EPA’s ENERGY STAR building benchmarking scores have little validity

I have been spending this week at the American Council for an Energy Efficient Economy’s (ACEEE) Summer Study on Energy Efficiency in Buildings. Yesterday I presented a paper that summarizes my findings from an 18-mos study of the science behind the EPA’s ENERGY STAR building rating systems.

The title of my paper, “ENERGY STAR building benchmarking scores: good idea, bad science,” speaks for itself.  I have replicated the EPA’s models for 10 of their 11 conventional building types: Residence Hall/Dormitory, Medical Office, Office, Retail Store, Supermarket/Grocery, Hotel, K-12 School, House of Worship, Warehouse, and Senior Care.  I have not yet analyzed the Hospital model — but I have no reason to believe the results will be different. (Data for this model were not available at the time I was investigating other models.  I have since obtained these data through a Freedom of Information Act request but have not yet performed the analysis.)

There are many problems with these models that cause the ENERGY STAR scores they produce to be both imprecise (i.e. have large random uncertainty in either direction) and inaccurate (i.e., wrong due to a errors in the analysis).  The bottom line is that, for each of these models, the ENERGY STAR scores they produce are uncertain by about 35 points! That means there is no statistically significant difference between a score of 50 (the presumed mean for the US commercial building stock) and 75 (an ENERGY STAR certifiable building).  It also means that any claims made for energy savings based on these scores are simply unwarranted.  The results are summarized by the abstract of my paper, reproduced below.

Abstract

The EPA introduced its ENERGY STAR building rating system 15 years ago. In the intervening years it has not defended its methodology in the peer-reviewed literature nor has it granted access to ENERGY STAR data that would allow outsiders to scrutinize its results or claims. Until recently ENERGY STAR benchmarking remained a confidential and voluntary exercise practiced by relatively few.

In the last few years the US Green Building Council has adopted the building ENERGY STAR score for judging energy efficiency in connection with its popular green-building certification programs. Moreover, ten US cities have mandated ENERGY STAR benchmarking for commercial buildings and, in many cases, publicly disclose resulting ENERGY STAR scores. As a result of this new found attention the validity of ENERGY STAR scores and the methodology behind them has elevated relevance.

This paper summarizes the author’s 18-month investigation into the science that underpins ENERGY STAR scores for 10 of the 11 conventional building types. Results are based on information from EPA documents, communications with EPA staff and DOE building scientists, and the author’s extensive regression analysis.

For all models investigated ENERGY STAR scores are found to be uncertain by ±35 points. The oldest models are shown to be built on unreliable data and newer models (revised or introduced since 2007) are shown to contain serious flaws that lead to erroneous results. For one building type the author demonstrates that random numbers produce a building model with statistical significance exceeding those achieved by five of the EPA building models.

In subsequent posts I will elaborate on these various findings.

When will the US Senate conduct hearings on “energy loss” programs?

Yesterday a Senate Committee grilled “Dr. Oz” about the promotion of weight-loss products on his show.  At issue are the unsupported claims made for these products and the false hopes of millions of viewers who are looking for quick ways to lose weight.  I would like to know when the Senate will grill proponents of green buildings in the same way.

Don’t get me wrong — I know that Americans need to lose weight, and there are very clear ways to do that with slow, determined change in behavior.  The same is true for improving building energy efficiency.  There are clear ways to cost-effectively improve buildings so that they use 10-20% less energy without any loss of performance.

But Americans want quick solutions — ways to lose 30 pounds in one month without pain or suffering.  And there is an entire industry out there selling products which promise to achieve these very results.  But there is no scientific evidence to support such claims, and mostly people spend their money on these products and never reap their promised benefits.  The few who do achieve the desired weight loss do it because of their regular exercise and reduction in caloric intake — perhaps coincident with the use of some new product, but having no other connection to it.

America’s energy-guzzling buildings have much in common with its overweight population.  And a government-sponsored industry – not unlike the one promoted by Dr. Oz – has emerged promoting green buildings, zero energy buildings, and high-performance buildings — all promising great energy savings for those who adopt their strategies. The US Green Building Council claims that its LEED-certified buildings are achieving 47% energy savings.  The EPA claims that its ENERGY STAR benchmarking program yields significant energy savings.  The New Buildings Institute promotes Zero Energy Buildings as the ultimate “weight loss program.”  The US Federal government pours millions of dollars into GSA, DOE and EPA programs that prumulgate these ideas.  My own state of Ohio has spent millions on LEED-certified schools without a single scientific study to demonstrate that these buildings actually save energy.  The list of organizations and claims goes on and on.

Yet the above claims are, at best, outrageous exagerations.  The USGBC claim is made for a “cherry picked” subset of its buildings and is based on ENERGY STAR scores which have no scientific credibility [see earlier post].  The EPA claims are similarly based on ENERGY STAR scores and do not stand up to close inspection.  And close inspection of data gathered by NBI shows that, at most, about 10 US commercial buildings in the country have demonstrated net-zero performance.  Amercia is spending millions on these green and high performance buildings efforts with little data to demonstrate efficacy.

Don’t get me wrong — I am a stong advocate of cost-effective energy efficiency and energy conservation.  I am also a strong advocate of exercise and sensible nutrition.

 

 

DC Benchmarking data show modest energy savings for LEED buildings

A few months ago Washington DC released its 2012 energy benchmarking data for private commercial buildings 150,000 sf and larger.  Credible energy and water consumption data for some 400 buildings were released, of which 246 were office buildings.  A recent article — stemming from the web site LEED Exposed — has claimed that these data show LEED buildings use more energy than non-LEED buildings.  Specifically it is claimed that LEED buildings have an average weather normalized source EUI of 205 kBtu/sf whereas non-LEED buildings have an average EUI of 199 kBtu/sf.   No details are provided to support this claim.

My students and I have cross-listed the DC benchmarking data with the USGBC LEED Project directory and identified 94 LEED-certified buildings in the 2012 DC benchmarking dataset — all but one being classified as office buildings.  The unweighted mean weather normalized source EUI for these 94 LEED certified buildings is 202 kBtu/sf.   The unweighted mean weather normalized source EUI for remaining 305 buildings is 198 kBtu/sf.  No doubt this is the basis for the claim that LEED buildings use more energy than non-LEED.  However, the difference is not statistically significant.

Moreover, the non-LEED dataset, in addition to 154 office buildings, contains 64 (unrefrigerated) warehouses and 90 multifamily housing buildings — all of which use significantly less energy than do office buildings.  The comparison of these two average EUI is not useful — just a meaningless sound bite.

It should also be noted that the unweighted mean EUI for a collection of buildings is an inappropriate measure of their total energy consumption.  The appropriate measure of energy consumption is their gross energy intensity — their total source energy divided by the total gross square footage.  This issue has been discussed in several papers [2008 IEPEC; 2009 Energy & Buildings].

Note that an apples-to-apples comparison of energy consumed by one set of buildings to that consumed by another requires that the two sets contain the same kinds of buildings in similar proportions.  When possible this is best accomplished by sticking to one specific building type. Since office buildings are far and away the most common in both datasets it makes sense to make an office-to-office comparison — pun intended. 

93 of the LEED-certified buildings are offices.  But many of these buildings were not certified during the period for which data were collected.  Some were certified during 2012 and others were not certified until 2013 or 2014.  Only 46 of the office buildings were certified before Jan. 1, 2012 and are then expected to demonstrate energy and GHG emissions savings for 2012.

The 2012 gross weather-normalized source energy intensity for the 46 LEED certified office buildings is 191 kBtu/sf.  This is 16% lower than the gross weather-normalized source energy intensity for the 154 non-certified office buildings in the dataset, 229 kBtu/sf.  These modest savings are real and statistically significant, though much lower than the 30-40% savings routinely claimed by the USGBC.

Note that similar savings were not found in 2011 or 2012 NYC energy benchmarking data. Analysis of these data showed that LEED-certified office buildings in NYC used the same amount of primary energy and emitted no less green house gases than did other large NYC office buildings.  So the 2012 results from Washington DC are significantly different.  It should be noted that NYC office buildings certified at the gold level were found to exhibit similar modest energy savings.  Perhaps this is a clue as to why Washington DC LEED buildings show energy savings.  More analysis is required.

For the last few years the USGBC has pointed to ENERGY STAR scores for LEED certified buildings as evidence of their energy efficiency.  While ENERGY STAR scores have two important characteristics — they use source rather than site energy and they are based on actual energy measurements — they simply do not have sound scientific basis.  The science has never been vetted, and my own analysis shows these scores are little more than placebos to encourage energy efficiency.  They certainly do not have any quantitative value.

So to summarize, in 2012 LEED offices in Washington used 16% less source energy than  did other office buildings in DC.  What this means and whether such savings justify the added costs of LEED are open questions.

Another look at the analysis by Pollock and Rosiak

A few months ago I called attention the Washington Examiner article by Richard Pollock and Luke Rosiak.  On first read it provided more evidence confirming a trend I have seen in several data sets — that LEED-certified buildings, on average, are not saving primary energy — in this particular case, as measured by ENERGY STAR scores.

But 24 hours later I pulled my original post.  I am not convinced that this cursory study is sufficiently rigorous to stand up to scrutiny.  There are two key reasons for my skepticism.  The first is that over the last year I have leaned that the EPA’s ENERGY STAR building rating system is built on a “house of cards.”  It may encourage building energy efficiency, but it is not founded on good science and there is little reason to believe that a higher ENERGY SCORE means a more energy efficient building.

The second reason is more complicated.  Comparing the energy use of one group of buildings with another is actually difficult to do correctly.  Several of my papers have concluded that other researchers have gotten it wrong in the past.  It is certainly not an activity to be left to people who begin the study with a stake in the outcome — either those promoting LEED or those “dug in against it.”  And, despite the publicity surrounding Thomas Frank’s study in USA Today, it is an activity best left to building professionals and researchers — not reporters.

In this particular case it appears to me that many of the buildings identified by Pollock and Rosiak as “LEED buildings” are not actually LEED-certified at all — they are mostly LEED registered projects.  Anyone can “register” a LEED project — simply stating intentions and requiring a modest fee.  But only minority of those projects that register actually see the process to completion and become LEED-certified.  As critical as I have been of the USGBC’s past claims about energy savings — I cannot hold the USGBC responsible for energy consumption of buildings that have never completed LEED certification. Maybe the registered LEED buildings in this article will soon complete certification.  If and when that occurs then we should look at their subsequent performance.

Washington DC has just released benchmarking data for all its large commercial buildings.  One of my students is in the process of parsing this list to identify LEED and ENERGY STAR certified buildings.  The goal will then be to compare the performance of LEED certified buildings with other buildings.  Only then can we draw any conclusions.