Keeping Score

A package of software tools is designed to make routine measurement of actual energy savings cost-effective for home performance contractors.

March 01, 2003
March/April 2003
A version of this article appears in the March/April 2003 issue of Home Energy Magazine.
Click here to read more articles about Auditing

More than 76 million residential buildings and nearly 5 million commercial buildings together use one-third of all the energy consumed in the United States, and two-thirds of all the electricity.  The demand that these buildings place on increasingly scarce energy resources will create inevitable growth for weatherization programs and the building performance industry. That’s the good news for those of us in the home performance profession. But as this industry moves toward maturity, it will have to face questions of efficiency and quality control that are common to any industry.

To start to address these questions, we first have to ask ourselves, as building performance professionals, What do we produce when we do a retrofit of a building? We aim to save energy and dollars, but do we really know what we have saved? My own and others’ experiences as building performance contractors have shown us that crews and auditors are anxious for real feedback on their savings performance.  Without it, program managers and crews can’t optimize their ability to produce savings. So how do we measure what we saved at a low enough cost that we can actually routinely afford to measure it? I have been wrestling with these questions sine the early 1990s when I was exposed, as part of an organization’s strategic planning, to the theory of Total Quality Management (TQM).

TQM was originally based on the application of the scientific method to industrial process.  The basic foundations of TQM began with the application of a simple evaluation cycle: Plan, Do, Study, Act. Planning determines your hypothesis, doing implements it on a test basis, studying analyzes the impact, and acting implements your newfound understanding.  TQM has grown far past its early foundations in industrial process to include all aspects of organizations and organizational change (in much the same way as measure-by-measure energy retrofits have now become house-as-a-system retrofits).

How Are We Really Doing?

The study part of the process has been defined as using the statistical evaluation of results to control the variability of the process. Several evaluation methods have been tried in the home performance industry (see Table 1). Unfortunately, even the least expensive per-building evaluation approach has had limited usefulness either because of cost considerations or because of problems accessing data. Still, evaluations are critical to any industrial process, because no industry can improve its process without feedback.

For evaluating a home performance retrofit from a TQM perspective, the absolute value of savings is less important than the comparison of expected savings to actual savings. The absolute value of savings is affected by a number of factors, including the type of houses, the investment available, and the need to deal with health and safety factors. The more experienced a home performance specialist is, the better he or she can take these factors into account when predicting the savings from a retrofit job.  The ability to predict savings with increasing reliability means that they have increasing control over the process of producing savings.

Those Crazy Occupants

“You say you want me to guarantee savings.  The occupants are way too unpredictable for me to ever do that.”  How many times have I heard that?  The variability in the behavior of the occupants in retrofitted buildings is always the factor cited when potential investments in energy feedback and evaluation are being considered.  Those crazy occupants change their behaviors, turn up thermostats, leave windows open, and do all sorts of things in their search for creature comfort.  But we have two things in our favor.  First, using statistical analyses to evaluate the results of our retrofits helps us to separate out randomly performing variables from variables that are, at least in part, consistent.  So a crew that is consistently doing a poor job of air sealing attics will show consistency in error in a way that the random customer behavior change will not.

Second, as part of our process, we are looking to take control of buildings.  Not as a terrorist act, but as a way to improve the environment in the buildings.  As building performance contractors, we make performance measurements and try to take control of the flows of air, heat, and moisture through the building.  It is the uncontrolled flows of air, heat, and moisture that cause all those problems with comfort, health and safety, building durability, and energy efficiency.  So when we take control of these flows, we generally reduce the occupant’s need to try to overcome performance problems.
Controlling Our Process

In a reasonably ideal world, in order to get solid control of our building performance process, we would like to have a cost-effective way to

  • collect energy usage information, before and after retrofit, from sources as diverse as the utilities’ own data and the customers themselves;
  • normalize this energy usage information using weather data, repeatedly over an ongoing period of time;
  • benchmark this energy usage information against information for other similar buildings;
  • compare the weather-normalized energy usage information to an accurate  preretrofit energy model and adjust the model to the energy use;
  • create a proposed postretrofit energy model and adjust that model to the actual results of retrofits (real blower door numbers, for example, not projected ones); and
  • track the results of this comparison of actual to predicted savings based on factors such as the crew, the auditor, building type, and the type of retrofit.

My company, Performance Systems Development, Incorporated (PSD), in conjunction with Taitem Engineering, has been working for the past three years to create tools for home performance contractors that will allow them to do all of the above and more (see “A Home Performance Fantasy”). What we came up with was a set of state-of-the-art software tools—TREAT, OTTER, and BenchMark.

Hardworking Software

TREAT, or Targeted Residential Energy Analysis Tools, was developed as a next-generation energy modeling tool for building performance contractors and weatherization agencies, with support from the New York State Energy Research and Development Authority (NYSERDA). TREAT builds on the tracking and feedback concepts that were embodied in the New York State Weatherization Assistance auditing program, TIPS, that was developed in the late 1980s. 

TREAT is a database that stores information about buildings, building components, weather, fuel bills, and workscopes, and it has an engine for calculating energy use.  Because TREAT is a database, it can do all sorts of database-type functions, like storing an ongoing flow of daily weather information and fuel bills, and provides an easy mechanism for storing and comparing multiple savings scenarios for multiple buildings. 

The data stored in TREAT are fed into an hourly energy calculation tool, SUNREL, formerly called SERI-RES, that was developed and is maintained by the National Renewable Energy Laboratory, in Golden, Colorado.  (PSD and Taitem license SUNREL for use in TREAT.)  TREAT is a single modeling tool that allows users to do what used to take several tools, including:

  • perform room-by-room and zone-by-zone calculations;
  • model unheated spaces, such as attics and crawlspaces, as buffer zones;
  • perform an easy comparison of multiple scenarios;
  • weather-normalize fuel bills and sum multiple meters in a single building;
  • compare weather-normalized fuel bill data to pre- and postenergy models;
  • perform full baseload audits;
  • model backup heat use with primary and secondary fuels;
  • use advanced stack effect calculations;
  • access state-of-the-art distribution system calculations;
  • store and report all types of observations and measurements;
  • perform home energy ratings (planned by presstime); and
  • perform Weatherization Assistance program audits in single-family, multifamily, and mobile homes (DOE pre-approval obtained last November).


TREAT models can be as simple as a single room in a single zone, or as complex as necessary to model a high-rise multifamily building. TREAT is located in the building performance contractor’s computer.  Along with a range of reports, it generates files that can be uploaded into an on-line database, OTTER, the Online Tracking Tool for Energy Retrofits.  OTTER is a state-of-the-art ASP.NET Web secure server-based application.  The data from a TREAT project can be uploaded into OTTER and the supervising entity, such as a utility or a state weatherization program, so that it can see exactly what all their contractors are doing, when, and to whom.  OTTER provides client, workscope, savings tracking, and invoicing.

OTTER allows a wide variety and number of users to share a common database while having access only to their own data.  OTTER is now being integrated with BenchMark, a PSD application that started as an effort to cost-effectively track the energy performance of multifamily buildings in New York City.  The energy use of a building in BenchMark is compared with the energy use of other similar buildings, and the potential for savings is calculated.  BenchMark will be linked to on-line utility bill data in New Hampshire.

Plan, Do, Study, Act

So after all this work creating a process that allows contractors and programs to obtain rapid, cost-effective feedback on savings, how well does that process work?  So far, programs and agencies in New Hampshire, California, and New York have signed up for part or all of the full system. In New Hampshire, the state electric utilities are working together to provide a coordinated program that features TREAT and OTTER as the energy modeling and program management software. Contractors across the state do audits, create workscopes in TREAT, and make presentations to customers.  The utilities look at the uploaded workscopes, approve the payment of contractor incentives, and track the progress of customers.

“Managing teams of contractors in a new program and getting their reporting in a timely manner can be an onerous task,” says Michael J. McQueeney, Public Service of New Hampshire Program Manager. “The online interface of OTTER means that all reporting is in real time.  I can, for example, review an individual job or see the progress of all jobs on a daily or weekly basis.  That sure beats the usual end-of-month rush to get information compiled and digested.”
The utility bill information from participating customers flows automatically from the utility databases into OTTER and BenchMark, and then into a contractor’s copy of TREAT via a file export.  Contractors can enter fuel from other vendors into BenchMark.

In California, the TREAT and OTTER system will be used to track the performance of private contractors as part of a Home Performance with Energy Star market transformation effort. In New York, all multifamily audits in NYSERDA’s Assisted Multifamily Program are now in TREAT, and single-family contractors in the Home Performance with Energy Star program will be able to migrate to TREAT by presstime. Agencies offering regional support for Assisted Home Performance (subsidized services for low- to moderate-income families) will be using TREAT. Some of these regional teams are currently evaluating OTTER as a tool to track the performance of their member agencies. Now that TREAT has been DOE preapproved for use in the Weatherization Assistance program across the country, the NYS Division of Housing and Community Renewal (DHCR) is moving forward with steps to allow the New York State Weatherization program to transition to TREAT during this program year.

Where Does It All Go From Here?

After years of development, data are finally flowing into the entire TREAT–OTTER–BenchMark system.  Now we expect to be generating a whole new round of design features to support the application of  TQM to the home performance business process.  New developments will include the addition of integrated statistical control tools; enhanced feedback to contractors, including customer lead tracking and linkages to Quickbooks; fee-for-service access for nonprogrammatic TREAT users to OTTER and BenchMark; and integration of BenchMark and TREAT with data streams from Web-enabled energy meters.

It is up to us as an industry to create the systems that will bring us up-to-date with the rest of the world to make the production of energy savings a reliable process that has the confidence of consumers and funding sources.  We have wandered around in the dark for too long, with only occasional access to long-delayed feedback on savings.  If we want to create an industry to capture the savings available in our vast numbers of existing buildings, we need to think like an industry and focus on results.  We hope that TREAT, OTTER, and BenchMark will help single-family and multifamily building performance contractors meet that goal.

Gregory Thomas manages a local home performance contracting company, Performance Systems Contracting, and he is president of a national consulting, software, and training company, Performance Systems Development. He built his own blower door in 1980 and has been developing energy software since 1988.

For More Information:

Gregory Thomas
Performance Systems Development, Incorporated
120 Brindley St., Ste. 5
Ithaca, NY 14850
Tel: (607)277-6250 ext. 201
Fax: (607)277-6224

Web sites:


A Home Performance Fantasy

Lee Contractor gets a call from Ms. Smith, who got his name from a previous client of his.  Lee talks to Ms. Smith about the problems that she is having with her home, recording the information on his customer interview form. He also explains to her the process that he uses to diagnose and solve her problems.

As a first response to Ms. Smith’s call, Lee sends Ms. Smith an e-mail that contains a password and a link to go to Lee’s BenchMark fuel data Web site (OTTER and BenchMark are really just two different interfaces to the same database). Using the password, Ms. Smith goes right to her own personal fuel oil bill Web page.  She enters the data from her records and then gets to see a chart of her usage.  She can go back later and add to the data.

Lee gets the fuel release from the Smiths and forwards it to the utility company, allowing BenchMark to automatically access the data for that customer from their BenchMark-connected fuel bill database.  Lee goes to the BenchMark Web site and compares the usage of this building to that of other similar buildings.  He then downloads the fuel data to his TREAT energy-modeling program on his PC.
The Inspection

Lee goes to the Smiths’ home and records building data on his modeling software datasheet.  During the at-home interview, Ms. Smith complains that the bonus room over the garage is too cold in the winter and too warm in the summer.  Lee does a number of performance tests and records the results.  He observes some damage to the building and records this information also.  Lee wants to think about the fix to the building, so he goes back to the office to model the building and do his estimate.

Back at the Office

Using TREAT, Lee models the home’s energy use, creating the room over the garage as a separate space.  Lee compares the model to the actual fuel bills, right on the main screen of TREAT.  When the model is accurate enough, he starts to model the improvements that the customer asked for, plus the ones that he himself thinks are important for energy efficiency, health, durability, and comfort.  After all, Lee is a home performance professional.

Setting up the bonus room as a separate space lets Lee model the improvements to the envelope of the space and then check the load of that zone compared to the distribution system air flow measurements that he made during the audit.  When he runs a package with the improvements to the bonus room, he discovers that after insulating and adding new windows, the room still will not have enough cooling energy delivered to meet the load.  He estimates the cost of improving the air flow and includes that in the package of improvements he has created. 

Lee also prepares an estimate of the cost to fix the moisture problems that he observed.  Since the total package price is higher than the customer had expected when she called, Lee quickly bundles the improvements into several packages, each at a different price point.  When Lee is finished assembling the packages, he prints out an automatically generated professional report that includes the results of the model, a description of the house, the results of performance tests, and any adverse conditions observed.  He also prints out a workscope report that provides detailed specification-level information on the proposed work, based on specifications in the program, with a few modifications to fit this job.  He will show this to the customer and will also pass it on to the installation crews and any subcontractors, if he sells the job. 

Lee also uses TREAT to generate an estimated energy rating for each of the packages and includes that on the report.  Some customers like knowing that when they sell their house, they will have a certificate with a number that buyers can use to compare this house to other houses.  Lee also does work for some of the local builders and uses his same modeling tool to help those builders get the Energy Star label on their homes.  When the job is done, he will generate a final rating from the stored actual as-built package. He goes on-line to OTTER to print the final rating certificate.

Lee does get hired by Ms. Smith—a satisfying, but not surprising, result of his new sales process. Lee has found that he sells more jobs now than he used to by doing an inspection, energy model, and report/proposal.  He gets a higher margin and closes a higher percentage of jobs, so he finds it worth the effort.  This keeps his crews busy year round, with fewer layoffs.  And he likes that he can do things right more often now, without as much cost pressure. 

He also gets more of his customers from referrals these days—a big advantage, since his referral customers have a higher percentage and tend to buy larger, more profitable jobs. BenchMark and TREAT make it easier for him to stay in contact with his previous customers and to remind them of his work. While he is at the BenchMark Web site, he checks to see if the Joneses, whom Lee worked for last year, have responded to his quarterly e-mail and have entered their postretrofit fuel bill data at their Web page. They did, so Lee downloads those data also and loads them into the TREAT model that he developed for the Joneses’ house.  In a minute or two, Lee sees that the Joneses are saving a bit more than expected.  He prints the TREAT savings report to a PDF file and sends the Joneses their report card along with his standard request for referrals.  Lee gets a lot of new customers this way. “The proof is in the numbers,” he always says.

An interesting escape from reality—or is it? Contractors and utilities in New Hampshire are already able to do about 80% of the above and will be close to 100%, within six months.  Weatherization agencies doing Assisted Home Performance work in New York and Home Performance with Energy Star contractors in California will be using part or all of this approach starting this spring.
Discuss this article in the INFRARED THERMOGRAPHY USERS and Tools of the Trade groups on Home Energy Pros!

Add a new article comment!

Enter your comments in the box below:

(Please note that all comments are subject to review prior to posting.)


While we will do our best to monitor all comments and blog posts for accuracy and relevancy, Home Energy is not responsible for content posted by our readers or third parties. Home Energy reserves the right to edit or remove comments or blog posts that do not meet our community guidelines.

Related Articles
SPONSORED CONTENT What is Home Performance? Learn about the largest association dedicated to home performance and weatherization contractors. Learn more! Watch Video