1 Background

BIT worked with the AER to conduct an online framed field experiment to test different versions of a proposed benefit change notification. 1,805 respondents saw one of four versions of the notification, and were asked about their intended behaviour. They were also asked to enter the information into a mock version of the EME website. The key findings included:

  • Intention to visit EME did not appear to vary by treatment
  • However, informing respondents that they were about to “lose [their] discount” may have led to fewer respondents stating that they would do nothing
  • In addition, the “lose discount” treatment saw higher comprehension
  • Those who saw the “lose discount” treatment spent less time looking at the letter

This document is intended as a supporting document to the final report, giving technical details of the analysis underpinning these findings.

2 Intervention

Below are the four versions of the letter that were trialled.

Increase, no chart Increase, chart
Increase, no chart Increase, chart
Lose discount, no chart Lose discount, chart
Lose discount, no chart Lose discount, chart

3 Estimation strategy

3.1 Statistical model

Before the experiment was run, a number of analyses were pre-specified, to ensure that these findings will be robust to future scaling and replication. Below, we have presented the results of all pre-specified analyses.

For each outcome, we have presented two estimates of the impact of each element of the letter that was varied:

  1. A regression using just the characteristics of the letter as a predictor

\[\text{Outcome}_i = \beta_1 \cdot \text{Losing_discount}_i + \beta_2 \cdot \text{Chart}_i + \beta_3 \cdot \text{Letter}_i \cdot \text{Chart}_i + \epsilon_i\]

  1. A regression using the random allocation and a set of covariates to improve precision

\[ \begin{aligned} \text{Outcome}_i = & \beta_1 \cdot \text{Losing_discount}_i + \beta_2 \cdot \text{Chart}_i + \beta_3 \cdot \text{Letter}_i \cdot \text{Chart}_i + \\ &\beta_4 \cdot \text{income}_i + \beta_5 \cdot \text{education}_i + \beta_6 \cdot \text{numeracy_score}_i + \\&\beta_7 \cdot \text{switched_provider}_i + \beta_8 \cdot \text{switched_plan}_i +\epsilon_i \end{aligned} \]

Additionally, if the outcome is binary, we have presented the results from both an standard linear regression and the estimated average marginal effect from a logistic regression. All errors reported are heteroskedasticity-consistent ‘robust’ errors.

For all analyses, we present a regression table with the estimates of interest, with standard errors in parentheses, and a bar chart showing the estimates as 4 different treatment groups. All regression tables will use ‘Bill increasing + No Chart’ as the omitted category, with the estimates of interest giving the difference in the outcome from this category.

3.2 Covariates

The covariates used in the model are define as follows:

  • Income - entered as a numeric variable, giving the average value of the income bracket they reported
  • Education - entered as a categorical variable, with four possible values:
    1. “Did not finish high school”
    2. “High school graduate”
    3. “Undergraduate”
    4. “Post-graduate”
  • Numeracy scores - entered as a numeric variable, based on their performance in the numeracy scores. Ranges from 0 to 4.
  • Switched provider - categorical variable, which indicates the answer given to “How long have you been with your current energy provider?”, chosen from the options below
    • Less than 1 year
    • Between 1 and 2 years
    • Between 2 - 4 years
    • More than 4 years
    • Don’t know
  • Switched plans - a binary variable, indicating whether they’ve ever switched plans with their current provider

4 Balance

Individuals were randomised into seeing one of the four versions of the letter. Below are summary statistics by group, showing differences in between these treatment groups by the covariates used in the full model. We did not observe any meaningful differences in the covariates.

5 Primary analysis

5.1 Did the treatment increase the likelihood that consumers would use the EME website?

Measure: Respondents were first asked what they would do on receipt of the letter – take action immediately, within a week, when they had time, or not take action at all. For those that chose any of the options that indicated they would take action, we then asked what action they would take. The options included visiting EME, visiting a non-EME comparison site, calling the retailer, doing research online (not via comparison sites), or something else.

For this analysis, the outcome variable is 1 if they responded that they would go to EME, and 0 otherwise

OLS Logistic OLS Logistic
(Intercept) 39.0% ***       -0.6%      
(2.3%)          (6.8%)       
Losing discount included 2.1%    2.1% 2.4% 2.3%
(3.3%)    (3.2%)  (3.2%)  (3.2%) 
Chart included 1.1%    1.1% 1.0% 0.9%
(3.3%)    (3.3%)  (3.3%)  (3.2%) 
Extra impact of both elements -2.2%    -2.2% -2.1% -2.0%
(4.6%)    (4.6%)  (4.6%)  (4.6%) 
Controls for demographics No        No      Yes      Yes     
Controls for numeracy No        No      Yes      Yes     
N 1,805        1,805      1,805      1,805     
*** p < 0.001; ** p < 0.01; * p < 0.05.

Based on the trial, the proportion of respondents that said they would take some action and visit EME did not vary between the treatments, and was consistently around 40%. There were no meaningful differences between the letters. The next most popular options were calling the retailer (22%) and doing research online but not through comparison websites (21%).

5.1.1 Did the treatment decrease the likelihood that the consumer would do nothing?

Measure: Respondents were first asked what they would do on receipt of the letter – take action immediately, within a week, when they had time, or not take action at all. For those that chose any of the options that indicated they would take action, we then asked what action they would take. The options included visiting EME, visiting a non-EME comparison site, calling the retailer, doing research online (not via comparison sites), or something else.

For this analysis, the outcome variable is 1 if they responded that they would ‘Do nothing’, and 0 otherwise

OLS Logistic OLS Logistic
(Intercept) 7.6% ***       15.0% **      
(1.3%)          (4.5%)        
Losing discount included -2.7%    -3.2% -2.8%   -3.3%
(1.6%)    (1.9%)  (1.6%)   (1.9%) 
Chart included 1.6%    1.4% 2.2%   2.0%
(1.9%)    (1.6%)  (1.9%)   (1.6%) 
Extra impact of both elements 1.3%    2.0% 1.0%   1.7%
(2.5%)    (2.5%)  (2.4%)   (2.5%) 
Controls for demographics No        No      Yes       Yes     
Controls for numeracy No        No      Yes       Yes     
N 1,805        1,805      1,805       1,805     
*** p < 0.001; ** p < 0.01; * p < 0.05.

There is some weak evidence to suggest that ‘Losing discount + No chart’ led to a smaller proportion of respondents stating that they would take no action in response to the letter, a reduction of approximately 3% of individuals. However this was only marginally statistically significant - not statistically significant at conventional levels. This is somewhat promising, however, as it is consistent with the broader literature, and with insights from the user testing, which suggest that loss aversion is a powerful motivator.

6 Secondary analysis

6.1 Did the treatment increase comprehension of the letter?

Measure: Respondents were asked two questions – firstly, what the letter was saying would happen to their energy bills next year (they would pay more because they were losing their discount, they would pay more because prices were rising generally, or they would pay less). Secondly, they were asked what the letter was asking them to do (go to EME, contact their retailer for information to use EME, contact their provider to get a better deal, use a comparison website, or something else).

For this analysis, they received 1 for each comprehension question correct and combined their scores. That means that they got 0 if they got both questions wrong, 1 if they answered one of two correctly, and 2 if they answered all questions correctly.

OLS OLS
(Intercept) 1.4 *** 0.6 ***
(0.0)    (0.1)   
Losing discount included 0.2 *** 0.2 ***
(0.0)    (0.0)   
Chart included -0.0    -0.0   
(0.0)    (0.0)   
Extra impact of both elements -0.0    -0.0   
(0.1)    (0.1)   
Controls for demographics No       Yes      
Controls for numeracy No       Yes      
N 1,805       1,805      
*** p < 0.001; ** p < 0.01; * p < 0.05.

Perhaps unsurprisingly, when the headline included a clear statement that respondents would lose their discounts, there was a higher average comprehension score. This different was statistically significant at the 1% level. This was driven largely by changes to the responses to the question about what would happen to their energy bill next year, with more respondents answering correctly. However, there also appear to be increases in the proportion of correct answers to the question about what the letter was asking them to do (i.e., use EME to find a better deal).