Background
BIT worked with the AER to conduct an online framed field experiment to test different versions of a proposed benefit change notification. This trial is the second in a series of three that were conducted. 2,285 respondents saw one of three versions of the notification, and were asked about their intended behaviour.
Following trial one, we took the “Lose discount” letter for trial 2, on the basis that it had the highest comprehension and may have reduced the proportion doing nothing.
In addition to the original two comprehension questions, we also added three extra questions, to allow for further analysis of comprehension. We measured “narrow” comprehension (comprehension on the two questions that appeared in trial 1) as well as “broad” comprehension (comprehension across all five questions). Finally, we removed the section that asked participants to enter their information into EME – we did not expect to gain any new information, and this section saw significant attrition from respondents.
The key findings included:
- Intention to visit EME did not appear to vary by treatment
- The “lose discount” headlines lead to higher comprehension
This document is intended as a supporting document to the final report, giving technical details of the analysis underpinning these findings.
Intervention
Trial 2 saw three new letters – a “no headline” letter, a letter that expressed the loss aversion as a percentage, and a letter that added a social norm to the “lose discount” letter.
Below are the four versions of the letter that were trialled.
| No headline |
Loss headline ($) |
| Loss headline (%) |
Loss + Social norm |
Estimation strategy
Statistical model
Before the experiment was run, a number of analyses were pre-specified, to ensure that these findings will be robust to future scaling and replication. Below, we have presented the results of all pre-specified analyses.
For each outcome, we have presented two estimates of the impact of each element of the letter that was varied:
- A regression using just the characteristics of the letter as a predictor
\[\text{Outcome}_i = \beta_1 \cdot \text{Letter 1}_i + \beta_2 \cdot \text{Letter 2}_i + \beta_3 \cdot \text{Letter 3}_i + \beta_4 \text{Letter 4}_i + \epsilon_i\]
- A regression using the random allocation and a set of covariates to improve precision
\[
\begin{aligned}
\text{Outcome}_i = &\beta_1 \cdot \text{Letter 1}_i + \beta_2 \cdot \text{Letter 2}_i + \beta_3 \cdot \text{Letter 3}_i + \beta_4 \text{Letter 4}_i + \\ &\beta_4 \cdot \text{income}_i + \beta_5 \cdot \text{education}_i + \beta_6 \cdot \text{numeracy_score}_i + \\&\beta_7 \cdot \text{switched_provider}_i + \beta_8 \cdot \text{switched_plan}_i +\epsilon_i
\end{aligned}
\]
Additionally, if the outcome is binary, we have presented the results from both an standard linear regression and the estimated average marginal effect from a logistic regression. All errors reported are heteroskedasticity-consistent ‘robust’ errors.
For all analyses, we present a regression table with the estimates of interest, with standard errors in parentheses, and a bar chart showing the estimates as 4 different treatment groups.
Covariates
The covariates used in the model are define as follows:
- Income - entered as a numeric variable, giving the average value of the income bracket they reported
- Education - entered as a categorical variable, with four possible values:
- “Did not finish high school”
- “High school graduate”
- “Undergraduate”
- “Post-graduate”
- Numeracy scores - entered as a numeric variable, based on their performance in the numeracy scores. Ranges from 0 to 4.
- Switched provider - categorical variable, which indicates the answer given to “How long have you been with your current energy provider?”, chosen from the options below
- Less than 1 year
- Between 1 and 2 years
- Between 2 - 4 years
- More than 4 years
- Don’t know
- Switched plans - a binary variable, indicating whether they’ve ever switched plans with their current provider
Balance
Individuals were randomised into seeing one of the four versions of the letter. Below are summary statistics by group, showing differences in between these treatment groups by the covariates used in the full model. We did not observe any meaningful differences in the covariates.
covars <- c("education_alt", "num_score", "energy1", "energy2")
results %>%
select(treatment, one_of(covars)) %>%
gather(key, value, -treatment) %>%
mutate(key = case_when(
.$key == "education_alt" ~ "Education",
.$key == "num_score" ~ "Numeracy",
.$key == "energy1" ~ "Switch provider",
.$key == "energy2" ~ "Switch plan"
),
treatment = str_c("Letter ", treatment)) %>%
count(treatment, key, value) %>%
filter(key == "Education") %>%
ggplot(aes(value, n, fill = treatment)) +
geom_col(position = "dodge") +
graph_flip() +
scale_fill_BIT() +
labs(x = "", title = "Education")
results %>%
select(treatment, one_of(covars)) %>%
gather(key, value, -treatment) %>%
mutate(key = case_when(
.$key == "education_alt" ~ "Education",
.$key == "num_score" ~ "Numeracy score",
.$key == "energy1" ~ "Switch provider",
.$key == "energy2" ~ "Switch plan"
),
treatment = str_c("Letter ", treatment)) %>%
count(treatment, key, value) %>%
filter(key == "Numeracy score") %>%
ggplot(aes(fct_rev(value), n, fill = treatment)) +
geom_col(position = "dodge") +
graph_flip() +
scale_fill_BIT() +
labs(x = "", title = "Numeracy score")
results %>%
select(treatment, one_of(covars)) %>%
gather(key, value, -treatment) %>%
mutate(key = case_when(
.$key == "education_alt" ~ "Education",
.$key == "num_score" ~ "Numeracy",
.$key == "energy1" ~ "Switch provider",
.$key == "energy2" ~ "Switch plan"
),
treatment = str_c("Letter ", treatment)) %>%
count(treatment, key, value) %>%
filter(key == "Switch provider") %>%
mutate(value = fct_relevel(value, "Less than 1 year", "Between 1 and 2 years", "Between 2 and 4 years", "More than 4 years")) %>%
ggplot(aes(fct_rev(value), n, fill = treatment)) +
geom_col(position = "dodge") +
graph_flip() +
scale_fill_BIT() +
labs(x = "", title = "'How long have you been with your current provider?'")
results %>%
select(treatment, one_of(covars)) %>%
gather(key, value, -treatment) %>%
mutate(key = case_when(
.$key == "education_alt" ~ "Education",
.$key == "num_score" ~ "Numeracy",
.$key == "energy1" ~ "Switch provider",
.$key == "energy2" ~ "Switch plan"
),
treatment = str_c("Letter ", treatment)) %>%
count(treatment, key, value) %>%
filter(key == "Switch plan") %>%
ggplot(aes(value, n, fill = treatment)) +
geom_col(position = "dodge") +
graph_flip() +
scale_fill_BIT() +
labs(x = "", title = "'Have you ever switched plans with them?'")
results %>%
select(treatment, Income = income_mid) %>%
gather(key, value, -treatment) %>%
mutate(treatment = str_c("Letter ", treatment)) %>%
count(treatment, key, value) %>%
ggplot(aes(value, fct_rev(treatment), height = n, scale = 0.05, fill = treatment)) +
ggridges::geom_ridgeline( colour = "white") +
scale_x_continuous(breaks = pretty_breaks(10), label = dollar) +
scale_fill_BIT() +
labs(x = "", y = "")
Descriptives
results %>%
select(id, starts_with("difficulty_")) %>%
gather(key, val, -id) %>%
lm(val ~ key, data = .) %>%
bar_data() %>%
select(-term, term = old_term) %>%
mutate(term = str_replace(term, "keydifficulty_sq00|Treatment ", "")) %>%
mutate(
term = case_when(.$term == "1" ~ "Using a comparison website",
.$term == "2" ~ "Using Energy Made Easy",
.$term == "3" ~ "Calling my retailer",
.$term == "4" ~ "Doing my own research"),
label_text = estimate %>% sprintf("%.1f", .),
stars = NA_character_
) %>%
ggplot(aes(term, estimate, ymax = ymax, ymin = ymin)) +
bit_bar(-0.3) +
scale_y_continuous(breaks = pretty_breaks(4)) +
hide_legend() +
labs(x = "", y = "")
#
# save_plot("research.png")
#
# results %>%
# select(id, starts_with("difficulty_")) %>%
# gather(key, val, -id) %>%
# lm(val ~ key, data = .) %>%
# bar_data() %>%
# select(-term, term = old_term) %>%
# mutate(estimate = estimate / 100,
# ymax = ymax / 100,
# ymin = ymin / 100,
# term = str_replace(term, "keydifficulty_sq00|Treatment ", ""),
# stars = NA_character_) %>%
# mutate(
# term = case_when(.$term == "1" ~ "Using a comparison website",
# .$term == "2" ~ "Using Energy Made Easy",
# .$term == "3" ~ "Calling my retailer",
# .$term == "4" ~ "Doing my own research")
# ) %>%
# bit_hchart() %>%
# hc_axis_score(1) %>%
# hc_title(text = "How easy do you think it would be to get a better deal through the following methods (1 - 10)", style = list(color = BIT[2], fontWeight = "bold", fontSize = 14))
#
