Background
BIT worked with the AER to conduct an online framed field experiment to test different versions of a proposed benefit change notification. This trial is the third in a series of three that were conducted. 2,099 respondents saw one of three versions of the notification, and were asked about their intended behaviour.
Following trial 2, we learned that some retailers may not be able to express the value of the benefit in terms of a dollar figure. Therefore, we elected to take forward the “lose discount %” arm from trial 2. It did not appear to have statistically significant differences from the “lose discount $” letter, as it preserved the high levels of comprehension and did not appear to vary on the other key outcome measures. Given that we had found limited differences on our outcome measures in preceding trials, we decided to include both the outcome measures for trial 3. The outcome measure from trial 1 (asking whether a person when they would take action, and what they would do) was asked immediately after the letter, while the outcome measure from trial 2 (offering the opportunity to view EME) was asked after the comprehension questions.
We also chose to try only three arms, instead of four, in order to increase the power of our test. The three arms were the “lose discount %” from trial 2, a letter with a “large callout” box just below the headline, and one with a “warning callout” that included the estimated bill for the coming year. (edited)
The key findings included:
- Intention to visit EME did not appear to vary by treatment
- Comprehension did not appear to vary by treatment, and was consistent with the most effective versions from the previous two trials conducted with AER.
Overall, these results suggest that we my have been observing ceiling effects, whereby we had reached maximum effectiveness for these measures. This document is intended as a supporting document to the final report, giving technical details of the analysis underpinning these findings.
Estimation strategy
Statistical model
Before the experiment was run, a number of analyses were pre-specified, reducing our risk of spurious findings. Below, we have presented the results of all pre-specified analyses (Primary and Secondary analysis).
For each outcome, we have presented two estimates of the impact of each element of the letter that was varied:
- A regression using just the characteristics of the letter as a predictor
\[\text{Outcome}_i = \beta_1 \cdot \text{Letter 1}_i + \beta_2 \cdot \text{Letter 2}_i + \beta_3 \cdot \text{Letter 3}_i + \epsilon_i\]
- A regression using the random allocation and a set of covariates to improve precision
\[
\begin{aligned}
\text{Outcome}_i = &\beta_1 \cdot \text{Letter 1}_i + \beta_2 \cdot \text{Letter 2}_i + \beta_3 \cdot \text{Letter 3}_i + \\ &\beta_4 \cdot \text{income}_i + \beta_5 \cdot \text{education}_i + \beta_6 \cdot \text{numeracy_score}_i +\epsilon_i
\end{aligned}
\]
Additionally, if the outcome is binary, we have presented the results from both an standard linear regression and the estimated average marginal effect from a logistic regression. All errors reported are heteroskedasticity-consistent ‘robust’ errors.
For all analyses, we present a regression table with the estimates of interest, with standard errors in parentheses, and a bar chart showing the estimates as 4 different treatment groups.
Covariates
The covariates used in the model are define as follows:
- Income - entered as a numeric variable, giving the average value of the income bracket they reported
- Education - entered as a categorical variable, with four possible values:
- “Did not finish high school”
- “High school graduate”
- “Undergraduate”
- “Post-graduate”
- Numeracy scores - entered as a numeric variable, based on their performance in the numeracy scores. Ranges from 0 to 4.
- Switched provider - categorical variable, which indicates the answer given to “How long have you been with your current energy provider?”, chosen from the options below
- Less than 1 year
- Between 1 and 2 years
- Between 2 - 4 years
- More than 4 years
- Don’t know
- Switched plans - a binary variable, indicating whether they’ve ever switched plans with their current provider
Balance
Individuals were randomised into seeing one of the four versions of the letter. Below are summary statistics by group, showing differences in between these treatment groups by the covariates used in the full model. We did not observe any meaningful differences in the covariates.
covars <- c("education_alt", "num_score", "energy1", "energy2")
results %>%
select(treatment, one_of(covars)) %>%
gather(key, value, -treatment) %>%
mutate(key = case_when(
.$key == "education_alt" ~ "Education",
.$key == "num_score" ~ "Numeracy",
.$key == "energy1" ~ "Switch provider",
.$key == "energy2" ~ "Switch plan"
),
treatment = str_c("Letter ", treatment)) %>%
count(treatment, key, value) %>%
filter(key == "Education") %>%
ggplot(aes(value, n, fill = treatment)) +
geom_col(position = "dodge") +
graph_flip() +
scale_fill_BIT() +
labs(x = "", title = "Education")
results %>%
select(treatment, one_of(covars)) %>%
gather(key, value, -treatment) %>%
mutate(key = case_when(
.$key == "education_alt" ~ "Education",
.$key == "num_score" ~ "Numeracy score",
.$key == "energy1" ~ "Switch provider",
.$key == "energy2" ~ "Switch plan"
),
treatment = str_c("Letter ", treatment)) %>%
count(treatment, key, value) %>%
filter(key == "Numeracy score") %>%
ggplot(aes(fct_rev(value), n, fill = treatment)) +
geom_col(position = "dodge") +
graph_flip() +
scale_fill_BIT() +
labs(x = "", title = "Numeracy score")
results %>%
select(treatment, one_of(covars)) %>%
gather(key, value, -treatment) %>%
mutate(key = case_when(
.$key == "education_alt" ~ "Education",
.$key == "num_score" ~ "Numeracy",
.$key == "energy1" ~ "Switch provider",
.$key == "energy2" ~ "Switch plan"
),
treatment = str_c("Letter ", treatment)) %>%
count(treatment, key, value) %>%
filter(key == "Switch provider") %>%
mutate(value = fct_relevel(value, "Less than 1 year", "Between 1 and 2 years", "Between 2 and 4 years", "More than 4 years")) %>%
ggplot(aes(fct_rev(value), n, fill = treatment)) +
geom_col(position = "dodge") +
graph_flip() +
scale_fill_BIT() +
labs(x = "", title = "'How long have you been with your current provider?'")
results %>%
select(treatment, one_of(covars)) %>%
gather(key, value, -treatment) %>%
mutate(key = case_when(
.$key == "education_alt" ~ "Education",
.$key == "num_score" ~ "Numeracy",
.$key == "energy1" ~ "Switch provider",
.$key == "energy2" ~ "Switch plan"
),
treatment = str_c("Letter ", treatment)) %>%
count(treatment, key, value) %>%
filter(key == "Switch plan") %>%
ggplot(aes(value, n, fill = treatment)) +
geom_col(position = "dodge") +
graph_flip() +
scale_fill_BIT() +
labs(x = "", title = "'Have you ever switched plans with them?'")
results %>%
select(treatment, Income = income_mid) %>%
gather(key, value, -treatment) %>%
mutate(treatment = str_c("Letter ", treatment)) %>%
count(treatment, key, value) %>%
ggplot(aes(value, fct_rev(treatment), height = n, scale = 0.05, fill = treatment)) +
ggridges::geom_ridgeline( colour = "white") +
scale_x_continuous(breaks = pretty_breaks(10), label = dollar) +
scale_fill_BIT() +
labs(x = "", y = "")
Descriptives
results %>%
select(id, starts_with("difficulty_")) %>%
gather(key, val, -id) %>%
lm(val ~ key, data = .) %>%
bar_data() %>%
select(-term, term = old_term) %>%
mutate(term = str_replace(term, "keydifficulty_sq00|Treatment ", "")) %>%
mutate(
term = case_when(.$term == "1" ~ "Using a comparison website",
.$term == "2" ~ "Using Energy Made Easy",
.$term == "3" ~ "Calling my retailer",
.$term == "4" ~ "Doing my own research"),
label_text = estimate %>% sprintf("%.1f", .),
stars = NA_character_
) %>%
ggplot(aes(term, estimate, ymax = ymax, ymin = ymin)) +
bit_bar(-0.3) +
scale_y_continuous(breaks = pretty_breaks(4)) +
hide_legend() +
labs(x = "", y = "")
#
# save_plot("research.png")
#
# results %>%
# select(id, starts_with("difficulty_")) %>%
# gather(key, val, -id) %>%
# lm(val ~ key, data = .) %>%
# bar_data() %>%
# select(-term, term = old_term) %>%
# mutate(estimate = estimate / 100,
# ymax = ymax / 100,
# ymin = ymin / 100,
# term = str_replace(term, "keydifficulty_sq00|Treatment ", ""),
# stars = NA_character_) %>%
# mutate(
# term = case_when(.$term == "1" ~ "Using a comparison website",
# .$term == "2" ~ "Using Energy Made Easy",
# .$term == "3" ~ "Calling my retailer",
# .$term == "4" ~ "Doing my own research")
# ) %>%
# bit_hchart_four() %>%
# hc_axis_score(1) %>%
# hc_title(text = "How easy do you think it would be to get a better deal through the following methods (1 - 10)", style = list(color = BIT[2], fontWeight = "bold", fontSize = 14))
#
#
#
