By Anthony SandersHotels face increasing pressure to demonstrate tangible returns on their sustainability investments, yet measuring the true impact of these initiatives remains a persistent challenge. Traditional A/B testing often falls short in complex, real-world scenarios, making it difficult to isolate the effect of a single sustainability program from other confounding factors. This is where synthetic control testing offers a powerful, data-driven methodology to accurately quantify the financial impact and sustainability ROI of interventions in the hospitality sector.
Key Takeaways
- Identify a suitable control group by carefully matching pre-intervention trends of key performance indicators (KPIs) in non-treated hotels.
- Use advanced statistical software like R with the
Synthpackage or Python withPySCto construct a weighted average of control units that mirrors the treated hotel. - Quantify the ROI by comparing the actual post-intervention performance of the treated hotel against the synthetic control’s predicted performance, attributing the difference directly to the sustainability initiative.
- Ensure data granularity, spanning at least 12 months pre-intervention, for strong baseline establishment and accurate synthetic control construction.
- Validate the synthetic control model’s fit by examining pre-intervention RMSPE (Root Mean Squared Prediction Error) and conducting placebo tests across the control pool.
| Feature | Traditional A/B Testing | Synthetic Control (PySC) | Synthetic Control (R Synth) |
|---|---|---|---|
| Isolates Single Intervention Effect | ✗ Limited in complex scenarios | ✓ Yes, data-driven | ✓ Yes, data-driven |
| Quantifies Financial ROI | ✗ Challenging | ✓ Directly attributes difference | ✓ Directly attributes difference |
| Requires Control Group Matching | ✓ Basic | ✓ Advanced pre-intervention trends | ✓ Advanced pre-intervention trends |
| Software/Tool Used | Various statistical tools | ✓ Python with PySC | ✓ R with Synth package |
| Minimum Pre-Intervention Data | Varies | ✓ At least 12 months | ✓ At least 12 months |
| Handles Confounding Factors | ✗ Struggles | ✓ Strong methodology | ✓ Strong methodology |
| Requires Granular Data | Beneficial | ✓ Foundational requirement | ✓ Foundational requirement |
1. Define Your Intervention and Key Performance Indicators (KPIs)
Before any analysis begins, clearly articulate the specific sustainability intervention being implemented and the precise metrics you expect it to influence. For a hotel, this might involve upgrading to energy-efficient HVAC systems, launching a water conservation program, or sourcing local, sustainable produce for its restaurants. Each intervention will have distinct, measurable outcomes. For instance, an energy-efficient HVAC upgrade would directly impact energy consumption (kWh) and associated utility costs ($). A water conservation program might target water usage (gallons) and water utility expenses. It is also important to define the timeframe of the intervention, including its start date.
I advise clients to select KPIs that are directly quantifiable and financially relevant. Think beyond just environmental impact. Focus on metrics that hit the balance sheet. This could include reductions in waste disposal costs, increases in guest satisfaction scores (if directly linked to a sustainability initiative, perhaps through a “green choice” program), or even changes in average daily rate (ADR) if the sustainability branding attracts a premium segment. Specificity here prevents scope creep and ensures the resulting ROI calculation is meaningful.
Pro Tip: Granular Data is Your Best Friend
The more granular your data, the better. Daily or weekly data points for energy, water, and waste are far superior to monthly aggregates. This level of detail allows for more accurate baseline establishment and a more sensitive detection of changes post-intervention. I have seen many synthetic control analyses falter due to insufficient data granularity. It is a foundational requirement for strong results.
2. Identify Potential Control Units
The core of synthetic control testing lies in creating a “synthetic” version of your treated hotel (the one undergoing the sustainability intervention) using a combination of other, untreated hotels. These untreated hotels form your donor pool or control group. The ideal control units are similar in characteristics to your treated hotel but have not implemented the specific intervention being studied. Think about hotels in the same market, of a similar size, star rating, and operational profile. For example, if you are analyzing a luxury boutique hotel in downtown Atlanta, your donor pool should consist of other luxury boutique hotels in similar urban environments, not a budget motel in a rural area or a large convention center hotel.
Gathering data for these potential control units is often the most labor-intensive step. This typically involves accessing historical performance data such as occupancy rates, ADR, revenue per available room (RevPAR), energy consumption, water usage, and operational costs. Data sharing agreements or publicly available financial reports (for larger chains) can be sources. According to a 2024 report by Statista, global hotel sustainability initiatives are projected to grow significantly, increasing the pool of hotels implementing various programs, which can make finding truly “untreated” controls more challenging but not impossible.
Common Mistake: Ignoring Pre-Intervention Trends
A frequent error is selecting control units based solely on static characteristics (e.g., “same star rating”) without examining their historical KPI trends. The goal is to find hotels whose pre-intervention performance for the target KPIs closely mirrors that of your treated hotel. A hotel that had significantly different energy consumption patterns before your intervention began will not make a good synthetic counterpart, regardless of how similar it looks on paper.
3. Collect and Prepare Data
Once you have identified your treated unit and a donor pool, the next step involves collecting and cleaning the historical data for all relevant KPIs. You will need data for a substantial period before the intervention started (the pre-intervention period) and for the period after the intervention began (the post-intervention period). I generally recommend at least 12 months of pre-intervention data, and ideally 24 months, to capture seasonal variations and establish strong baselines. Data should be consistent in its aggregation level (e.g., monthly, quarterly). Missing data points need to be addressed. Interpolation or imputation methods can be used, but transparency about these adjustments is vital.
For example, if you are analyzing energy consumption, collect monthly kWh figures for the treated hotel and all potential control hotels for 24 months prior to the HVAC upgrade and for the 12 months following it. In addition to the primary outcome KPIs, collect relevant predictor variables that could influence these KPIs. These are called covariates. For a hotel, this might include average occupancy rate, average daily rate, local weather data (for energy consumption), local event calendars (influencing demand), or even regional economic indicators. These covariates help the synthetic control algorithm find the best possible match.
Screenshot Description: Data Table Example
Imagine a spreadsheet with columns for ‘Hotel ID’, ‘Date’, ‘Energy Consumption (kWh)’, ‘Water Usage (gallons)’, ‘Occupancy Rate (%)’, and ‘ADR ($)’. Each row represents a specific month for a specific hotel. The ‘Hotel ID’ column would differentiate between the treated hotel (e.g., “Hotel A – Treated”) and various control hotels (e.g., “Hotel B – Control”, “Hotel C – Control”).
4. Construct the Synthetic Control
This is the analytical heart of the process. You will use statistical software to create a weighted combination of your control hotels that best matches the pre-intervention KPI trends of your treated hotel. The goal is to find weights for each control hotel such that their weighted average closely tracks the treated hotel’s performance before the sustainability initiative was implemented. This synthetic control then is your counterfactual: what would have happened to the treated hotel had it not undergone the intervention.
My preferred tools for this are R with the Synth package or Python with libraries like PySC. These packages automate the complex optimization process of finding the optimal weights. For instance, in R, you would load your prepared data into a data frame, specify the treated unit, the control units, the outcome variable (e.g., ‘Energy Consumption’), the pre-intervention period, and any covariates. The synth() function then calculates the weights and produces the synthetic control.
Screenshot Description: R Code Snippet for Synth Package
A block of R code showing something like:
library(Synth)
dataprep.out <- dataprep(
foo = hotel_data,
predictors = c("Occupancy Rate", "ADR", "Pre_Intervention_Energy_Avg"),
predictors.op = "mean",
time.predictors.prior = 1990:2000,
dependent = "Energy Consumption (kWh)",
unit.variable = "Hotel_ID",
unit.names.variable = "Hotel_Name",
time.variable = "YearMonth",
treatment.unit = "Hotel A - Treated",
contro.units = c("Hotel B - Control", "Hotel C - Control", "Hotel D - Control"),
time.optimize.ssr = 1990:2000,
time.plot = 1990:2005
)
synth.out <- synth(dataprep.out)
path.plot(synth.out, dataprep.out)
5. Evaluate the Synthetic Control's Fit
After constructing the synthetic control, it is absolutely critical to assess how well it mimics the treated hotel's pre-intervention performance. A visual comparison using a path plot is the first step. This plot overlays the actual treated hotel's KPI trend with that of its synthetic counterpart over both the pre- and post-intervention periods. You want to see a very close alignment during the pre-intervention period. Quantitatively, examine the Root Mean Squared Prediction Error (RMSPE) for the pre-intervention period. A low RMSPE indicates a good fit. If the pre-intervention RMSPE is high, it suggests your synthetic control is not a reliable counterfactual, and you may need to refine your donor pool or covariates.
Another important validation step is conducting placebo tests. This involves applying the synthetic control method to each of the control hotels as if they were the treated unit. If the intervention effect is truly unique to your treated hotel, you should not see similar "effects" when you apply the method to the placebo controls. This helps rule out the possibility that observed changes are due to general market trends rather than your specific intervention. A good synthetic control will show a clear divergence only for the actual treated unit.
Screenshot Description: Path Plot Example
A line graph showing two lines. One line, labeled "Treated Hotel Actual," shows the actual energy consumption of the treated hotel over time. The second line, labeled "Synthetic Control," closely tracks the first line during the pre-intervention period (e.g., up to 2001) and then diverges significantly downward in the post-intervention period (e.g., from 2001 to 2005), indicating a positive effect of the intervention.
6. Quantify the Impact and ROI
With a validated synthetic control, you can now quantify the impact of your sustainability initiative. The difference between the actual performance of the treated hotel and the predicted performance of its synthetic counterpart in the post-intervention period represents the direct causal effect of your program. For example, if your synthetic control predicted the hotel would consume 100,000 kWh per month, but the actual consumption after the HVAC upgrade was 80,000 kWh, then the intervention saved 20,000 kWh per month. Multiply this by the cost per kWh to get the direct financial savings.
To calculate the sustainability ROI, you will need the total cost of the intervention. This includes not just the purchase price of new equipment but also installation costs, training, and any associated operational expenses. The formula is straightforward: ROI = (Total Financial Savings - Total Intervention Cost) / Total Intervention Cost. Express this as a percentage. For example, if an HVAC upgrade cost $50,000 and generated $75,000 in energy savings over two years, the ROI would be (75,000 - 50,000) / 50,000 = 0.5 or 50%. This clear, data-backed ROI is invaluable for justifying future sustainability investments.
Pro Tip: Consider Indirect Benefits
While synthetic control testing excels at quantifying direct financial impacts, do not forget to qualitatively (or if possible, quantitatively through other methods) consider indirect benefits. These might include enhanced brand reputation, increased guest loyalty, improved employee morale, or reduced regulatory compliance risks. While harder to directly fold into the synthetic control ROI calculation, they contribute significantly to the overall value proposition of sustainability.
Synthetic control testing offers hotels a strong, statistically sound method for demonstrating the financial benefits of their sustainability initiatives. This approach moves beyond anecdotal evidence, providing concrete ROI figures that are essential for strategic decision-making and justifying further green investments. For more insights into how data drives business decisions and growth, explore our article on CX data driving growth and reducing churn.
What is the main advantage of synthetic control over traditional A/B testing for sustainability ROI?
Synthetic control testing excels in situations where random assignment (required for true A/B testing) is impossible or impractical, such as a single hotel implementing a unique sustainability program. It creates a strong counterfactual by statistically constructing a control group that perfectly matches the treated unit's pre-intervention trends, allowing for causal inference even without randomization.
How many control units do I need for a reliable synthetic control analysis?
There is no fixed number, but a larger and more diverse donor pool of potential control units generally increases the likelihood of finding a good match for your treated hotel. I recommend having at least 10-15 potential control units to draw from, allowing the algorithm sufficient options to create a well-fitting synthetic counterpart.
What if my synthetic control does not fit the pre-intervention data well?
If the pre-intervention fit is poor (high RMSPE, visible divergence in path plots), it means your synthetic control is not a good counterfactual. You should re-evaluate your donor pool, ensuring the control hotels are truly comparable. Consider adding more relevant covariates to the model or extending the pre-intervention period to capture more trend data. Sometimes, it means the intervention is simply not suitable for this methodology due to lack of comparable control units.
Can synthetic control testing be used for multiple sustainability initiatives simultaneously?
Synthetic control testing is most effective when analyzing the impact of a single, well-defined intervention. If multiple initiatives are launched at the same time, it becomes difficult to isolate the specific impact of each. For concurrent initiatives, you would ideally need separate synthetic controls for each, or a more complex quasi-experimental design.
What kind of data sources are typically used for hotel sustainability KPIs?
Common data sources include utility bills (electricity, water, gas), property management systems (PMS) for occupancy and revenue data, waste management reports, and internal accounting records. For covariates like weather, publicly available meteorological data can be integrated. Consistency in data collection and reporting across all units is paramount.
