After transition from phone to web-only methods, surveys stay true to trends

EXPERT Q&A

Following a four-month transition from phone-based methods, the University of Michigan Surveys of Consumers has entered its third month of exclusively using web interviews as of October.

With these advancements in information-gathering methodology, the survey remains robust, trustworthy and dedicated to amplifying consumer voices, says U-M economist Joanne Hsu, director of the surveys.

She highlights some of the key points of this transition.

What changes in methodology were implemented this year?

Since the University of Michigan began collecting Surveys of Consumers in 1946, its methodology has regularly evolved to reflect advancements in data collection science and shifts in communication methods. In 2024, the Surveys of Consumers transitioned from cellular phone interviews with random digit dialing to web-based surveys using address-based sampling.

This phase-in began in April, and all interviews are conducted online as of July. This change continues the University of Michigan’s longstanding commitment to producing monthly, nationally representative data on consumer attitudes toward key economic indicators.

The transition facilitated a substantial increase in the sample size from approximately 600 interviews per month in recent years to 900 to 1,000 interviews per month under web interviewing.

Why did the survey switch from phone to web?

The shift from phone to web surveys happened because it’s getting harder to keep long-term data quality with phone interviews. More importantly, the web offers new opportunities for future measurement improvements, like speed, flexibility, precision, respondent experiences and population coverage.

Over time, phone surveys have become more challenging, making it difficult to run detailed and timely surveys. The Surveys of Consumers research team needed to make strategic adjustments to keep the data accurate and representative for the long term.

Even without budget limitations, there are few ways to tackle future challenges with phone surveys. Reaching a randomly selected household by phone now requires considerably more time and effort, so the team started experimenting with web surveys in 2010 and began regular monthly web data collection in 2017 on an experimental basis.

Do phone and web interviews capture the same information?

The methodological transition was informed by over seven years of running parallel data collection via the web and official phone data collection. Survey questions and statistical methods were developed specifically to ensure continuity between phone and web. Our research shows that the Index of Consumer Sentiment exhibits a remarkably strong 0.97-time series correlation between the two data collection methods.

This means that both interview methods show the same trends and turning points in the data. All evidence indicates that the index captures the same underlying information from consumers. Moreover, phone- and web-based survey data exhibit similar trends and relative levels regarding demographic groups.

Do survey responses to web and phone interviews differ?

Some survey questions yield slightly different responses online versus by phone, which is expected based on previous research and detailed in our announcement. While these differences vary by question, they are minor and don’t impact key topics like economic expectations and inflation expectations. Month-to-month trends are unaffected.

Despite the shift to web-based surveys, the core narrative remains the same: consumer sentiment has gradually improved since June 2022, although it’s still well below pre-pandemic levels. Inflation expectations, which surged in mid-2022, have since eased.

For overall trends in economic sentiment, the current data remains reliable as reported. Researchers interested in mode-specific differences can adjust data as needed. Our phase-in of web interviews was specifically designed with these researchers in mind. Adjusting is complex, as there’s no one-size-fits-all method; each approach has trade-offs that may affect results.

Because these minor differences don’t alter the main story, we opted to report data as collected, which is consistent with our practice of not applying seasonal adjustments. Researchers who need adjustments can use them as relevant, though many users find the unadjusted data sufficient.

Does the change in methodology affect how we should interpret sentiment numbers today?

The transition to web-based surveys has confirmed the same narrative of consumer sentiment, which has risen over the past two years since hitting an all-time low in June 2022. The latest data show three consecutive months of improvement, though sentiment still lags behind pre-pandemic levels. The Surveys of Consumers include around 40 additional questions, which provide essential context for understanding consumer attitudes and confirm that sentiment trends remain interpretable across the methodology change. Critically, consumer views remain subdued because of continued frustration over high prices, even as they know that inflation has slowed.

The slight decline in sentiment from April to May primarily reflected an actual downturn in consumer views, only mildly amplified by the introduction of web-based interviews in April. Although sentiment dipped modestly for two months, these movements were insignificant, and sentiment has improved steadily since the full web rollout.

Has the quality of open-ended responses declined?

The qualitative data collected by the Surveys of Consumers has always been valuable, offering important context for the headline sentiment measures. High-quality, open-ended responses have continued with web surveys, maintaining the reliability of this data over time.

The open-ended answers gathered online are generally similar to those from phone surveys. Common reasons people mention for unfavorable personal finances, economic news, conditions for buying homes and conditions for buying vehicles all show very high consistency over time (with correlations higher than 0.96) and at levels similar for phone and web. Some less commonly used categories have slightly lower consistency but tend to be more variable each month, no matter the survey method.

How do the Surveys of Consumers ensure transparency and flexibility in transitioning from phone to web-based data collection, particularly for adjustment factors and diverse user needs?

In line with our commitment to transparency, our website provides documentation on how the transition affects survey data, time series charts and correlations based on seven years of parallel data collection via the web. We also offer microdata, allowing users to analyze phone and web data from the transition period separately.

After thorough evaluation, we decided not to apply adjustment factors, as a single model would not suit the diverse needs of our data users or account for variable-specific differences. The methodological transition was explicitly designed to offer transparency and flexibility to meet the varied goals of data users relying on the Surveys of Consumers. We will continue to assess and share our methods to serve our users and the public best.