Issue Brief: Volume 3, Number 1

What Window Shopping the Health Insurance Exchanges in Year Two Revealed about the State of the Consumer Experience

Authors: Daniel Polsky, PhD and Heather Howard

Soon after the launch of, the exchange websites that formed the vanguard of the Affordable Care Act quickly became notorious for numerous bugs, crashes, and painfully slow loading times. Over a year later, the portals have reached a sufficient level of stability and core functionality on the back end. But what about the front end?


  • In the second open enrollment period that just closed, many exchanges still focused on achieving core functionality, especially in light of the troubles of the first year.
  • For coverage starting in 2016 and beyond, exchanges have work to do in optimizing consumer experiences with the websites.
  • The roles of Navigators and others who assist consumers in the plan selection process are especially vital, as the exchanges can still be confusing and difficult for many.
  • The anticipated ending of Navigator funding may provide an opening for brokers to play a role in supporting consumers.

In a tacit acknowledgement that the portals did not offer a level of usability to handle a complex product like health care, federal funding was provided for Navigators during the initial enrollment period. These trained individuals functioned like travel agents or real estate brokers, working with consumers to find the plan that best fit their income and needs (without the bias of a commission).

Funding for the Navigator program expired with the close of the 2015 enrollment period. Without Navigators, consumers will be forced to rely solely on the exchange websites – or on private brokers who step into the void – to find their optimal coverage. But at this point, even after years of development, can the portals offer consumers an organized, helpful shopping experience that enables consumers to make informed and efficient choices for health insurance? This brief examines “window shopping” on the state and federal health insurance exchanges, to assess current practices for structuring how consumers browse for plans and obtain information on plan options and features. While some portals showed signs of promise, most were completely incapable of providing a stand-alone consumer experience.


In order to assess the current state of the various exchanges, we had a team of research assistants go into each one to take screen shots and collect data on the default presentation of plans; the number of screens it takes consumers to get to information about those plans; and the way in which information about plans and providers is organized and presented. Throughout, we used a standardized consumer profile of a 29 year-old male of median national income residing in the largest metropolis county in the state. We did not account for Medicaid eligibility.


Sort and Filter Options

Regardless of product, online shoppers are accustomed to a certain degree of control over their browsing experience. This customization largely involves the way products are organized and displayed in their browser – namely, sorting and filtering products by price, metal level, popularity, recommendations, etc. In the case of the exchanges, we found that different exchanges varied widely in the number of sort/filter options they offered: while Washington State had twelve, Washington, DC had zero. Across the exchanges, the average number of filters was four, with an average of about three sorting options.  Examples of sortable and filterable options included: premium, total cost, maximum out of pocket cost, deductible, metal level, insurance company, and quality rating, among others.

Of course, the extent to which traditional filtering and sorting options actually enhance the shopping experience on the portals is unclear. While many online purchases can be made by efficiently organizing the products available, health care is a complex, highly considered purchase where user needs (and user understanding of the product options) vary dramatically. With this type of product, it’s doubtful that filtering and sorting alone will help users find the best option. 

In fact, these options can be detrimental to the user without proper guidance, as demonstrated by the research of Amanda Starc of the Wharton School of the University of Pennsylvania and Keith Ericson of Boston University. Using Massachusetts as a case study, they found that when users were provided with non-standardized plans sorted by price, an overwhelming 60% relied on a simple rule of thumb for making their selection: choose the plan with the lowest monthly premium.[1] This emphasis on premium cost defeats the entire purpose of the exchanges. The ability to sort features other than price would nudge shoppers to examine other factors while choosing a health plan, resulting in a more constructive shopping experience on the exchanges [see Figure 1].

Personalized Defaults

While the ability to sort and filter is still important (and easily accomplished from a technological standpoint), an ideal exchange would help users narrow down their options by providing well-organized and accurate information. Online retailers offer traditional filtering and sorting options, but also include other features such as personalized recommendations, written reviews, and alternate vendors. Even though structured, crowd-sourced data like ratings and reviews may be difficult to implement in the short-term, portals will be more robust if they offer personalized plan recommendations and enough information for the user to make an informed decision.

A lack of personalization and attention to user-experience was the biggest issue with, a predecessor to Thousands of random plans were simply thrown at the user with no respect to the needs of the individual consumer. In a first-step to guide users towards personalized plans, every exchange requires users to navigate through a number of screens before they can start browsing. The number of screens or pop-ups users navigate through before they start browsing plans ranged from four or fewer in eight of the portals observed to ten or more in DC, Kentucky, and Minnesota [see Figure 2]. While this makes window-shopping more time-consuming, the intention is to aid users by offering personalized plans based on the unique needs of the consumer.

Unfortunately, it was difficult to determine whether these extra steps actually resulted in more personalized options. Some of these pages required users to input information, but others were simply text pages with general information on health insurance. Of all the exchanges, Minnesota’s portal appeared to have the most promise with respect to personalizing the options. Although it required that consumers click through many introductory screens, it offered a preference match feature to aid consumers in selecting between plans. However, it is still unclear how it calculates consumer preferences. By contrast, the exchanges in DC and Kentucky also entailed clicking through multiple screens, but there did not seem to be any relationship between what our research assistants entered and the plan options they were offered.

Plan Information

The portals also came up short in helping consumers understand what they were purchasing. Research has shown that health insurance consumers have only a limited understanding of technical aspects of how health insurance works. In a study by the Penn Center for Health Incentives and Behavioral Economics at the Leonard Davis Institute, only 14% of consumers were able to correctly answer four multiple-choice questions about the most important terms in health care: deductibles, copays, premiums and maximum out of pocket costs.[2]

Figure 1: The Default Presentation of Plans

The exchanges offered only limited assistance in bridging this information gap. Many portals offered some further reading about insurance plan terms and options, and except for Hawaii, Minnesota, Vermont and the Federal exchange, these were provided in separate pop-up windows—likely the best practice as they do not interfere with browsing or clutter the screen with information some users will find unnecessary.[3]

While the information was displayed well, the information provided was inconsistent and unhelpful. Moreover, many portals did not include examples with numbers, further limiting their usefulness to the user. On the DC and Kentucky portals, no further information was offered, and users were directed to nothing more than a phone number and an email address.

Figure 2: The Disparity in the Window Shopping Experience

Beyond basic health insurance literacy, consumers also would benefit from directories that would allow them to search for preferred providers. But as noted in a recent paper produced by the Leonard Davis Institute, only six states (Colorado, Hawaii, Kentucky, Massachusetts, Maryland and Washington) displayed any sort of provider lookup, and only three (Kentucky, Massachusetts and Washington) provided a lookup for participating hospitals. Most portals simply redirected users to the insurance company’s website, but those sites were only fully accessible to those who already had an account with that company. For consumers, this feature was essentially worthless.

Further Technical Issues

There are still numerous user-inter face and user-experience issues. For example, California’s portal failed to save information if the user clicked the back button, resulting in time wasted reentering information. In the Rhode Island portal, 2014 was entered as the default coverage year. Some portals were updated daily, causing large amounts of downtime when they were unavailable. For instance, the Maryland exchange announced its updates prominently and the site was unavailable from 12am-6am EST, but Hawaii and Minnesota went down unpredictably.

Design issues were also prevalent. On the Federal exchange, it was impossible to compare plans on the same page, forcing users to open multiple tabs to get a point-by-point comparison. Idaho presented plans on the same page, but displayed the plans using a grid layout. This meant plans were compared both vertically and horizontally, an experience that was confusing. There were many other design issues that diminished the user experience; for instance, Hawaii had numerous problems with formatting (CSS), and only about half the screen was scrollable on the DC and California portals.


By combining the best practices of each exchange, the consumer experience nationally can improve. Colorado’s portal was among the most user-friendly, featuring an excellent layout, an easy-to-use compare system, and a drug formulary lookup [See Figure 3]. Five portals (California, Connecticut, Kentucky, Maryland, and New York) featured quality ratings, although that approach fell short because the methodology of determining those ratings was unclear. Minnesota made some progress with the “Percent Match to User Preferences” feature, which made window-shopping easier.

Colorado and Connecticut featured a very helpful interactive CGI assistant to aid users while browsing. Vermont’s portal, although it crudely displayed plans on a static PDF (a document which actually changed partway through the enrollment period), regularly pointed users towards a useful subsidy calculator. Most portals had dedicated space for social media icons, but Massachusetts stood out by using Twitter as a Q&A forum to effectively supplement its help features.

Figure 3: Colorado Plan Finder Screenshot: Highlighting Best Practices

Even with these features, our researchers, who were trained to analyze exchanges, said that they would have sought the assistance of a Navigator while shopping on any of the portals. While some portals show promise, there’s a long way to go before they are user-friendly enough to be easily operable without a Navigator. Our analysis suggests that the following features are especially effective at promoting usability:

More personalized defaults:User-inputted information about their status and needs can be used to prioritize the best plans for that particular individual. Minnesota’s system of using “Percent Match to User Preferences” was the most effective example of providing this customization.

More comprehensive information:In order to make an informed choice, users need comprehensive information regarding provider, hospital, and drug formulary directories. Having this information prominently displayed enables users to make quick comparisons.

Plan standardization:Even though portals like Colorado have a compare feature, its usefulness is limited by the complexity of the plans involved. Making the plans easier to compare would limit the number of users who simply choose the cheapest option. As Starc and Ericson note, once plans were standardized in Massachusetts, the number of users choosing the cheapest plan available dropped from 63% to 20%.

These are not radical or technically insurmountable changes to make, and all would go a long way toward making sure consumers make more informed and efficient selections on the health care exchanges in 2016 and the years ahead.

The Wharton Public Policy Initiative thanks Kat McKay and Andrew Klimaszewski for their assistance with the research for this study.

About the Authors

Daniel Polsky, PhD
Executive Director, Leonard Davis Institute of Health Economics; Professor of Medicine, Perelman School of Medicine; Professor of Health Care Management, The Wharton School

Daniel Polsky, PhD, Executive Director of the Leonard Davis Institute of Health Economics, is a Professor of Medicine in the Perelman School of Medicine and the Robert D. Eilers Professor of Health Care Management in the Wharton School of the University of Pennsylvania. He has served on the Congressional Budget Office’s Panel of Health Advisers and he was the Senior Economist on health issues for the President’s Council of Economic Advisers in 2007-08. He received a PhD in Economics from the University of Pennsylvania in May 1996 and a Master of Public Policy from the University of Michigan in 1989. His research areas include access to health care, workforce, and economic evaluation of medical and behavioral health interventions. He is a coauthor of the book “Economic Evaluation in Clinical Trials,” published by Oxford University Press.

Heather Howard
Director, State Health Reform Assistant Network

Heather Howard brings a wealth of experience in both federal and state government to her work at the State Network. She is a lecturer in public affairs at the Woodrow Wilson School and a faculty affiliate at the Center for Health and Wellbeing. She most recently served as the commissioner of New Jersey’s Department of Health and Senior Services, where she was responsible for oversight of public health services, regulation of health care institutions, hospital financing, senior services and health care policy and research. Her prior public policy experience includes work in the U.S. Senate and House of Representatives, the Domestic Policy Council at the White House, and the Health Care Task Force within the Antitrust Division at the U.S. Justice Department. Heather received a Bachelor of Arts from Duke University and a JD from New York University School of Law.


 [1]  Keith Marzilli Ericson and Amanda Starc, “Heuristics and Heterogeneity in Health Insurance Exchanges: Evidence from the Massachusetts Connector” (2012), See also “Optimizing Outcomes on the Health Insurance Exchanges,”

 [2]  George Loewenstein, Joelle Y. Friedman, Barbara McGill, Sarah Ahmad, Suzanne Linck, Stacey Sinkula, John Beshears, James Choi, Jonathan Kolstad, David Laibson, Brigitte Madrian, John List, Kevin G. Volpp, “Consumers’ Misunderstanding of Health Insurance,” Journal of Health Economics 32 (2013), 850-862. See also “Optimizing Outcomes on the Health Insurance Exchanges,”

 [3]  “Window Shopping on and the State-Based Marketplaces: More Consumer Support is Needed,”