What went wrong with America’s $44 million vaccine data system?Cat Fergusonon January 30, 2021 at 11:00 am

The first time Mary Ann Price logged into her employer’s system to schedule a vaccine, she found an appointment three days later at a nearby Walgreens pharmacy. She woke up the next day to an email saying it had been canceled.

So she logged in again and found an opening that afternoon at the local surgical hospital.

“When I showed up, they said they wouldn’t honor it—they were only doing their own staff,” Price says. But when she tried a third time to make an appointment, she was blocked from doing so: according to the system, she was already in the middle of getting a vaccine.

Price is 70 and works for the West Virginia state senate, which has deemed her an essential worker. Her state has been lauded for its rollout of vaccinations—so far 10% of its citizens have been given at least one shot. 

Her frustration is echoed by millions of Americans who have struggled to get vaccines through various chaotic systems. But unlike others in some states, she wasn’t encountering these problems with a third-party consumer service like Eventbrite, or even through an antiquated government system. She was on the US Centers for Disease Control and Prevention’s brand-new, $44 million website called VAMS—the Vaccine Administration Management System, built by the consulting firm Deloitte.

Unless you’re in one of the few states using it, you may not have heard of VAMS. But it was supposed to be a one-stop shop where employers, state officials, clinics, and individuals could manage scheduling, inventory, and reporting for covid shots—and free for anyone to use. 

Instead, “VAMS has become a cuss word,” Marshall Taylor, head of South Carolina’s health department, told state lawmakers in January. He went on to describe how the system has badly hurt their immunization efforts so far. Faced with a string of problems and bugs, several states, including South Carolina, are choosing to hack together their own solutions, or pay for private systems instead.

Clinic workers in Connecticut, Virginia, and other states say the system is notorious for randomly canceled appointments, unreliable registration, and problems that lock staff out of the dashboard they’re supposed to use to log records. The CDC acknowledges there are multiple flaws it’s working to fix, although it attributes some of the problems to user error.

As for Price, she waited at the hospital for 45 minutes before getting an administrator to actually cancel the appointment they wouldn’t honor. “She said they keep trying to get taken off the list [of vaccine sites], but VAMS puts them back on,” she says. 

“We needed a way to run these clinics”

The chaos of the vaccine rollout in the US has been well documented: states receiving half their expected doses; clinics canceling first shots because of unreliable supplies; people endlessly hitting “Refresh” on sign-up websites or lining up outside clinics without an appointment, hoping for a spare shot. 

The CDC saw this coming.

“VAMS was intended to fill a need that states and jurisdictions were not equipped to do themselves,” says Noam Arzt, the president of HLN Consulting, which helps build health information systems. 

“It was clear we needed a way to run these clinics, to schedule people to go, and try to make sure they come back for their second dose,” he says.

So early in the pandemic, the CDC outlined the need for a system that could handle a mass vaccination campaign, once shots were approved. It wanted to streamline the whole thing: sign-ups, scheduling, inventory tracking, and immunization reporting. 

In May, it gave the task to consulting company Deloitte, a huge federal contractor, with a $16 million no-bid contract to manage “covid-19 vaccine distribution and administration tracking.” In December, Deloitte snagged another $28 million for the project, again with no competition. The contract specifies that the award could go as high as $32 million, leaving taxpayers with a bill between $44 and $48 million.

Why was Deloitte awarded the project on a no-bid basis? The contracts claim the company was the only “responsible source” to build the tool.

In reality, many states are choosing to pay other vendors rather than using VAMS for free. Others are doing essentially nothing, leaving the planning up to county health departments. That’s how you get a situation like the one in Florida, where counties are desperately scheduling clinic visits on the website Eventbrite, more commonly used for concerts and conferences. 

“The public health agencies are under significant duress,” says Arzt. “These folks are working night and day, with the same constraints and problems—their kids are at home, people are getting sick. It’s very difficult to manage anything, let alone a campaign like this.”

MIT Technology Review has yet to get a complete list of states using VAMS, despite multiple attempts. According to a CDC press officer, “The VAMS system doesn’t make this data publicly available yet.” 

According to a Deloitte spokesperson, however, “10 jurisdictions, three federal agencies, and one hospital system” across the whole of America are current users, and “more than 1,150,000 vaccines have been administered through VAMS.”

“It really requires something flexible”

Several states that have been using VAMS are backing away. Virginia, where many individual vaccination sites had already chosen to adopt alternatives, is moving from VAMS to a commercial system, PrepMod. After participating in a trial of VAMS, California also picked PrepMod but clinics there have blamed that system for delays in their vaccine reporting.

To better understand the challenges facing people building these tools, I called Derrick Stone, a software development manager at UVA Health in Virginia. While the public health department is using VAMS, his organization is using VaxTrax, a tool built in house by Stone’s team two years ago to track employees’ mandatory flu vaccines. 

“A lot of what these systems do is trivial, but the task was undefined,” he says. “You had to try giving some shots, learn from it, and then figure out how to revise it to be more efficient. It really requires something flexible.”

Indeed, a lack of flexibility has become a block for many clinics trying to use the CDC system. This has led to confusion, and difficulty in keeping patients properly informed.

“A good system is easier to use than it is not to use. If people are writing this on paper, there’s something wrong,”

“When people signed up with VAMS, I couldn’t send specific instructions about the drive-through: Eat before you come, use the bathroom,” says Lorrin Pang, the district health officer in Maui, Hawaii, who’s been running a drive-in clinic at the island’s college. “Elderly guys who didn’t get the message—a lot of them had to get out of their cars and be helped to the bathroom.”

Pang says he spent three weeks trying to sign into VAMS, but he constantly ended up in the dashboard for patients instead of clinic administrators. In the meantime, his staff was vaccinating hundreds of people a day and keeping track of their information on paper forms. The college set up a bank of volunteers to sit in a room and copy all the information into VAMS. 

Eventually, the local hospital helped him get signed into the system. The clinic used it for three days. On the last day, 20 new volunteers came in ready to work. But they’d already signed into VAMS to get their mandatory shots, and there was no way to switch them from patient accounts to staff ones. 

The next day, they went back to paper. 

“A good system is easier to use than it is not to use. If people are writing this on paper, there’s something wrong,” says Stone. “How are you going to do 100 million shots in 100 days and have someone enter it all in by hand?”

“There is zero way it’ll happen without help”

“VAMS is fussy. There’s days when VAMS works, and days when VAMS doesn’t work,” says Courtney Rowe, a pediatric urologist at Connecticut Children’s Medical Center, who has been volunteering to monitor people for reactions after their shots. She takes it as an opportunity to help people get set up for their second appointments. “I basically function as tech support,” she says.

Online sign-ups are especially challenging for older people, perhaps the worst group to beta-test a new system. Many seniors probably lost their internet access when libraries and senior centers closed; only 59% have broadband connections at home, according to a 2019 Pew survey. While many states offer phone lines for making appointments, people around the country have complained about endless waits.

“There’s days when VAMS works, and days when VAMS doesn’t work.”

“It won’t work on Internet Explorer; it only works in Chrome. The ‘Next’ button is all the way down and to the right, so if you’re on a cell phone, you literally can’t see it,” says Rowe. “In the first round, people using VAMS mostly had advanced degrees. If you’re 75 and someone asks you to log into VAMS, there is zero way it’ll happen without help.”

After I spoke with Rowe, Connecticut opened up vaccinations to anyone over 70. Her prediction came true immediately. On the first day of a new vaccination clinic in Vernon, Connecticut, 204 vaccines were ready but only 52 seniors had made appointments in VAMS.

“Our residents, and those from around the state that we’re serving at this clinic, are frustrated, angry, and confused by the ineffectiveness of this registration system,” town administrator Michael Purcaro said at a press conference. 

Elderly people aren’t the only ones who will struggle if vaccination requires online sign-up. Language barriers will become a significant problem, especially for non-native English speakers doing high-risk essential work. People in rural or poor urban areas often have limited access to the internet in the first place, a problem disproportionately affecting the same Black and Latino communities that have suffered the worst traumas of the pandemic.

“There are some real equity concerns,” says Stone. “What happens when you go to a city and 20% of the population can’t get the notices?”

So what went wrong? In an email, a CDC spokesperson defended the system and said that appointments are not randomly canceled, despite what many clinicians have claimed: the problem, she said, was user error. She also outlined several fixes that have been made in response to feedback. VAMS now includes warnings when administrators do something that might change patient appointments, for example. 

The CDC also recommends that the authorities using VAMS help those who might have trouble with the system, by staffing call center hot lines and using “third-party clinics that do not require recipients to register in advance or have an email.”

To some watchdogs, VAMS is the latest example of a broken system for building government technology. Deloitte has a long history of making malfunctioning things for state and federal governments: most recently, it was in the news for charging states hundreds of millions of dollars for unemployment websites that did not work.

In response to questions about the flaws with VAMS, a Deloitte spokesperson sent a statement that the company was proud to support the vaccine campaign and “help end the covid-19 pandemic so that our families and communities can recover and thrive.” He did not address specific questions.

Deloitte may be representative of a broken system, but it’s certainly not alone. CGI Federal, for instance, has landed over $5.6 billion in federal IT contracts since getting fired after its disastrous development of the Healthcare.gov website.

“Nobody wants to hear about it, because it sounds really complicated and boring, but the more you unpeel the onion of why all government systems suck, the more you realize it’s the procurement process,” says Hana Schank, the director of strategy for public-interest technology at the think tank New America. 

The explanation for how Deloitte could be the only approved source for a product like VAMS, despite having no direct experience in the field, comes down to onerous federal contracting requirements, Schank says. They often require a company to have a long history of federal contracts, which blocks smaller or newer companies that might be a better fit for the task. 

Those inefficiencies are magnified in the health sector. America’s heavily privatized medical system was held together by duct tape and bubble gum long before the biggest public health crisis of our lifetimes.

“The health-care software industry is enormous, and it exists largely because it’s privatized, it’s not standardized,” says Stone. “There are a lot of free-market inefficiencies. And the country doesn’t have a public health infrastructure, so there isn’t any real drive to fix it.”

“You think about the industries that have been transformed by technology—someone said, How do we get a pizza to your house faster? That’s a competitive advantage,” he says. “That has not happened in American health care.”

To Rowe, the doctor at Connecticut Children’s, it’s frustrating to see so much innovation in making vaccines, and so little in actually getting them to people. “How much money was put into the science of making the vaccine? How much money is being put into the distribution?” she asks. “It doesn’t matter that you made it if you can’t distribute it.”

To Mary Ann Price, it’s important to give credit where it’s due: when VAMS works, it works. After two failed appointments and a 45-minute wait at the hospital, her third time in the system was a success. She got a QR code that made check-in a breeze at the health department’s mass vaccination site the next day, and then immediately received her vaccination card as a PDF. 

But she hasn’t scheduled her second shot. 

“There’s no point in scheduling it three weeks out, because the state of West Virginia doesn’t know if they’ll even have any vaccines,” she says. 

This story is part of the Pandemic Technology Project, supported by The Rockefeller Foundation.

Read More