This blog post first appeared in Government Executive

Behind the roller-coaster politics rocking Washington, a much quieter but just as important revolution is underway. Government managers are advancing the use of “big data,” and it’s having a big impact. It’s the center of an important effort to transform the health of the federal government and improve the outcomes of federal programs, as a recent National Academy of Public Administration report argued.

What’s the big deal about big data?

It’s tempting to look at the quiet data revolution as just the next, small, logical step in measuring government performance, a follow-on to the ongoing evolution that began with Al Gore’s National Performance Review. After that came a series of management agendas in the Bush and Obama administrations, each of which focused (in very different ways) on producing better information to drive better results. And it would be easy to say that the big data thing is just an incremental improvement.

But big data is bigly different, in 10 important respects.

1. The supply of data has exploded, with far more data from far more sources. The government has no choice about whether to embrace the big data revolution. Everything that government does, from the cost of healthcare to the state of the environment, is simply awash in numbers, as never before. The movement is too big to for anyone to ignore.
2. Not only is the government producing its own data in vastly increasing volume, m ore data are springing up in the private sector, both through the investment of private companies and through the explosive growth of social media. The city of Chicago, for example, is using restaurant reviews from Yelp to better target restaurant inspections. Even if government wanted to ignore the data revolution, it couldn’t—the data are flooding in from everywhere.
3. More data are available in real time. New York City’s famed CompStat program began because police commanders realized they didn’t need to wait until the end of the year to get crime numbers. They started collecting crime stats weekly. But now, through everything from street cameras to personnel information, government officials everywhere can see what’s happening, here and now.
4. More of the data are granular, connected to the actual operations of programs. The Centers for Disease Control and Prevention knows how flu shots are affecting the flu outbreak, and the Environmental Protection Agency can see what impact clean air rules are having on air quality—including the emissions from individual facilities. Smartphone apps track every flight flying everywhere, with planespotting possible in real time. It’s possible to drill down from broad policy to actual results, and to track what’s actually happening now.
5. For the first time, officials at the very top—in the Office of Management and Budget and in cabinet secretaries’ offices—can see the same real-time data, at the same time, as front-line managers. This is an emerging opportunity not fully developed. But the ability of top officials to understand what’s happening in individual offices and to ask the managers of those offices to respond will inevitably transform federal management.
6. More of these data are living separately from the previous worlds of policy analysis (like benefit-cost analysis and program evaluation) and performance management (like broad-scale assessments of program results). This tsunami of data is living a separate—and growing—life. It is building on different questions, using different data, processed through different tools, providing different answers.
7. These data and big-data analytics are making it possible to ask questions that were unimaginable before. Which government regulations have been on the books longest? Which regulations are most connected with others? A recent big-data study by Deloitte found that 12 percent of all sections of the Code of Federal Regulations haven’t been touched since at least the 1970s—and that 67 percent haven’t been edited since they went on the books. There are 17,800 sections that are extremely close matches with other sections, differing only in a few words. If we’re interested in simplifying federal regulations, big data analytics provide a way to know where to look. That simply wasn’t possible before.
8. The data revolution not only provides a lot more information about what’s going on. It also creates new ways of communicating what’s happening. There’s a lot of debate about “food deserts” in big cities—areas where people have little access to healthy food and no good transportation to reach better places to buy it. A Baltimore study produced a fascinating map of food deserts there. It layers four different databases on top of each other to produce a clear and compelling analysis of a very complicated problem. It’s one thing to produce sophisticated multiple regression equations that explore the interactions of important variables, but in a form no one but the cognoscenti can digest. It’s another thing to draw a map that captures important issues in individual neighborhoods, in a form that makes sense to everyone. Seeing it is believing it, and believing it makes acting on it easier.
9. The very nature of the big data revolution can help negotiate past the fundamental problem of so many previous efforts to bring better analysis to government: so much investment in so many studies that so few people read and that produced so little result. A good map—on food deserts, for example—can both identify big problems and help drive solutions. The transparency of the information makes it hard to ignore. If we can discover which federal regulations are nearly identical and overlap, there’s a natural constituency for fixing this.
10. This revolution makes it easier to attack the even more fundamental supply-and-demand problem: producing answers to questions that policy makers have, instead of trying to get them to ask the questions on which we have data. With so much information, so potentially digestible and important, it can be a lot easier to give policy makers what they want, when they want it, in a language they can understand—and act on.

It’s not an easy job. In fact, one of the biggest limitations is that there just aren’t many government workers skilled in data mining and data analytics. Another is that much of this work requires people who know how to drill down, to ask what’s going on, why it matters, and what to do about it. In fact, the data revolution isn’t so much about numbers as it is about human capital: getting people who know how to drive the revolution forward.

But it’s not like government has a choice. Society, in general, is galloping forward, and citizens won’t have much patience with a government that doesn’t keep up.

The good news, though, is that the data revolution is providing the government with enormous new opportunities. The world is constantly searching for the Next Big Thing. In government, this is it. The data revolution not only has the potential for improving the health of government organizations. It can also improve government’s effectiveness.

Donald F. Kettl is a professor at the University of Maryland School of Public Policy and a nonresident senior fellow at the Brookings Institution and the Volcker Alliance. He is the author of many books, including Escaping Jurassic Government: How to Recover America's Lost Commitment to Competence, The Politics of the Administrative Process, System Under Stress and The Next Government of the United States.