Sometimes the most powerful and transformative technologies emerge by accident, an unintended consequence of other developments. When this happens, the scope and power of the new technology can't be fully appreciated until after we have embedded it in our culture.
Big Data is all that and much, much more.
By now we all recognize the many revolutions of the Internet. But who could have known back in the 1960s that getting a few hulking mainframe computers to swap digital spit could lead to the Facebooked/Googled/e-banking/YouTubed world we inhabit today? But as head spinning as it all has been, simmering within the Internet upheaval bubbles another possibility that has slowly been taking shape. For now, it's called Big Data. If it lives up to its promise (or peril), it will rework the architecture of human experience in ways we simply cannot imagine.
And because our urban centers have always been engines of information, there is likely no nexus of human culture more susceptible to Big Data's hurricane winds than Big Cities.
By now, of course, you may be wondering if there's really something going on or if Big Data is just this year's overheated hype. The answer to that question is a definitive NOT HYPE and the reason can be summed up in two words: Digital Breadcrumbs.
For years now we have all been dropping digital breadcrumbs — electronic markers in 1s and 0s — spread across the wired world. From cell-phone locations to grocery store shopping choices to Facebook posts, we are leaving a record of our life that is out there to be followed by anyone with the resources and the time.
And it's not just us.
Every function of our culture is generating reams of numbers that flow into the data sphere: from the monthly billing records of public utilities to the traffic data recorded by municipal street sensors, it's all getting recorded and most of it is getting electronically archived.
If you want a physical representation of Big Data, consider this: to store all the information humanity created in just one year you'd need 80 billion 16-GB iPhones. That's enough iPhones to create a ring circling the Earth 100 times.
The premise of Big Data is deceptively simple: hidden in all that information lies a hyper-resolution map of the world's behavior in space and time. It's a representation of human life and the natural world with a fidelity we have never had before. Think of those movies where a character can stop time and then walk around poking at people and objects frozen in action. Now give that character X-ray microscopic eyes and you begin to get a feel for what Big Data allows.
The science behind Big Data lies in learning how to dive into its digital oceans to find patterns in the real world. Those patterns are the key. From the movie-renting habits of 28-year-old factory workers to the daily flow of stock trades in companies processing salmon, those patterns represent the contours of real life captured in numbers. Once you see the patterns you can understand the world's behavior. Once you understand behavior you can predict it. Once you can predict behavior you can control it. That is the true promise — and danger — of Big Data.
Cities are created human environments. They are ecosystems of energy and matter imagined into existence through human effort. Because cities are essentially ideas transformed into action, they are creatures of information and a Big Data problem. By breathing in the torrents of data cities generate every second, Big Data scientists and engineers believe they can make cities efficient, effective and responsive to human needs in ways that will reshape their very nature.
In the most ambitious vision, the Big Data of Big Cities will mean these dense hubs of human habitation, where 85 percent of all people will live by 2050, might become adaptive, almost self-aware. Given the need to create a sustainable global human culture on a finite planet with finite resources, some say the Big Data revolution can't come fast enough for Big Cities.
Examples of Big Data/Big Cities projects are everywhere as researchers, engineers and municipal planners struggle to put the rivers of data being generated to good use on issues like sustainability, security and public health.
Consider that most basic aspect of urban life for a moment: traffic. These days a trip in a car from one side of town to another would not be complete without a quick peek at Google Maps and its traffic data. Google (and a host of other platforms, like Waze) gives you a map of the city and a representation of the traffic flow (green for good, red for "you're screwed"). But how does Google get this data. To get a better real time picture of traffic flow, Google recently began using a crowd-sourcing model where people's smart phones become sensors. Smart phone locations are tracked by cell-phone companies and that gives a measure of how traffic is (or is not) flowing. So the same people who are using Google's map application may be sending Google traffic data to inform the map. Any city — or in this case Google — can buy that data and use it to monitor traffic.
But you can do more than just track traffic. You might take that traffic data and use it to find patterns that are shaping the life of your city. You could, for example, combine the traffic data with other municipal datasets to see how traffic patterns correlate with, say, the electricity use of households in neighborhoods, or the rates of heart attacks or the shopping patterns of the 18-28 year olds. All that data is out there to be mined. Within it lies the secret life of our cities.
If you're looking for the unexpected in Big Data/Big City projects you need go no further than social media. At my own school, the University of Rochester, computer scientist Henry Kuatz and his student Adam Sadilek are "mining" twitter data to track the spread of disease. Normally, the way a city gathers data about who is sick — say, how many people have the flu during flu season — is to wait for folks to come in to a hospital or to their doctor's office. So people get sick, they report it, and the numbers eventually get tallied up. But that analysis is pretty backward looking, revealing only how many people have been sick last week or last month.
Kuatz and his collaborators can search through tweets containing the words "I feel sick" in specific urban areas as they appear now. By intelligently manipulating the constraints on their data-mining, Kautz's team are trying to see the spread of the flu from one neighborhood to another by watching the degree that digital social networks map out real networks of human contact and its contagious consequences.
These kinds of Big Data uses of social media are just beginning and there will be lots of mistakes made. But, taken as a whole, they hold enormous promise. Perhaps in the future you'll get an automatic tweet telling you that yesterday you were in a room with someone who had the flu. "Drink extra fluids and get some more sleep just in case."
For all the promise of Big Data and Big Cities it's not hard to imagine the dark side. All these digital breadcrumbs we are leaving may very well leave us without a shred of privacy. And while Big Data may allow a kind hyper-subtle control over a city's functions for benign uses, like developing sustainability, that control can also be used to reign in the functions that make a vital democracy work.
Regardless, we and our big cities are entering the era of Big Data. What comes next will be what we make of it and take from it.
MELISSA BLOCK, HOST:
The Operations Center in Rio speaks to one promise of the smart city: command and control. But smarter cities are also about using data to help make a city more efficient. These days, there's more data than ever just waiting for us to analyze it: Big Data. NPR blogger and University of Rochester astrophysicist Adam Frank joins us now for a scientist's view of how a city might benefit. Hey, Adam.
ADAM FRANK, BYLINE: Hi, Melissa.
BLOCK: And, Adam, I gather you're in a spot where you can easily illustrate some of what we're talking about. You're on the street in Rochester, New York. And explain this buzzword that we're hearing, Big Data, and why you're so interested in it.
FRANK: Well, to understand all the excitement about Big Data, you need to keep just one idea in mind, and that's digital breadcrumbs.
BLOCK: Digital breadcrumbs. What's a digital breadcrumb?
FRANK: Well, they're basically the electronic traces in ones and zeroes that we've all been leaving, you know, from cellphone location records to grocery store shopping choices, even to Facebook posts. Last year, we all generated so much data that it would take about 60 billion iPhones to hold it all. That's a lot of information, and it's all out there, getting stored and being processed somewhere by super fast computers.
Now, I've been using supercomputers my whole life, but what's new about Big Data is that rather than using equations to generate wads of information, with Big Data, all that information is actually coming from the world itself. It's the real world, and that opens up really profound new possibilities.
And to see what I'm talking about, you could go to Google Maps right now and look up Highland Hospital in Rochester because that's where I'm standing in front of.
BLOCK: OK. Highland Hospital, bordered by South Avenue.
FRANK: So now turn on that traffic feature that's up on the upper right.
BLOCK: OK. And now I'm looking at South Avenue, and it shows me yellow lines on South Avenue, indicating what, slightly stalled traffic?
FRANK: Yeah. They tell - it's telling you basically traffic is not completely flowing freely. And the interesting thing about that is I can see it right here in front of me, but you, who are hundreds of miles away, can also see it. And that's because of Big Data.
BLOCK: And how is that data turning into this traffic map that I'm seeing? How are they getting this picture?
FRANK: Right. That's the amazing thing, really. So what's happening is that Google is getting the traffic information basically from cellphones. It's tracking how the mobile phones are moving. And it's a case where the data is simply available on - and it's the digital breadcrumbs that we're all leaving.
So a city - or Google in this case - could actually, you know, buy up that data and use it to monitor traffic. But you can do even more than that. And this is really the amazing thing about Big Data, is that you can take that traffic data and use it to find patterns and draw conclusions about how things work when you combine it with other kinds of Big Data.
For example, you could look at those traffic patterns and see if there's any correlation, say, with the electricity use of the people in this neighborhood or the shopping patterns of the 18 to 28-years-olds in this neighborhood.
All that data is out there, and it's all ready to be mined to be able to tell us how the world works. And then, of course, there's that other big elephant in the room of Big Data: social media.
BLOCK: And how would that work out? And how would a city benefit, say, from data gathered through social media?
FRANK: All right. Well, I think a great example of this would be health. So right now, I'm here by Highland Hospital. And to make this point, I'm going to walk into the emergency center right here.
FRANK: OK. Well, one of the reasons I'm in the Highland Hospital right now is that, you know, I'm looking at people basically talking to their doctors. And those doctors are collecting information about who's sick and how.
And the old way of doing things is you'd take all that information, put it together, and then you'd know how many people had the flu last week or last month.
But with Twitter, you can actually look at tweets and see who is reporting that they're being sick right now. So a colleague of mine at the University of Rochester, Henry Kautz, is mining Twitter data to find out who's typing in I feel sick. And using that data, they can actually watch the spread of the flu from one neighborhood to the other.
So they use these digital social networks to map out real networks of human contact and its contagious consequences. So think how powerful that could be. Maybe in the future, you're going to get a tweet at some point telling you that you were in a room yesterday with someone who had the flu, so you should drink lots of fluids and get more sleep.
BLOCK: You know, it seems like there would be lots of pitfalls there, right? So, I mean, you're not going by an official medical diagnosis at that point. You're going by somebody saying, hey, I think I have the flu, on Twitter.
FRANK: Well, you know, that's the thing. These kinds of Big Data uses of social media are just beginning, and there's going to be a lot of mistakes. But they hold enormous promise that perhaps we can design our cities to be more responsive to human behavior as its occurring.
BLOCK: What do you think the big takeaway is here, Adam?
FRANK: Well, there's enormous promise and enormous perils, really, that's going on. But I think the important thing to understand is that, you know, urban centers have always been engines of information, and hidden in all that real-world data are patterns in the way human beings go about their urban business.
So the hope is that we'll be able to use Big Data to draw new correlations that will help city officials, you know, make more effective cities, and ultimately make them more sustainable. And, of course, since 85 percent of all people are going to be living in cities by 2050, building smarter, more efficient cities has got to be the part of the solution to the problem. You know, the unintended consequences are also going to be out there, and that's the other part of what we're going to have to deal with.
BLOCK: Right. Because with Big Data, of course, come really big concerns about privacy.
FRANK: Exactly. All those digital breadcrumbs we're leaving allow people to find us and see what we're doing.
BLOCK: That's NPR blogger and astrophysicist Adam Frank talking with us from Rochester, New York. Adam, thanks so much.
FRANK: Oh, thank you.
ROBERT SIEGEL, HOST:
Next Tuesday on MORNING EDITION, the city's project goes to a Spanish city on the cutting edge of urban innovation. You can find all the stories at our website, npr.org/nprcities, and you can follow the project on Twitter, @nprcities. Transcript provided by NPR, Copyright NPR.