Zero Genie

Imagine you’re taking an online business class — the kind where you watch video lectures and then answer questions at the end. But this isn’t a normal class, and you’re not just watching the lectures: They’re watching you back. Every time the facial recognition system decides that you look bored, distracted, or tuned out, it makes a note. And after each lecture, it only asks you about content from those moments.

This isn’t a hypothetical system; it’s a real one deployed by a company called Nestor. And if you don’t like the sound of it, you’re not alone. Neither do the actual students.

When I asked the man behind the system, French inventor Marcel Saucet, how the students in these classes feel about being watched, he admitted that they didn’t like it. They felt violated and surveilled, he said, but he shrugged off any implication that it was his fault. “Everybody is doing this,” he told me. “It’s really early and shocking, but we cannot go against natural laws of evolution.”

As a reporter who covers technology and the future, I constantly hear variations of this line as technologists attempt to apply the theory Charles Darwin made famous in biology to their own work. I’m told that there is a progression of technology, a movement that is bigger than any individual inventor or CEO. They say they are simply caught in a tide, swept along in a current they cannot fight. They say it inevitably leads them to facial recognition (now even being deployed on children), smart speakers that record your intimate conversations, and doorbells that narc on your neighbors. They say we can’t blame these companies for the erosion of privacy or democracy or trust in public institutions — that was all going to happen sooner or later.

“When have we ever been able to keep the genie in the bottle?” they ask. Besides, they argue, people buy this stuff so they must want it. Companies are simply responding to “natural selection” by consumers. There is nobody to blame for this, they say. It’s as natural as gravity.

Perhaps no one states this belief more clearly than inventor and futurist Ray Kurzweil in his 2005 book The Singularity Is Near: “The ongoing acceleration of technology is the implication and inevitable result of what I call the law of accelerating returns, which describes the acceleration of the pace of and the exponential growth of the products of an evolutionary process.”

In fact, our world is shaped by humans who make decisions, and technology companies are no different.

To claim that these devices are the result of some kind of ever-improving natural process not only misunderstands how evolution works, but it also suggests that everything from biological weapons to fraudulent startups like Theranos to Juicero (the $400 machine that squeezed juice out of packets) are necessary and natural.

While these “innovations” range from the dangerous to the silly, they share a common thread: Nothing about them is “natural.” No natural process is creating a “smart” hairbrush or a “smart” flip flop or a “smart” condom. Or a Bluetooth-enabled toaster, a cryptocurrency from a photography company, or an internet-connected air freshener.

Evolution is a terrible metaphor for technology

Technologists’ desire to make a parallel to evolution is flawed at its very foundation. Evolution is driven by random mutation — mistakes, not plans. (And while some inventions may indeed be the result of mishaps, the decision of a company to patent, produce, and market those inventions is not.) Evolution doesn’t have meetings about the market, the environment, the customer base. Evolution doesn’t patent things or do focus groups. Evolution doesn’t spend millions of dollars lobbying Congress to ensure that its plans go unfettered.

In some situations, even if we can’t literally put a technological genie back in a bottle, we can artificially intervene to make sure the genie plays by specific rules.

There are clear laws about what companies can and can’t do in the realm of biological weapons. The FDA ensures drugs are tested for efficacy and safety before they can be sold. The USDA ensures new food research is done with care. We don’t let anybody frack or drill for oil or build nuclear power plants wherever they like. We don’t let just anybody make and sell cars or airplanes or guns.

So the assertion that technology companies can’t possibly be shaped or restrained with the public’s interest in mind is to argue that they are fundamentally different from any other industry. They’re not.

This idea persists in part because Americans cannot resist the allure of “progress”

In many ways, the American tech mogul blaming some invisible arc of innovation for his choices is simply an echo of the very ethos that founded America: progress at all costs. The men and women who colonized America largely saw progress as a march, a continual straight path, forward at all times, never to be questioned or slowed. Benjamin Franklin (himself an inventor) once wrote: “The rapid progress true science now makes occasions my regretting sometimes that I was born so soon. It is impossible to imagine the height to which may be carried, in a thousand years, the power of man over matter.”

So to suggest a slowing — or perhaps even (god forbid) a reverse — seems antithetical to the fabric of America. When former Uber engineer Anthony Levandowski says, “What already happened doesn’t really matter. You don’t need to know that history to build on what they made. In technology, all that matters is tomorrow,” he is echoing the sentiments of our nation’s founders.

“These arguments for corporate supremacy and technological progress are so deeply connected in the American mind,” says Luke Stark, a researcher who studies behavior and computing at Microsoft Research Montreal. “It’s really tied to the whole American manifest destiny and the broader history of American settler colonialism.” The attitude is that we must march on, forward, westward, no matter what stands in our way. Going back is simply not an option.

Even those afraid of all-seeing corporations argue that instead of pressing pause, we, the consumers, must simply move forward more quickly. “What I try to focus on is not to try to stop the march of technological progress,” Yuval Noah Harari, a historian and the author of Sapiens, told Time magazine. “Instead, I try to run faster.”

This endless, punishing race in the name of “progress” is often what drives consumer behavior, too. Despite the “American dream” — security, safety, prosperity — being more and more out of reach for everyday Americans, the idea that it’s just around the corner drives people to purchase these products.

If you have the newest app, people think their lives will be easier, you’ll have more free time, more quality time. Commercials promise more backyard barbecues under sparklers and birthday surprise parties facilitated by internet-connected light bulbs.

And when we buy the products, tech companies take that as a green light to continue on their “inevitable” path, inching ever toward a world where Amazon knows exactly what you’re doing, thinking, feeling — perhaps even before you do. “It’s all a loop,” says Stark. “It’s weird. That’s what puts people in this bind. They think they should be able to have it all. They can’t, and technology is a kind of prophylactic to cope with this stuff.”

Perhaps the most telling place this kind of loop shows up is in the high-tech baby device market: Bluetooth diapers, internet-connected baby monitors, an anklet that keeps track of your infant’s every move, heartbeat, and temperature fluctuation. In a country with an ever-eroding social safety net, parents are sold these surveillance devices under the guise of care and love, with a healthy dash of paranoia.

As Stark writes in his paper on consumers who surveil their loved ones, “Parents (in particular mothers) have long been reminded by advertisers that their children are at risk — from external threats, poor nutrition, social exclusion, the mother’s own failures as a parent — and that their surest route to protecting their children, and ameliorating their (newly provoked) anxiety about these dangers, is consumption.”

Buying this device, even if the details are a bit creepy, shows that you care, that you’re a good parent. And because parents are shamed and nudged into buying these tracking devices, more and more of them pop up on the market. It’s these purchases that technologists equate to “natural selection,” but it’s nearly impossible for most people to opt out of a lot of these arrangements.

With great power comes great responsibility

Often consumers don’t have much power of selection at all. Those who run small businesses find it nearly impossible to walk away from Facebook, Instagram, Yelp, Etsy, even Amazon. Employers often mandate that their workers use certain apps or systems like Zoom, Slack, and Google Docs. “It is only the hyper-privileged who are now saying, ‘I’m not going to give my kids this,’ or, ‘I’m not on social media,’” says Rumman Chowdhury, a data scientist at Accenture. “You actually have to be so comfortable in your privilege that you can opt out of things.”

And so we’re left with a tech world claiming to be driven by our desires when those decisions aren’t ones that most consumers feel good about. There’s a growing chasm between how everyday users feel about the technology around them and how companies decide what to make. And yet, these companies say they have our best interests in mind. We can’t go back, they say. We can’t stop the “natural evolution of technology.” But the “natural evolution of technology” was never a thing to begin with, and it’s time to question what “progress” actually means.

Rose Eveleth is a writer and producer who explores how humans tangle with science and technology. She’s the creator and host of Flash Forward, a podcast about possible (and not so possible) futures.