By Sapna Maheshwari | The New York Times
April 1, 2018
Amazon ran a commercial on this year’s Super Bowl that pretended its digital assistant Alexa had temporarily lost her voice. It featured celebrities like Rebel Wilson, Cardi B and even the company’s chief executive, Jeff Bezos.
While the ad riffed on what Alexa can say to users, the more intriguing question may be what she and other digital assistants can hear — especially as more people bring smart speakers into their homes.
Amazon and Google, the leading sellers of such devices, say the assistants record and process audio only after users trigger them by pushing a button or uttering a phrase like “Hey, Alexa” or “O.K., Google.” But each company has filed patent applications, many of them still under consideration, that outline an array of possibilities for how devices like these could monitor more of what users say and do. That information could then be used to identify a person’s desires or interests, which could be mined for ads and product recommendations.
In one set of patent applications, Amazon describes how a “voice sniffer algorithm” could be used on an array of devices, like tablets and e-book readers, to analyze audio almost in real time when it hears words like “love,” bought” or “dislike.” A diagram included with the application illustrated how a phone call between two friends could result in one receiving an offer for the San Diego Zoo and the other seeing an ad for a Wine of the Month Club membership.
Some patent applications from Google, which also owns the smart home product maker Nest Labs, describe how audio and visual signals could be used in the context of elaborate smart home setups.
One application details how audio monitoring could help detect that a child is engaging in “mischief” at home by first using speech patterns and pitch to identify a child’s presence, one filing said. A device could then try to sense movement while listening for whispers or silence, and even program a smart speaker to “provide a verbal warning.”
A separate application regarding personalizing content for people while respecting their privacy noted that voices could be used to determine a speaker’s mood using the “volume of the user’s voice, detected breathing rate, crying and so forth,” and medical condition “based on detected coughing, sneezing and so forth.”
The same application outlines how a device could “recognize a T-shirt on a floor of the user’s closet” bearing Will Smith’s face and combine that with a browser history that shows searches for Mr. Smith “to provide a movie recommendation that displays, ‘You seem to like Will Smith. His new movie is playing in a theater near you.’”
In a statement, Amazon said the company took “privacy seriously” and did “not use customers’ voice recordings for targeted advertising.” Amazon said that it filed “a number of forward-looking patent applications that explore the full possibilities of new technology,” and that they “take multiple years to receive and do not necessarily reflect current developments to products and services.”
Google said it did not “use raw audio to extrapolate moods, medical conditions or demographic information.” The company added, “All devices that come with the Google Assistant, including Google Home, are designed with user privacy in mind.”
Tech companies apply for a dizzying number of patents every year, many of which are never used and are years from even being possible.
Still, Jamie Court, the president of Consumer Watchdog, a nonprofit advocacy group in Santa Monica, Calif., which published a study of some of the patent applications in December, said, “When you read parts of the applications, it’s really clear that this is spyware and a surveillance system meant to serve you up to advertisers.”
The companies, Mr. Court added, are “basically going to be finding out what our home life is like in qualitative ways.”
Google called Consumer Watchdog’s claims “unfounded,” and said, “Prospective product announcements should not necessarily be inferred from our patent applications.”
A recent Gallup poll found that 22 percent of Americans used devices like Google Home or Amazon Echo. The growing adoption of smart speakers means that gadgets, some of which contain up to eight microphones and a camera, are being placed in kitchens and bedrooms and used to answer questions, control appliances and make phone calls. Apple recently introduced its own version, called the HomePod.
But many consumers are also becoming increasingly nervous that tech companies are eavesdropping on them in order to serve them targeted ads, no matter how often the companies deny it. The recent revelations that a British political data firm, Cambridge Analytica, improperly harvested the information of 50 million Facebook users has only added to the public’s wariness over the collection and use of personal information.
Facebook, in fact, had planned to unveil its new internet-connected home products at a developer conference in May, according to Bloomberg News, which reported that the company had scuttled that idea partly in response to the recent fallout.
Both Amazon and Google have emphasized that devices with Alexa and Google Assistant store voice recordings from users only after they are intentionally triggered. Amazon’s Echo and its newer smart speakers with screens use lights to show when they are streaming audio to the cloud, and consumers can view and delete their recordings on the Alexa smartphone app or on Amazon’s website (though they are warned online that “may degrade” their experience). Google Home also has a light that indicates when it is recording, and users can similarly see and delete that audio online.
Amazon says voice recordings may help fulfill requests and improve its services, while Google says the data helps it learn over time to provide better, more personalized responses.
But the ecosystem around voice data is still evolving.
Take the thousands of third-party apps developed for Alexa called “skills,” which can be used to play games, dim lights or provide cleaning advice. While Amazon said it didn’t share users’ actual recordings with third parties, its terms of use for Alexa say it may share the content of their requests or information like their ZIP codes. Google says it will “generally” not provide audio recordings to third-party service providers, but may send transcriptions of what people say.
And some devices have already shown that they are capable of recording more than what users expect. Google faced some embarrassment last fall when a batch of Google Home Minis that it distributed at company events and to journalists were almost constantly recording.
In a starker example, detectives investigating a death at an Arkansas home sought access to audio on an Echo device in 2016. Amazon resisted, but the recordings were ultimately shared with the permission of the defendant, James Bates. (A judge later dismissed Mr. Bates’s first-degree murder charge based on separate evidence.)
Kathleen Zellner, his lawyer, said in an interview that the Echo had been recording more than it was supposed to. Mr. Bates told her that it had been regularly lighting up without being prompted, and had logged conversations that were unrelated to Alexa commands, including a conversation about football in a separate room, she said.
“It was just extremely sloppy the way the activation occurred,” Ms. Zellner said.
The Electronic Privacy Information Center has recommended more robust disclosure rules for internet-connected devices, including an “algorithmic transparency requirement” that would help people understand how their data was being used and what automated decisions were then being made about them.
Sam Lester, the center’s consumer privacy fellow, said he believed that the abilities of new smart home devices highlighted the need for United States regulators to get more involved with how consumer data was collected and used.
“A lot of these technological innovations can be very good for consumers,” he said. “But it’s not the responsibility of consumers to protect themselves from these products any more than it’s their responsibility to protect themselves from the safety risks in food and drugs. It’s why we established a Food and Drug Administration years ago.”
Because of an editing error, an earlier version of this article misstated Jeff Bezos’s role in Amazon’s Super Bowl ad. He was trying to figure out what to do when Alexa lost her voice; he did not fill in for her.
