Think back to your favorite childhood doll or stuffed animal—the places you took it, the awkward situations it inadvertently found its way into, the conversations you had with it. For most children, dolls are constant companions, providing a calming sense of normalcy to a world that’s quickly moving around them. (I was more of a Lego kid myself.)
Now imagine that that doll was not only recording every one of your most intimate conversations and curious questions but learning how to respond to you.
In February, Mattel announced it was partnering with interactive toy company ToyTalk to produce an artificial intelligence Barbie doll. Called Hello Barbie and marked for ages 6 and up, the Wi-Fi-enabled doll looks like your average NYC hipster, with skinny pants, black flats, and a motorcycle jacket—only it’s a foot tall and actually smiling. When you talk to it, it responds using one of 8,000 preprogrammed lines of dialogue.
Hey… wanna talk about fashion for a bit?
As long as we’re talking about family…
You said you wanted to be a veterinarian when you grow up; why don’t we talk about animals!
Good morning! Oh, I had such a funny dream last night. I was dancing a ballet with a bear wearing a tutu! What’d you dream about?
The doll’s appeal is obvious: It’s like a having a hip older sibling for only $74.99. But critics and consumer advocacy groups have called the concept “creepy” and likened it to having Big Brother in the bedroom. The Campaign for a Commercial-Free Childhood (CCFC) created a petition in March that so far has garnered more than 37,000 signatures, and earlier this month, the organization’s executive director, Josh Golin, called Hello Barbie “the perfect storm of a bad toy.”
Mattel has attempted to quell some of the controversy by posting a “privacy commitment” on its website that addresses the concerns around privacy, recording, and advertising to kids. But the debate over Hello Barbie—and its implications for the future of AI toys—is far from settled. As the toy hits shelves in time for holiday shopping, it’s worth asking: Why are grown-ups so deeply freaked out by a doll?
Opening Pandora’s toy box
Hello Barbie is a modern toy in every regard. It records speech and stores it: A microphone, concealed inside Barbie’s necklace, can be activated only when a user pushed and held down its belt buckle. What’s said to Barbie is recorded and transmitted via Wi-Fi to the computer servers of ToyTalk. Speech-recognition software then converts the audio signal into a text file, which is analyzed. The correct response is chosen from thousands of lines scripted by ToyTalk and Mattel writers, and pushed to Hello Barbie for playback—all in less than a second.
Hello Barbie is programmed to flag certain important discoveries that arise during conversation with a child; anything as significant as, say, “divorce” will be retained in Barbie’s “brain” so that the doll knows how to steer the conversation around rough patches. It remembers good things, too—with the resulting effect being that Hello Barbie becomes so well-versed in the inner life of its child companion that it’s essentially the ideal best friend.
Hello Barbie’s realism is contained in her voice.
It’s important to note that ToyTalk isn’t listening in on the recordings of children talking to Hello Barbie. ToyTalk CEO Oren Jacob told the Kernel that the data is bounced from the doll to a server and back in a way that’s similar to what Apple does when an iPhone user converses with Siri. As with Siri’s voice data, the record of a child’s conversations with Hello Barbie is stored for up to two years, or until a parent deletes the account (at which point the data is scrubbed from the servers). Jacob stressed that the recordings and text data aren’t monitored or used for anything other than improving the doll’s voice recognition features.
“The parent has control over things like religious holidays—we can let them configure that in the companion app,” said Jacob, who previously spent 20 years at Pixar, in a phone interview. “If we find out that a kid’s fave color is red, we’ll flag that as the favorite color. And if we find out that something unfortunate has happened, such as the loss of a parent, that is flagged, too.”
In a video from earlier this year posted to Mattel’s Barbie hub, a woman chats with a Hello Barbie who asks her what her favorite part of New York City is. When the woman answers “Central Park,” the doll immediately agrees with her and responds: “Central Park is my favorite place in New York City. I like to wander around the park and get lost!” Barbie’s voice is a little breathless and heavily inflected with emotion: You can hear when it’s simulating excitement, amusement, or sympathy. It even jokes; at one point it tells the woman in the video: “Did you know I have a superpower? I can make myself invisible… only when no one’s looking.”
Hello Barbie’s realism is contained in its voice. While the voice is as lively and mutable as a human’s, its plastic face never moves, only emanates sound. The user is left feeling as though they are talking to a real person, as if someone is in there—or maybe there’s a great Oz hiding behind a curtain. The effect is uncanny in the way psychologist Ernst Jentsch defined it—that icky sensation that arises when one “doubts whether an apparently animate being is really alive; or conversely, whether a lifeless object might not be in fact animate.”
Kids don’t seem to have that same reaction to dolls. In fact, according to a 2010 paper by Elena Smirnova, director of the Center of Play and Toys at Moscow State University, dolls that resemble humans are essential for the kind of role-playing that kids need to do to work through the events of their lives:
The child attributes his/her own words, thoughts and feelings to the doll, to its essence, which thus becomes an outward expression of the inner world of the child and in a sort of way his/her “mouthpiece.” Such self-expression can be regarded as a form of spontaneous play reflection, through which the child begins to understand himself and events of his life. Children are known to enjoy reliving repeatedly momentous – happy or dramatic – events of their lives. With the help of toys they enact different characters and speak for them. All that, beyond doubt, enables them to place themselves outside a significant situation and, consequently, to take an attitude to it and to comprehend it. By observing how children play, one can understand their inner world and what is bothering them.
Given the way children have been shown to interact with dolls, then, there’s a strong likelihood that they will tell Hello Barbie everything: from the minutiae of their daily routines to their deepest family secrets. For privacy researchers and advocates, that’s the scariest fear of all.
The issue of consent
There are a host of issues posed by Hello Barbie that parents should probably consider.
“If we find out that a kid’s fave color is red, we’ll flag that as the favorite color. And if we find out that something unfortunate has happened, such as the loss of a parent, that is flagged, too.”
“If I had a young child, I would be very concerned that my child’s intimate conversations with her doll were being recorded and analyzed,” said Georgetown University law professor and privacy expert Angela Campbell in a statement on the CCFC website. “In Mattel’s demo, Barbie asks many questions that would elicit a great deal of information about a child, her interests, and her family. This information could be of great value to advertisers and be used to market unfairly to children.”
According to Oren Jacob, parents have to sign a consent form before allowing their child to play with Hello Barbie. Because the child’s speech is recorded, stored, and converted to text files, the interactive element of the doll raises the specter of COPPA, the Children’s Online Privacy Protection Rule. Jacob said the Hello Barbie consent form is similar to the consent form parents sign when setting up ToyTalk’s Web-based speech recognition games like Thomas & Friends Talk to You.
But while parents are fully informed of the doll’s recording features—and can even log in to access all of the data Barbie collects about their child and post that data to Twitter or Facebook for laughs—children themselves are unaware that everything they say to the doll is being listened to.
“Parents should never spy on their kids without their consent,” said Danielle Citron, a privacy expert and University of Maryland law professor who studies the way spyware is often used by perpetrators of domestic violence. “When you’re 13 and under, if companies are going to collect data from children, the parent has to consent. But the law doesn’t say that parents can spy on their kids.”
Hello Barbie isn’t spyware per se, but the option for parents to access everything the child has ever said to the doll—or what anyone says within range of the doll as a child activates its recording feature—raises questions about the security of kids who might have troubles at home.
Pam Dixon, executive director of the World Privacy Forum, told the Kernel in a phone interview, “Reading what a child has spoken to a doll is akin to reading a diary.” For the 1 in 10 children who are physically and/or emotionally abused per year, or the 1 in 16 children who are sexually abused per year, a parent secretly listening as you innocently try to confide in your doll is a disturbing prospect.
“If you have parents that are less than stellar, and a child says to the doll, ‘My parents are hitting me’ or something… it raises some sticky issues,” Dixon said. “There are all sorts of kids who are raised in terrible homes. A conversation with the doll about being beaten by your parents may lead to more beatings. There are a lot of questions.”
Hello Barbie isn’t a therapy doll—especially because it talks rather than providing a blank slate for the kind of role-play Smirnova describes—but it is uniquely poised to provide evidence in cases of child abuse. Jacob told the Kernel that ToyTalk, as a company, is not liable for the actions of parents. The company might stumble across some conversational data, though, during the course of tweaking program features.
“Reading what a child has spoken to a doll is akin to reading a diary.”
“It is possible that in the act of debugging the service or responding to a support issue, we may on occasion hear or review transcripts of conversations that we can associate with a particular account,” Jacob said. “In the very unlikely situation that we did become aware of suspected abuse, we would of course comply with applicable laws and cooperate with law enforcement agencies as we deem appropriate on a case-by-case basis.”
As with any other stored personal data controlled by a private company, Hello Barbie is also vulnerable to subpoenas and hacking.
According to Lee Tien, a senior staff attorney at the Electronic Frontier Foundation, a company like ToyTalk is likely considered a “remote computing service” (RCS) under the law—meaning that unlike a phone company or an email host, you probably don’t need a court order to subpoena its data.
“Unfortunately, the RCS protections aren’t so great,” Tien told the Kernel via email. “The government can get even speech content with a mere subpoena with prior notice to the subscriber or user.”
While it might seem far-fetched to imagine anyone wanting conversations between a child and a doll, it is feasible that the data could be subpoenaed. A parent mired in a child custody battle might call on the data to prove that things are bad at home or to try and disprove allegations from the other partner. It’s even possible to imagine the government reviewing a child’s conversation data to try to find evidence of something involving a parent; after all, when the recorder is turned on, there’s a small chance it could pick up background snippets of adult conversation. That background loophole raises questions, too, about financial privacy and HIPAA compliance: What if a child reveals to the doll that they are HIV positive, or that a parent is ill, or innocently offers up other information that is protected by federal law?
The potential for hacking is far more troublesome. Hello Barbie sends its recordings to a cloud server, where security isn’t guaranteed. When the infamous cheating-spouse site Ashley Madison had its servers hacked this year, roughly 36 million people found their data publicly available—including closeted LGBT people living in countries where homosexuality is illegal. By revealing names, credit card numbers, secret sexual proclivities, and other information, the Ashley Madison hack showed the world just how much damage can come from our interactions with technology when we take security for granted. A hack of Hello Barbie could reveal information far more intimate.
It’s important to remember that the introduction of Hello Barbie is just one part of a new interactive landscape in which nearly everything kids do is recorded and uploaded somewhere. Some parents have balked at such networked omnipresence, refusing to post any photos or otherwise identifying information of their kids online.
Eventually, every child is going to grow up to have a digital footprint, if they don’t already. For parents, deciding whether to limit that cache of identifying data—be it Facebook photos or voice data collected by Hello Barbie—is a personal choice, one that they shouldn’t be taking lightly.
At this point at least one thing’s clear: Hello Barbie is far more than mere child’s play.
Illustration by J. Longo