Hearsay

tl;dr

Client

Role

Timeframe

November 2015 – February 2016

Brief

What if devices that we talk to, could be designed to be more transparent about what they listen to and interpret?

Could these devices be more conversational and personal in the way they respond to us?

What are the possibilities that machine learning presents to interaction designers as a design material?

Outcome

Hearsay is a smart voice-enabled lamp that is ‘always in conversation’.

It is an internet-connected lamp that can be turned on and off using voice commands. But it also listens to people’s voices all the time and responds to it on an enclosed screen and with voice. For example, if it hears someone say “Hey” close by, it might say, “Hey, I’ve been waiting for you”, even if you didn’t really mean to talk to it. 

Hearsay – a smart lamp that is ‘always in conversation’

How may we frame the relationships smart devices have with their users? How may we design the manner and extent of smart devices’ participation in everyday lives? Can these devices be designed to be more transparent and evocative in their form and interaction? 

Brief

Designing for transparency in smart devices to enable reflection on their presence and participation in everyday lives

Machine learning and smart devices present designers with a new kind of material to design with – one that actively learns, adapts, and changes based on people’s interactions with it. This material is different from traditional algorithms which work based on predefined rules and conditions. But smart devices are also often designed as black-boxes that continuously and invisibly capture data about our lives. This data is used to train algorithms and help businesses understand people’s preferences, habits, and interactions.

So, their presence can lead to a lack of clarity about what is it that smart devices see, hear, capture, interpret, and share. It also raises ethical concerns such as those related to privacy. I explored these concerns and alternate design possibilities through this project. 

Mapping cultural perceptions of smart voice-based technologies

What I did

Exploration

Concept Design 
Material Exploration 
Speculative Exploration
Technical Development

Design

Prototyping 
Fabrication 
Visual Design 
Scenario Building

Role: interaction design researcher (Contract)

The project was a part of my work with the People's Smart Sculpture project. I collaborated with Jorun Børsting and Michelle Cheung during the initial concept exploration phase of the project. I worked independently on the remaining aspects of design and research in this project.

Hearsay

Hearsay is a lamp that can be switched on and off using voice commands. But it continuously captures, transcribes, interprets, and responds to everything that is being said close by. Although Hearsay can infer commands related to switching the lamp on or off, machine learning is not just used for understanding commands. It invites conversations by casually generating intriguing responses to what is being said close to it. 

If the lamp’s cover is removed, the responses are played aloud, and the light from the lamp is dimmed to create a soft and intimate environment for conversations (the uncovered state). Alternately, if the lamp is covered, the responses are simply transcribed on its screen, and the evocative form is hidden from view (the covered state).

The modes of operation of the lamp

Its internal form draws inspiration from the playful yet striking aesthetic of Italian radical design, particularly that of the Memphis-Milano design group and Studio Alchimia. The removable lampshade is translucent and shows a faint outline of an evocative physical form inside. The contrast between the minimal covered and the evocative uncovered state gives Hearsay a layered aesthetic.

How it works

Hearsay records sound through a mic that is activated whenever audible voices are detected. The mic is connected to a portable computer that streams the sound to an online speech to text service, which transcribes it. The portable computer inside Hearsay runs seq2seq, a machine learning algorithm developed for conversational modelling. This algorithm is trained on a movie subtitle data-set (the Cornell Movie-Dialogs Corpus), and so, it generates responses that have a casual and often surprising tone.

The functional elements of the lamp

Approach

Speculation and Material Exploration

INITIAL EXPLORATIONS

I collaborated with two of my colleagues from the university, Jorun Børsting and Michelle Cheung, during the initial concept exploration phase. We looked at news and popular technology websites for reports of the different ways in which machine learning was being used in smart devices. Specifically, we looked for future trends and examples of breakdowns of smart devices in everyday use. 

Together, we explored different conceptual directions for this project using the contrast between hyperbolic media trends and real-world reports of breakdowns as a kind of mood board. We created lo-fidelity cardboard prototypes of different concepts and used techniques like the wizard-of-oz to role-play the experience of interacting with these concepts.

01/

Exploring the concept, form, and technical feasibility 

MACHINE LEARNING AS A DESIGN MATERIAL

After the initial phase of the project concluded, I began exploring machine learning as a design material as a part of my PhD research.

Exploring technology as a design material requires developing a hands-on understanding of its properties, how it can be configured in different ways, and the design possibilities and limitations it creates. I began by creating small experiments with machine learning algorithms based on the ideas from the previous phase. These experiments helped me understand and experience how the algorithms worked and are trained and how they might shape the interactions and experience of a device. I explored machine learning algorithms such as recurrent networks and conversational modelling that are usually involved in voice-based interaction.

The simplified technical flow of the algorithm

Reflections: Designing with machine learning

Two takeaways that emerged from exploring and experimenting with machine learning were that these algorithms were their adaptability to different (and sometimes new) contexts of interactions, and the generative nature of the output data that they create. This makes the output from the algorithm hard to predict and can possibly bring an element of unpredictability, surprise, and curiosity to the interaction. A more detailed account of my reflections from this process can be read here and here. 

02/

Sample interaction with the algorithm

Technical prototype of the lamp built using a portable computer, sensors, relays, and an Arduino mini

This process of conceptual and material exploration helped me converge on the idea of a transparent smart voice-enabled lamp, Hearsay. The adaptability of machine learning algorithms meant that I could design Hearsay as a smart lamp that is always in conversation. In comparison, a rule-based system could only respond to pre-configured commands and would throw an error message in other cases.  Its generative, movie dialogue like responses, created a personal yet casual experience for its users. 

I wanted to design Hearsay as an object that is routinely used in everyday life. This is why the idea of Hearsay as a lamp felt like a close fit since generally, lamps are accessible and easy to open and repair. At the same time, I wanted to bring in an evocative and almost absurd aesthetic to create a contrast with commercial voice-enabled devices.

03/

A pamphlet advertising different ways in which Hearsay can be used

However, even though I tried to develop Hearsay into a functional domestic device, it wasn’t stable enough to differentiate between multiple voices simultaneously and background noises. I also did not have access to far-field microphones, so the quality of voice recognition I could actually implement was also quite poor. So, I decided to explore Hearsay conceptually through fictional scenarios of use that built on my own personal/autobiographical reflections and interactions with the prototype.

Scenarios of use for Hearsay

Publications

Presenting Hearsay at the Nordes conference in 2017