Why we have input devices
I think it could be reasonably argued that when we first started using personal computers, we were trying to emulate things that we already did, but with a degree of automation. For instance, there’s nothing a word processor can do that you can’t do with a pen and paper. You can write, edit, erase, shape, even make copies. But with a word processor, a lot of these tasks could be done in a fraction of a second, rather than minutes or hours.
“We’ve gotten pretty good at displaying information on a screen and finding fast and intuitive ways of interacting with it. Unfortunately, it’s still all through proxies.”
This wasn’t done without a trade-off, however. Yes, you could work infinitely faster with the information, but it all had to be done through a layer of abstraction. You couldn’t literally draw a line on the screen or wipe away an unneeded word. Instead, we had to invent an input system that simulated physical interaction. Instead of pointing at something on a screen, you use a mouse to move a virtual pointer around the screen.
Now, years later, we’ve gotten pretty good at displaying information on a screen and finding fast and intuitive ways of interacting with it. Unfortunately, it’s still all through proxies. If you have a spreadsheet column you want to move, you can’t physically grab that information and drop it somewhere else — you have to take your mouse and push a couple buttons. It’s become second nature.
Abstraction: right problem, wrong solution
The mouse, as an input device, got a lot of things right. Moving around the cursor has a direct relation to moving the physical device, something lacking in other input devices like trackballs. The button of a mouse corresponds to buttons on the screen, providing a tactile response when a button is clicked, which is something that tablets lack. Trackpads, while similar in operation to a mouse, are often limited by size and multiplied proportional movements.
What we really want — what we really need — are interfaces that mimic the world around us. We pick things up in order to move them, push them around to change their order, and stretch and compress them to change their size. That isn’t to say everything we do on a computer should be a 1:1 representation of what we do with physical objects, since it would be a huge physical strain to do the amount of work we do on computers every day if you had to wave your arms around in the air as you did them. We just need interfaces that we naturally understand and play into our base instincts.
“What we really want — what we really need — are interfaces that mimic the world around us.”
A common solution to the interface problem is to have the interface on the screen mimic interfaces we already know. For example, make an online music player that looks and behaves like a counter-top radio. You can click and drag knobs to change the volume and click in buttons to change stations. It’s a natural interface that most of us already know, and can invoke a degree of nostalgia at the same time.
Some companies have taken this to the extreme. One website in particular aims to “redefine online music” by putting the music on “virtual stereos” in “virtual rooms” inside of a “virtual building”.
Rather than solving the interface problem, this approach makes the problem infinitely worse. Not only does this create a horrendiously tedious interface for listening to music (listening to music on a website that simply has a play button takes only as long as the music takes to buffer, while this approach requires the user to do a large degree of disorientated navigation before they can listen to anything — with long load periods between each navigation point), it also introduces numerous levels of abstraction that pushes the data further and further away from the user. Instead of two intermediate levels of abstraction (user > mouse > website > music), this approach has five (user > mouse > website > avatar > virtual environment > virtual stereo > music).
Abstraction makes the problem worse, not better. Information displayed on a virtual screen in a virtual classroom (displayed on a physical screen, in a physical classroom) isn’t better than just displaying that information on a real screen. The question is: how can we make these interfaces as intuitive as possible, and make the level of abstraction between the user and data as narrow as possible?
Display and haptics
Apple and Microsoft have both been struggling with this problem. Both arrived at similar conclusions — touchscreens are a step in the right direction. By doing away with the mouse, you’re bringing the information closer to the user and inviting them to make a more personal — and physical — connection with it. Finally, you can actually drag around the information itself rather than a mouse. However, the technology is far from perfect, as a key component is missing: haptic perception, or the ability to recognize something through the sense of touch.
“Without tactile feedback, there’s no way of knowing exactly what parts of the interface your fingers are resting on.”
I can pull my cell phone from my pocket and dial a number without ever glancing at it. This is because there are physical buttons on the phone, and I can feel each individual button protruding from the phone. The button positions were quickly memorized, and how I don’t even have to look at the phone to know what I’m entering. Computer keyboards are the same way: you don’t have to look at your fingers as you type, because you can feel the edges of the keys and use that tactile information to keep your hands properly oriented. Now imagine a keyboard with a perfectly smooth surface, giving no tactile feedback whatsoever. Without the ability to feel the physical keys and use that information for orientation, you would find your hands constantly drifting slightly, and typing accurately without looking at the keys would become nearly impossible. In the same way, it’s nearly impossible to enter a number into an iPhone without looking at the screen — without tactile feedback, there’s no way of knowing exactly what parts of the interface your fingers are resting on.
This is where we face a major technological gap that’s yet to be addressed. Haptics are an important component of input devices. Not only do they remove the visual identification requirement, they also provide a more meaningful connection between the user and the information.
Augmented reality and efficient input
One of the ways in which the keyboard and mouse combination gets it right is in reducing the amount of physical exertion. Physically writing letters with a pen and paper is a good deal more strenuous than hitting buttons on a keyboard, just as clicking a button on a mouse is less strenuous than having to lift your arm and tap something on a screen. These abstract methods of inputting and manipulating information may be less than ideal, but they are efficient.
“A virtual keyboard can’t be touched, the contours of the buttons can’t be felt, and there’s no physical feedback letting you know that you have caused a change in the interface.”
One solution to this problem is to bring the interface closer to your hands, rather than your hands closer to the interface, minimizing the amount of physical movement required to interact with it. This is done via a concept called augmented reality. With augmented reality, interfaces can be projected on any surface. This can be done through displays built into the surface of glasses, making it appear to the wearer as though these displays exist in the physical world. A flat tabletop can suddenly become a keyboard, and motion tracking of the hands enables the wearer to interact with the keyboard as if it were physically there. Of course, since they keyboard is not a physical object, it’s not limited in terms of interaction. It can move, change shape, or spawn and discard features. The possibilities in terms of interaction are virtually limitless.
This technology isn’t limited to science fiction — virtual reality devices have been available for consumer purchase for the last decade, and advances in display technology has made it feasible to place transparent screens in ordinary glasses. Motion tracking continues to get better all the time. That said, we are still several years away from consumer-model augmented reality, a profitable and accessible joining of these technologies.
But again, this brings us back to the problem of haptic feedback. A virtual keyboard can’t be touched, the contours of the buttons can’t be felt, and there’s no physical feedback letting you know that you have caused a change in the interface (such as pushing a button). This puts a huge damper on the interaction, and remains one of the great interface problems yet to be solved.
When abstraction is good
“There are many situations in which abstraction is a good thing. […] Dollar bills are easier to exchange than gold, and numbers on a debit card are easier still.”
So far, it probably sounds like I think that introducing layers of abstraction between users and data is fundamentally a bad thing. This isn’t true — there are many situations in which abstraction is a good thing. For instance, there’s a lot of data that is easier to work with when abstracted. Dollar bills are easier to exchange than gold, and numbers on a debit card are easier still. Abstraction also encourages imagination, having the brain fill in incomplete details to make an experience personal. There’s a big difference between abstracted content and abstracted interfaces, and yet, imagination and personal connection can’t be ignored when developing interfaces. I think I’ve made it clear, though, that avatars and virtual interfaces are entirely the wrong way of abstracting an interface.
In short
We shouldn’t be controlling interfaces through proxies. Instead, they should be naturalistic and transparent. Augmented reality and haptics are the keys future UI design, and while the technology is still a couple years away from being consumer-ready, these are already design philosophies we should be thinking about.
These thoughts are a bit of a work in progress. If you have any suggestions or corrections, please let me know.