Hotkeys for Normal People

Proposals for making 3D, gesture-based typing a reality


Humans have used keyboards for a relatively short time. Before keyboards, we used telegraphs to efficiently encode and transmit information over long distances, and before telegraphs, the French used large nightmare windmill things to — no joke — encode semaphore messages.

Claude Chapp’s Semaphore tower, from Wikipedia.

So, the natural question is: what’s next?

If you assume, as I do, that we waste a lot of time typing individual letters, the answer is likely AR enabled, three-dimensional gesture based “typing.”

My intention in this piece is to propose enough details for you to see that 3D, gesture-based typing is worth pursuing, rather than give you the final spec itself — clearly, you’d need to hire some linguists and designers to help out.

For 3D, gesture based typing to become a reality, we’ll need both new hardware and new software / a new way of thinking about how we transmit information with fine-grained motor-movements.

Hardware for the Future of Typing

Currently, the future of typing could be realized with Google’s Project Soli or Leap Motion’s VR hand-tracking technology. (Also, note that Natural, gesture-based control is one of the missing pieces from making VR widespread.)

A rendering of how Project Soli works.

Why Bother?

Before we dive in, you may be wondering why we should bother with 3D typing.

My standard answer is that if you speed up the efficiency of typing by even 5%, the amount of increased human output is massive. (I must confess my interest in this might also be that my brain associates structured ideas with 3D physical metaphors; which makes me extra-interested in 3D typing.)

This system would be like hotkeys, but for normal people. The goal would be to touch the keys and type individual letters as infrequently as possible.

Concrete Considerations for Future Finger/Keyboard Collusion

When thinking about AR-enabled, 3D keyboards, it is important to note that any “future of typing” system must extend the 2D keyboard, not replace it. The population of people using keyboards is now too large to introduce anything completely new.

  • To stand any chance of adoption, a new typing system must build on top of the old system, and not replace it entirely. The first thing you’d need is a traditional keyboard, or something that could function as a traditional keyboard.
  • Common words that aren’t nouns, verbs or adjectives, (things like prepositions, articles, determiners and other unsexy parts of speech), could be “stored” as a section behind your thumbs, arranged by frequency. If you need to “type” the word “of”, you could flick your right thumb back. To type the word “the” you could flick your left thumb back.
  • You might also machine-learn when the user intends to type a noun or verb phrase, instead of just a noun.
  • If you use the “scoot your thumb back” mechanism, you’d probably want to have different words correspond to different locations along the arc that your thumb could describe, rather than a straight line. Flick your right thumb back real hard, and you’d get the word “that” instead of “the” etc.
  • Both the Keyboard and the connector words would be on the plane of the surface your desk.
  • Rising vertically, at different depths from the keyboard would be one or more planes where the software would place probable words that it attempts to guess that you’ll type.
  • Potentially, different planes would change at different rates — one plane could show what the next word you’ll type is likely to be, a second plane further away from you might show what the next pair of words is likely to be, or the next three-word chunk, etc.
  • The basic idea is that different locations in space map to different words. However, you could extend this by showing how words or word phrases might expand radially from a sphere. (You could select “in” and have related phrases like “in between” or “in front of” or “in time” arranged radially from the point where “in” is located.
  • You could use AR to visualize which words are where, but it could also be done with an iPad arranged to act as a “lens” into your virtual keyboard.
  • A piano gives three types of immediate feedback: haptic, visual, and sonic feedback to the player. So far, the system we’ve discussed only gives visual feedback in the AR environment. Gloves with small actuators, or better yet, ring for each finger that lightly buzz when a word is successfully intersected and selected, could make this system much easier to use.
  • For non technical use-cases: Just as in programming where you set up your tools first, you’d likely spend some time selecting words and related concepts you think are relevant first. In other words, you write the outline, use that to select then compose the article.
  • On the other hand, this would work better for technical, non-fiction articles in the beginning.
  • Actually, it would probably work best, at first, on restricted vocabularies. Like legal documents, as well as python or ruby code.
  • Some examples of gestures for python (assuming the hands are like they would be when typing, and come half an inch off the keyboard.)
  • “And” — move one’s hands away from each other along a plane
  • “Or” — move the right fingers down while simultaneously moving the left fingers up
  • For” (Like a ‘forward Or’) — both sets of fingertips dip together.
  • Def” — both sets of fingertips rise together.
  • Custom user defined variables can be assigned to different fingers… if the name of a variable that the coder imagines she or he will use frequently is “LAST_YEARS_RETURNS”, it can be assigned that to flicking the first finger of the dominant hand upwards.

These are some rough ideas designed to give you a concrete sense for how this system might work. If you have any ideas, feel free to comment below or reach me on twitter. And if you know any project Soli people who are interested, I would love to talk with them further (and/or get my hands on a dev-kit).

Data Scientist. Writer. Humanist

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store