Here are some additional sketches and photos of the finished product:

The drawer left ample space for a variable voltage power supply (for the solenoids), a pair of speakers (to amplify the music), and my MacBook Pro (to run Stockfish).

Here’s a look beneath the board:

On Wednesday, I presented my progress thus far on the Stenogloves for my final project in Introduction to Physical Computing with Tom Igoe. Since my last post, I have connected the prototype keyboard to an Arduino Micro, developed an algorithm for translating chords into keystrokes, updated the typing tutor game I had demonstrated previously, and iterated through three chord layouts.

Here is the current prototype in action, with my final chord layout and updated typing tutor game:

After connecting the keyboard I discussed in my previous post to an Arduino Micro, I developed the following Arduino sketch for detecting chords and translating them into keystrokes:

int pins[10] = {2, 3, 4, 5, 6, 7, 8, 9, 10, 11};
int keyStatus[10];
int keyStatus2[10];
boolean waiting = false;
char ctrlKey = KEY_LEFT_CTRL;

boolean alt = false;

//int chords[1024] = {0, 116, 115, 117, 114, 0, 0, 118, 111, 39, 62, 0, 112, 0, 113, 119, 32, 46, 58, 93, 59, 0, 0, 125, 44, 0, 9, 0, 91, 0, 123, 45, 0, 84, 83, 85, 82, 0, 0, 86, 79, 0, 0, 0, 80, 0, 81, 87, 10, 0, 0, 0, 0, 0, 0, 0, 63, 47, 0, 0, 0, 0, 0, 0, 110, 0, 0, 0, 0, 0, 0, 0, 120, 0, 0, 0, 0, 0, 0, 0, 49, 0, 0, 0, 0, 0, 0, 0, 43, 0, 0, 0, 0, 0, 0, 0, 78, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0, 0, 0, 0, 64, 92, 0, 0, 0, 0, 0, 0, 61, 0, 0, 0, 0, 0, 0, 0, 105, 0, 0, 0, 0, 0, 0, 0, 121, 0, 0, 0, 0, 0, 0, 0, 50, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 73, 0, 0, 0, 0, 0, 0, 0, 89, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 106, 0, 0, 0, 0, 0, 0, 0, 107, 0, 0, 0, 108, 0, 109, 0, 51, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 74, 0, 0, 0, 0, 0, 0, 0, 75, 0, 0, 0, 76, 0, 77, 0, 36, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 124, 0, 0, 0, 101, 0, 0, 0, 0, 0, 0, 0, 122, 0, 0, 0, 0, 0, 0, 0, 52, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 69, 0, 0, 0, 0, 0, 0, 0, 90, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 60, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 53, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 94, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 102, 0, 0, 0, 0, 0, 96, 0, 0, 0, 0, 0, 0, 0, 0, 0, 54, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 70, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 38, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 103, 0, 0, 0, 0, 0, 0, 0, 104, 0, 0, 0, 0, 0, 0, 0, 55, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 71, 0, 0, 0, 0, 0, 0, 0, 72, 0, 0, 0, 0, 0, 0, 0, 42, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 97, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 56, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 65, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 34, 0, 0, 0, 0, 0, 0, 0, 0, 126, 0, 0, 0, 0, 0, 0, 57, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 41, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 48, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 98, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 66, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 99, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 67, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 100, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 95, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 68, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 500};
//int chords[1024] = {0, 116, 115, 117, 114, 0, 0, 118, 111, 39, 62, 0, 112, 0, 113, 119, 32, 46, 58, 93, 59, 0, 0, 125, 44, 0, 9, 0, 91, 0, 123, 45, 0, 84, 83, 85, 82, 0, 0, 86, 79, 0, 0, 0, 80, 0, 81, 87, 10, 0, 0, 0, 0, 0, 0, 0, 63, 47, 0, 0, 0, 0, 0, 0, 110, 120, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 49, 0, 0, 0, 0, 0, 0, 0, 43, 0, 0, 0, 0, 0, 0, 0, 78, 88, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 64, 92, 0, 0, 0, 0, 0, 0, 61, 0, 0, 0, 0, 0, 0, 0, 105, 109, 108, 0, 107, 0, 0, 0, 106, 0, 0, 0, 0, 0, 0, 0, 50, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 73, 77, 76, 0, 75, 0, 0, 0, 74, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 51, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 124, 0, 0, 0, 101, 121, 104, 0, 103, 0, 0, 0, 102, 0, 0, 0, 0, 0, 0, 0, 52, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 69, 89, 72, 0, 71, 0, 0, 0, 70, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 60, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 53, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 94, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 96, 0, 0, 0, 0, 0, 0, 0, 0, 0, 54, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 38, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 55, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 42, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 97, 122, 100, 0, 99, 0, 0, 0, 98, 0, 0, 0, 0, 0, 0, 0, 56, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 65, 90, 68, 0, 67, 0, 0, 0, 66, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 34, 0, 0, 0, 0, 0, 0, 0, 0, 126, 0, 0, 0, 0, 0, 0, 57, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 41, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 48, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 95, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 500};
int chords[1024] = {0, 116, 115, 0, 114, 0, 0, 0, 111, 39, 62, 0, 112, 0, 113, 0, 32, 46, 58, 93, 59, 0, 0, 125, 44, 0, 9, 0, 91, 0, 123, 45, 0, 84, 83, 0, 82, 0, 0, 0, 79, 0, 0, 0, 80, 0, 81, 0, 10, 0, 0, 0, 0, 0, 0, 0, 63, 47, 0, 0, 0, 0, 0, 0, 110, 117, 118, 0, 119, 0, 0, 0, 120, 0, 0, 0, 0, 0, 0, 0, 49, 0, 0, 0, 0, 0, 0, 0, 43, 0, 0, 0, 0, 0, 0, 0, 78, 85, 86, 0, 87, 0, 0, 0, 88, 0, 0, 0, 0, 0, 0, 0, 64, 92, 0, 0, 0, 0, 0, 0, 61, 0, 0, 0, 0, 0, 0, 0, 105, 109, 108, 0, 107, 0, 0, 0, 106, 0, 0, 0, 0, 0, 0, 0, 50, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 73, 77, 76, 0, 75, 0, 0, 0, 74, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 51, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 124, 0, 0, 0, 101, 121, 104, 0, 103, 0, 0, 0, 102, 0, 0, 0, 0, 0, 0, 0, 52, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 69, 89, 72, 0, 71, 0, 0, 0, 70, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 60, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 53, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 94, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 96, 0, 0, 0, 0, 0, 0, 0, 0, 0, 54, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 38, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 55, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 42, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 97, 122, 100, 0, 99, 0, 0, 0, 98, 0, 0, 0, 0, 0, 0, 0, 56, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 65, 90, 68, 0, 67, 0, 0, 0, 66, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 34, 0, 0, 0, 0, 0, 0, 0, 0, 126, 0, 0, 0, 0, 0, 0, 57, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 41, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 48, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 95, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 500};

void setup() {
  for (int i=0; i<10; i++) {
    pinMode(pins[i], INPUT_PULLUP);
  }
  Keyboard.begin();
}

void loop() {
  checkKeys();
  if (keyPressed()) {
    waitForRelease();
  } else {
    waiting = true;
  }
  
}

void checkKeys() {
  for (int i=0; i<10; i++) {
    int keyState = digitalRead(pins[i]);
    if (keyState == HIGH) {
      keyStatus[i] = 0;
    } else {
      keyStatus[i] = 1;
    }
  }
}

void checkKeys2() {
  for (int i=0; i<10; i++) {
    int keyState = digitalRead(pins[i]);
    if (keyState == HIGH) {
      keyStatus2[i] = 0;
    } else {
      keyStatus2[i] = 1;
    }
  }
}

void waitForRelease() {
  checkKeys();
  delay(10);
  checkKeys2();
  boolean released = oneToZero();
  while (!released) {
    checkKeys();
    delay(10);
    checkKeys2();
    released = oneToZero();
  }
  if (waiting) recordChord();
  waiting = false;
  delay(10);
}

void recordChord() {
  int ch = convert_bin2dec();
  int toType = chords[ch];
  if (toType < 256) {
    Keyboard.write(toType);
  } else {
    
    if (toType == 500) {
      alt = !alt;
      Keyboard.press(ctrlKey);
      delay(100);
      Keyboard.releaseAll();
    }    
    
  }
}

boolean keyPressed() {
  boolean kp = false;
  for (int i=0; i<10; i++) {
    if (keyStatus[i] == 1) kp = true;
  }
  return kp;
}

boolean oneToZero() {
  boolean released = false;
  for (int i=0; i<10; i++) {
    if (keyStatus[i] == 1 && keyStatus2[i] == 0) {
      released = true;
    }
  }
  return released;
}

int convert_bin2dec() {
    int val = 0;
    for ( int i = 0; i<=9 ; ++i ) {
        val = (val << 1) | keyStatus[i];
    }
    return val;
}

I experimented with a number of possible solutions involving timing windows in which a chord would be detected. However, I eventually determined that the best solution would involve detecting the chord upon key release rather than key press. The sketch above waits for any key to be released, then records the chord detected immediately prior to release.

Note that there are three arrays named “chords”—the first two are commented out. Unfortunately, the Arduino Micro’s limited storage capacity could not accommodate more than one 1024-unit integer array of chords at a time. Thus, switching between potential chord layouts required uploading a new sketch to the Arduino each time.

After developing the Arduino software, I updated the typing tutor game for use with the keyboard. Rather than a timed animation, I changed the code so that the text cursor doesn’t advance to the next letter until the prior letter has been typed. Additionally, I implemented a score system based on chord accuracy, a hint screen that pops up for 3 seconds after the grave accent (a.k.a. backtick: ‘`’, which is currently left and right ring and middle fingers together) is typed, and an “easy mode” in which the hint screen is displayed constantly and score is not kept.

After receiving feedback on my initial chord layout in class, I decided to try a new layout that included more two-finger chords (rather than three- and four-finger chords), with more coordination between right and left hands. Here is the raw JSON file for this layout:

{"\b": 640, " ": 16, "$": 240, "(": 560, ",": 24, "0": 656, "4": 272, "8": 528, "<": 320, "@": 112, "D": 546, "H": 290, "L": 162, "P": 44, "T": 33, "X": 97, "\\": 113, "`": 390, "d": 514, "h": 258, "l": 130, "p": 12, "t": 1, "x": 65, "|": 252, "#": 176, "'": 9, "+": 88, "/": 57, "3": 208, "7": 464, ";": 20, "?": 56, "C": 548, "G": 292, "K": 164, "O": 40, "S": 34, "W": 47, "[": 28, "_": 976, "c": 516, "g": 260, "k": 132, "o": 8, "s": 2, "w": 15, "{": 30, "\n": 48, "\"": 576, "&": 432, "*": 496, ".": 17, "2": 144, "6": 400, ":": 18, ">": 10, "B": 552, "F": 296, "J": 168, "N": 96, "R": 36, "V": 39, "Z": 545, "^": 368, "b": 520, "f": 264, "j": 136, "n": 64, "r": 4, "v": 7, "z": 513, "~": 585, "\t": 26, "!": 688, "%": 304, ")": 624, "-": 31, "1": 80, "5": 336, "9": 592, "=": 120, "A": 544, "E": 288, "I": 160, "M": 161, "Q": 46, "U": 35, "Y": 289, "]": 19, "a": 512, "e": 256, "i": 128, "m": 129, "q": 14, "u": 3, "y": 257, "}": 23}

As with the initial version, each typed character has a corresponding integer, which is translated into a 10-digit binary number corresponding to the 10-finger chord that must be typed.

I tested this layout extensively, and found that test subjects preferred it (almost) unanimously to the initial layout. The exact reasons varied, but I observed that individuals had an easier time typing two-finger chords than chords that involved three or more fingers. Typing speed was also 50% faster on average compared to the initial layout.

Accounting for these observations, I set out to devise another improved layout that would incorporate even more two-finger chords. In the prior layout, the letters  ‘V’ and ‘W’ still involved three- and four-finger combinations. In this layout, all letters except for ‘P’ and ‘Q’ involve two-finger combinations with the left and right hands together.

{"\b": 640, " ": 16, "$": 240, "(": 560, ",": 24, "0": 656, "4": 272, "8": 528, "<": 320, "@": 112, "D": 546, "H": 290, "L": 162, "P": 44, "T": 33, "X": 104, "\\": 113, "`": 390, "d": 514, "h": 258, "l": 130, "p": 12, "t": 1, "x": 72, "|": 252, "#": 176, "'": 9, "+": 88, "/": 57, "3": 208, "7": 464, ";": 20, "?": 56, "C": 548, "G": 292, "K": 164, "O": 40, "S": 34, "W": 100, "[": 28, "_": 976, "c": 516, "g": 260, "k": 132, "o": 8, "s": 2, "w": 68, "{": 30, "\n": 48, "\"": 576, "&": 432, "*": 496, ".": 17, "2": 144, "6": 400, ":": 18, ">": 10, "B": 552, "F": 296, "J": 168, "N": 96, "R": 36, "V": 98, "Z": 545, "^": 368, "b": 520, "f": 264, "j": 136, "n": 64, "r": 4, "v": 66, "z": 513, "~": 585, "\t": 26, "!": 688, "%": 304, ")": 624, "-": 31, "1": 80, "5": 336, "9": 592, "=": 120, "A": 544, "E": 288, "I": 160, "M": 161, "Q": 46, "U": 97, "Y": 289, "]": 19, "a": 512, "e": 256, "i": 128, "m": 129, "q": 14, "u": 65, "y": 257, "}": 23}

If I move forward with the Stenogloves, the layout above is most likely what I will integrate. Much work remains on the punctuation marks, which I am not yet satisfied with. Backspace, in particular, involves an awkward two-finger combination with the left hand, and that can be improved with the real estate gained from the new keyboard layout. In a new version of the keyboard layout, most of the punctuation marks would, in fact, resemble the alphabetical characters from the initial version of the layout (simple two- and three-finger chords).

For me, the most interesting part was going to be the proposed “typing mode” for the gloves. So, I’m going to focus on that part alone—making a pair of general-purpose typing gloves, or “stenogloves” as I’ve begun calling them.

My first step was to develop a chorded, 10-key typing system and a simple typing tutor game to learn the system. To accomplish this, I examined the Google Ngram data on English letter frequency. With over 3.5 trillion letters in the data set, here are the frequency counts for each letter:

[Chart via Peter Norvig]

At first, I attempted to create the typing system using the simplest one and two-finger chords, and mapping the letters to chords in descending order of chord difficulty. (Single-finger chords for the most common letters, simple two-finger chords for less common letters, more complex two-finger and three-finger chords for even less common letters, etc.) After creating this initial draft of the typing system, I attempted to mime my way through the alphabet, only to discover that such a system would be incredibly difficult to learn.

The system needed a common reference point—ideally, one that would allow for a mnemonic that could make learning the system easy—so I decided to try an alphabetical orientation. In this scheme, the eight most common letters in alphabetical order—’A’, ‘E’, ‘I’, ‘N’, ‘O’, ‘R’, ‘S’, ‘T’, which together account for 65% of all keystrokes—would be mapped to single-finger chords with each of the eight fingers on both hands (excluding thumbs), in sequential order from left pinky to right pinky with palms facing down. Until the letter ‘T’, letters in between these eight key letters would be typed by adding an appropriate number of fingers after the single-finger chord. (For example, ‘A’ would be left pinky, ‘B’ would be left pinky + left ring fingers, ‘C’ would be left pinky + left ring + left middle, and so on.) After the ‘T’ chord, the system continues in the opposite direction, with right pinky + right ring = ‘U’, and so on. ‘X’, ‘Y’, and ‘Z’ have special chords (left index + right index, left middle + right index, left ring + right index) because of where they fall in the alphabet. Left thumb is reserved for shift, and right thumb is reserved for space / number / punctuation.

After creating this system, I found I could mime my way through the alphabet very quickly and easily, which should provide some indicator of the difficulty with which such a system could be learned. I also added chords for numbers 0-9 and every punctuation mark, turned all the chords into 10-digit binary numbers, and converted these numbers to integers to allow them to be read easily into any computer program as a JSON file.

Here is the Python script I used to generate that JSON file:

import json

aToZ = {'A': ['LP'],
		'B': ['LP','LR'],
		'C': ['LP','LR','LM'],
		'D': ['LP','LR','LM','LI'],
		'E': ['LR'],
		'F': ['LR','LM'],
		'G': ['LR','LM','LI'],
		'H': ['LR','LM','LI','RI'],
		'I': ['LM'],
		'J': ['LM','LI'],
		'K': ['LM','LI','RI'],
		'L': ['LM','LI','RI','RM'],
		'M': ['LM','LI','RI','RM','RR'],
		'N': ['LI'],
		'O': ['RI'],
		'P': ['RI','RM'],
		'Q': ['RI','RM','RR'],
		'R': ['RM'],
		'S': ['RR'],
		'T': ['RP'],
		'U': ['RR','RP'],
		'V': ['RM','RR','RP'],
		'W': ['RI','RM','RR','RP'],
		'X': ['LI','RI'],
		'Y': ['LM','RI'],
		'Z': ['LR','RI']}

aToZbin = {'a':[],'b':[],'c':[],'d':[],'e':[],'f':[],'g':[],'h':[],'i':[],'j':[],'k':[],'l':[],'m':[],'n':[],'o':[],'p':[],'q':[],'r':[],'s':[],'t':[],'u':[],'v':[],'w':[],'x':[],'y':[],'z':[]}

fingers = ['LP', 'LR', 'LM', 'LI', 'LT', 'RT', 'RI', 'RM', 'RR', 'RP']

# lower case letters
for key, value in aToZ.iteritems():
	for finger in fingers:
		if finger in value:
			aToZbin[key.lower()].append(1)
		else:
			aToZbin[key.lower()].append(0)

# capital letters
for key in aToZbin.keys():
	l = aToZbin[key]
	m = l[:]
	m[4] = 1
	aToZbin[key.upper()] = m

# numbers 0-9
aToZbin[0] = [1,0,1,0,0,1,0,0,0,0]
aToZbin[1] = [0,0,0,1,0,1,0,0,0,0]
aToZbin[2] = [0,0,1,0,0,1,0,0,0,0]
aToZbin[3] = [0,0,1,1,0,1,0,0,0,0]
aToZbin[4] = [0,1,0,0,0,1,0,0,0,0]
aToZbin[5] = [0,1,0,1,0,1,0,0,0,0]
aToZbin[6] = [0,1,1,0,0,1,0,0,0,0]
aToZbin[7] = [0,1,1,1,0,1,0,0,0,0]
aToZbin[8] = [1,0,0,0,0,1,0,0,0,0]
aToZbin[9] = [1,0,0,1,0,1,0,0,0,0]



# symbols !-)
num_symbols = ['!', '@', '#', '$', '%', '^', '&', '*', '(', ')']
for key in aToZbin.keys():
	if key in range(10):
		l = aToZbin[key]
		m = l[:]
		m[4] = 1
		aToZbin[num_symbols[key]] = m

# space and return
aToZbin[' '] = [0,0,0,0,0,1,0,0,0,0]
aToZbin['\n'] = [0,0,0,0,1,1,0,0,0,0]

# / ?
aToZbin['/'] = [0,0,0,0,1,1,1,0,0,1]
aToZbin['?'] = [0,0,0,0,1,1,1,0,0,0]

# = +
aToZbin['='] = [0,0,0,1,1,1,1,0,0,0]
aToZbin['+'] = [0,0,0,1,0,1,1,0,0,0]

# < >
aToZbin['<'] = [0,1,0,1,0,0,0,0,0,0]
aToZbin['>'] = [0,0,0,0,0,0,1,0,1,0]

# [ ]
aToZbin['['] = [0,0,0,0,0,1,1,1,0,0]
aToZbin[']'] = [0,0,0,0,0,1,0,0,1,1]

# { }
aToZbin['{'] = [0,0,0,0,0,1,1,1,1,0]
aToZbin['}'] = [0,0,0,0,0,1,0,1,1,1]

# " '
aToZbin['\"'] = [1,0,0,1,0,0,0,0,0,0]
aToZbin['\''] = [0,0,0,0,0,0,1,0,0,1]

# , ; : . (and +space)
aToZbin[','] = [0,0,0,0,0,1,1,0,0,0]
aToZbin[';'] = [0,0,0,0,0,1,0,1,0,0]
aToZbin[':'] = [0,0,0,0,0,1,0,0,1,0]
aToZbin['.'] = [0,0,0,0,0,1,0,0,0,1]
aToZbin[', '] = [1,0,0,0,0,1,1,0,0,0]
aToZbin['; '] = [1,0,0,0,0,1,0,1,0,0]
aToZbin[': '] = [1,0,0,0,0,1,0,0,1,0]
aToZbin['. '] = [1,0,0,0,0,1,0,0,0,1]

# underscore, dash, ndash, mdash
aToZbin['_'] = [1,1,1,1,0,1,0,0,0,0]
aToZbin['-'] = [0,0,0,0,0,1,1,1,1,1]
aToZbin[u"\u2013"] = [0,0,1,1,0,1,1,1,0,0]
aToZbin[u"\u2014"] = [0,1,1,1,0,1,1,1,1,0]

# \ |
aToZbin['\\'] = [0,0,0,1,1,1,0,0,0,1]
aToZbin['|'] = [0,0,1,1,1,1,1,1,0,0]

# ~ `
aToZbin['~'] = [1,0,0,1,0,0,1,0,0,1]
aToZbin['`'] = [0,1,1,0,0,0,0,1,1,0]


# print and test uniqueness

print aToZbin

jb = aToZbin.values()

print len(jb)

unique_jb = []
duplicate = []

for i in jb:
	if i not in unique_jb:
		unique_jb.append(i)
	else:
		duplicate.append(i)

print len(unique_jb)

print duplicate


# Turn into binary

chords = {}
lookup = [[] for _ in range(1024)]

for key, value in aToZbin.iteritems():
	number = int(''.join([str(i) for i in value]), 2)
	chords[unicode(key)] = number
	lookup[number].append(key)

print chords
print lookup

with open('data.txt', 'w') as outfile:
	json.dump(chords, outfile)

And here’s the JSON file:

{": ": 530, "\u2014": 478, " ": 16, "$": 240, "(": 560, ",": 24, "0": 656, "4": 272, "8": 528, "<": 320, "@": 112, "D": 992, "H": 488, "L": 236, "P": 44, "T": 33, "X": 104, "\\": 113, "`": 390, "d": 960, "h": 456, "l": 204, "p": 12, "t": 1, "x": 72, "|": 252, "\u2013": 220, "#": 176, "'": 9, "+": 88, "/": 57, "3": 208, "7": 464, ";": 20, "?": 56, "C": 928, "G": 480, "K": 232, "O": 40, "S": 34, "; ": 532, "W": 47, "[": 28, "_": 976, "c": 896, "g": 448, "k": 200, "o": 8, "s": 2, "w": 15, "{": 30, "\n": 48, "\"": 576, "&": 432, ". ": 529, "*": 496, ".": 17, "2": 144, "6": 400, ":": 18, ">": 10, "B": 800, "F": 416, "J": 224, "N": 96, "R": 36, "V": 39, "Z": 296, "^": 368, "b": 768, "f": 384, "j": 192, "n": 64, "r": 4, "v": 7, "z": 264, "~": 585, "!": 688, "%": 304, ", ": 536, ")": 624, "-": 31, "1": 80, "5": 336, "9": 592, "=": 120, "A": 544, "E": 288, "I": 160, "M": 238, "Q": 46, "U": 35, "Y": 168, "]": 19, "a": 512, "e": 256, "i": 128, "m": 206, "q": 14, "u": 3, "y": 136, "}": 23}

Using this data, I made a simple typing tutor game in Processing that pulls the text from any news article on the web for users to type out. The code is available on Github.

The next step was to begin tinkering with actual gloves, and so I purchased a pair of inexpensive motorcycle gloves to experiment with.

I also needed to make a decision about the actuation method for each fingertip. I settled on using mechanical keyboard switches instead of force-sensitive resistors because I knew the switches would be easier to work with and would provide better tactile feedback for users. After doing a significant amount of research on mechanical keyboard components, I settled on Cherry MX Blue switches, due to their tactile feel and clicky responsiveness.

Here is a cross sectional gif of a Cherry MX Blue switch:

Tom Igoe suggested I build a simple keyboard before attaching keys to the gloves. However, I was eager to begin working with the gloves, so I turned the right one into a mouse glove using parts from a wireless mouse I purchased. Next, I plan to mount an accelerometer on the left glove, then mount keyboard switches to the fingertips on both gloves.

After playtesting the mouse glove, I built a 10-key keyboard by mounting Cherry MX Blue switches to a wooden board. I still need to connect the switches to an Arduino in order to test this keyboard, which I hope to do very soon.

It all began with an observation I made while riding the subway in the SF Bay Area. I saw a young woman using a stenographer’s keyboard on the subway, typing very quickly. I did not want to eavesdrop, so I could not determine what she was doing on the keyboard, but she was clearly producing a lot of output. Since then, I’ve been interested in the question of whether certain individuals who perform large amounts of typing on QWERTY keyboards could benefit from the use of stenographer’s (chorded) keyboards.

More recently, I had a long discussion with my classmate, Tigran Paravyan, about graphical user interfaces, and particularly the possibility of a three-dimensional interface. The interface, as we discussed it, would be for browsing the internet, but could work for browsing files on a personal computer as well. It would exist as a forest of trees, with each tree representing a browsing session that could be returned to at a later time.

Beneath each tree, on the ground, the current browsing session would be projected. The branches of the tree could contain ornaments with files or pages linked to from the current page, with progressively more remote files or pages on higher branches of the trees. The user’s browsing history would be displayed in similar structures in the roots of the trees. A user would be able to walk, climb, dig, or fly through the environment as necessary to view the desired files or pages.

The input device we discussed would be a glove with force-sensitive resistors (FSRs) in the finger tips. A user could touch her thumb to her index finger (or press her index finger on her leg, or on a table) to perform one of the four actions (walk, climb, dig, or fly), and her three other fingers (thumbs excluded) would be mapped to the three other actions. Speed of movement would be determined be the pressure a user applies to each pad. Turning could be accomplished by pressing the left hand’s pad(s) or right hand’s pad(s) independently, pressing both at the same time to move forward. Tilt sensors or accelerometers in the gloves could be mapped to other actions, such as zooming in on a particular file or page.

Alternatively, “walking” with one’s hands could translate to walking (like in the speculative video game in the movie Her—except with more tapping—see clip below), and one of the finger pads could be mapped to an auxiliary action.

NSFW LANGUAGE WARNING

Below are various notes and sketches I made in my notepad to outline this project. I plan to discuss it with Tom Igoe on Thursday and with Daniel Shiffman shortly thereafter.

Edit: Adding (low torque) servos and stiff metal plates would result in gloves that could “feel” virtual objects. Also, I purchased this pair of motorcycle gloves to begin working with.

Here are some drawings from my notebook:

The controller interfaces with a Processing sketch (the game) via an Arduino through serial communication. The player navigates a dog through a maze of cats, and when he or she touches a wall, a series of scary images and noises are displayed. If the player’s heart rate becomes elevated, the analog stick in the stuffed cat’s paw becomes more sensitive and difficult to control.

My midterm project will be a chess set that generates music and ejects pieces from the board based on Stockfish chess engine analytics. My eventual plan is to implement a physical (hardware) version of the chess set, using magnets in the pieces, Hall Effect sensors in the board, and solenoids beneath the board. However, I may rely on a software version (a chess GUI rather than a physical board) as my initial prototype. Such a version would still be connected to a physical board with solenoids beneath it to demonstrate that aspect of the project.

COMPOSITION

The chess board will be connected to the Stockfish chess engine — the world’s most powerful chess engine, which also happens to be open source. The engine will provide real-time analytics for games-in-progress, providing a score (above 0 if white is winning, below 0 if black is winning), along with the “best move” from any given board position. Mapping these variables to music will provide auditory feedback for players, turning an otherwise normal game of chess into “advanced chess” (chess where both players have access to engine analytics), but without the traditional chess engine interface. The solenoids beneath the board will provide an element of surprise and a unique way to signal that the game has ended, due to one player coming within range of a checkmate.

CONTEXT

Creating an auditory interface for the game of chess could have interesting consequences, both for chess itself and the possibility of applying such an interface to other games. I am not sure how auditory feedback will effect the game, but I hope it will make players more acutely aware of their relative strategic positions at all times. Ideally, it would provide an avenue for improvement by helping people think more like the computer chess engines.

BILL OF MATERIALS

Chess board & housings for Hall Effect sensors
64 Hall Effect sensors
32 (or more) magnets
4 solenoids
1 Arduino Mega
1 Raspberry Pi
16 multiplexor ICs
64 LEDs (if “best move” feature implemented)

TECHNICAL DRAWINGS & IMAGES

Initial Drawing (with conductive pads instead of hall effect sensors):

Rendering of Hall Effect Sensor Enclosure (for laser cutter):

Hall Effect Sensor Enclosure Prototype:

Chess GUI (software version):

SIGNAL CHAIN

Magnets >> Hall Effect Sensors >> Multiplexors >> Arduino >> Raspberry Pi (>> Music) >> Arduino >> Multiplexors >> Solenoids/LEDs

• Applications Presentation on September 16
• ITP Code Poetry Slam on November 14
• The Mechanical Turk’s Ghost
• Che55

On Tuesday, September 16, I helped deliver a presentation to our class in Applications. Yingjie Bei, Rebecca Lieberman, and Supreet Mahanti were in my group, and we utilized my Poetizer software to create an interactive storytelling exercise for the entire audience. Sarah Rothberg was kind enough to record the presentation, and Rebecca posted it on Vimeo:

I’ve also been organizing an ITP Code Poetry Slam, which will take place at 6:30pm on November 14. Submissions are now open, and I’m hoping the event will serve as a conduit for productive dialogue between the fields of poetry and computer science. Announcements regarding judges, special guests, and other details to come.

Various explorations related to the Mechanical Turk’s Ghost [working title] have consumed the rest of my time. While I wait for all the electronic components I need to arrive, I have been focusing on the software aspects of the project, along with some general aspects of the hardware.

The first revision to the preliminary design I sketched out in my prior post resulted from a friend‘s suggestion. Rather than using conductive pads on the board, I now plan to use Hall effect sensors mounted beneath the board that will react to tiny neodymium magnets embedded in each chess piece. If everything works properly, this design should be far less visible, and thus less intrusive to the overall experience. I ordered 100 sensors and 500 magnets, and I look forward to experimenting with them when they arrive.

In the meantime, the parts I listed in my prior post arrived, and I was especially excited to begin working with the Raspberry Pi. I formatted an 8GB SD card and put NOOBS on it, then booted up the Raspberry Pi and installed Raspbian, a free operating system based on Debian Linux that is optimized for the Pi’s hardware.

The Stockfish chess engine will be a major component of this project, and I was concerned that its binaries would not compile on the Raspberry Pi. The makefile documentation listed a number of options for system architecture, none of which exactly matched the ARM v6 chip on the Raspberry Pi.

First, I tried the “ARMv7” option. The compiler ran for about 10 minutes before experiencing errors and failing. I then tried several other options, none of which worked. I was about to give up completely and resign myself to running the chess engine on my laptop, when I noticed the “profile-build” option. I had never heard of profile-guided optimization (PGO), but I tried using the command “make profile-build” rather than “make build” along with the option for unspecified 32-bit architecture. This combination allowed Stockfish to compile without any issues. Here is the command that I used (from the /Stockfish/src folder):

$ make profile-build ARCH=general-32

With Stockfish successfully compiled on the Raspberry Pi, I copied the binary executable to the system path (so that I could script the engine using the Python subprocess library), then tried running the Python script I wrote to control Stockfish. It worked without any issues:

My next set of explorations revolved around the music component of the project. As I specified in my prior post, I want the device to generate music. I took some time to consider what type of music would be most appropriate, and settled on classical music as a starting point. Classical music is ideal because so many great works are in the public domain, and because so many serious chess players enjoy listening to it during play. (As anecdotal evidence, the Chess Forum in Greenwich Village, a venue where chess players congregate to play at all hours of the day and night, plays nothing but classical music all the time. I have been speaking to one of the owners of the Chess Forum about demonstrating my prototype device there once it is constructed.)

Generating a classical music mashup using data from the game in progress was the first idea I pursued. For this approach, I imagined that two classical music themes (one for black, one for white) could be combined in a way that reflected the relative strength of each side at any given point in the game. (A more complex approach might involve algorithmic music generation, but I am not ready to pursue that option just yet.) Before pursuing any prototyping or experimentation, I knew that the two themes would need to be suitably different (so as to distinguish one from the other) but also somewhat complementary in order to create a pleasant listening experience. A friend of mine who studies music suggested pairing one song (or symphony or concerto) in a major key with another song in the relative minor key.

Using YouTube Mixer, I was able to prototype the overall experience by fading back and forth between two songs. I started by pairing Beethoven’s Symphony No. 9 and Rachmaninoff’s Piano Concerto No. 3, and I was very satisfied with the results (play both these videos at once to hear the mashup):

I then worked on creating a music mashup script to pair with my chess engine script. My requirements seemed very simple: I would need a script that could play two sound files at once and control their respective volume levels independently, based on the fluctuations in the score calculated by the chess engine. The script would also need to be able to run on the Raspberry Pi.

These requirements ended up being more difficult to fulfill than I anticipated. I explored many Python audio libraries, including pyo, PyFluidSynth, mingus, and pygame’s mixer module. I also looked into using SoX, a command line audio utility, through the python subprocess library. Unfortunately, all of these options were either too complex or too simple to perform the required tasks.

Finally, on Gabe Weintraub’s suggestion, I looked into using Processing for my audio requirements and discovered a library called Minim that could do everything I needed. I then wrote the following Processing sketch:

import ddf.minim.*;

Minim minim1;
Minim minim2;
AudioPlayer player1;
AudioPlayer player2;

float gain1 = 0.0;
float gain2 = 0.0;
float tgtGain1 = 0.0;
float tgtGain2 = 0.0;
float level1 = 0.0;
float level2 = 0.0;
float lvlAdjust = 0.0;

BufferedReader reader;
String line;
float score = 0;

void setup() {
  minim1 = new Minim(this);
  minim2 = new Minim(this);
  player1 = minim1.loadFile("valkyries.mp3");
  player2 = minim2.loadFile("Rc3_1.mp3");
  player1.play();
  player1.setGain(-80.0);
  player2.play();
  player2.setGain(6.0);
}

void draw() {
  reader = createReader("score.txt");
  try {
    line = reader.readLine();
  } catch (IOException e) {
    e.printStackTrace();
    line = null;
  }
  print(line); 
  score = float(line);
  
  level1 = (player1.left.level() + player1.right.level()) / 2;
  level2 = (player2.left.level() + player2.right.level()) / 2;  

  lvlAdjust = map(level1 - level2, -0.2, 0.2, -1, 1);
  tgtGain1 = map(score, -1000, 1000, -30, 6);
  tgtGain2 = map(score, 1000, -1000, -30, 6);
  tgtGain1 = tgtGain1 * (lvlAdjust + 1);
  tgtGain2 = tgtGain2 / (lvlAdjust + 1);
  
  gain1 = player1.getGain();
  gain2 = player2.getGain();
  
  print(' ');
  print(gain1);
  print(' ');
  print(gain2);
  print(' ');
  print(level1);
  print(' ');
  println(level2);
  
  if (level2 > level1) {
    tgtGain2 -= 0.1;
  } else if (level1 < level2) {
    tgtGain1 -= 0.1;
  }
  
  player1.setGain(tgtGain1);
  player2.setGain(tgtGain2);
}

The script above reads score values from a file created by the Python script that controls the chess engine. The score values are then mapped to gain levels for each of the two tracks that are playing. I input a chess game move by move into the terminal, and the combination of scripts worked as intended by fading between the two songs based on the relative positions of white and black in the chess game.

Unfortunately, a broader issue with my overall approach became highly apparent: the dynamic qualities of each song overshadowed most of the volume changes that occurred as a result of the game. In other words, each song got louder and quieter at various points by itself, and that was more noticeable than the volume adjustments the script was making. I attempted to compensate for these natural volume changes by normalizing the volume of each song based on its relative level compared to the other song (see lines 42-45, 48-49, and 63-67 in the code above). This did not work as effectively as I hoped, and resulted in some very unpleasant sound distortions.

After conferring with my Automata instructor, Nick Yulman,  I have decided to take an alternate approach. Rather than playing two complete tracks and fading between them, I plan to record stems (individual instrument recordings) using the relevant midi files, and then create loop tracks that will be triggered at various score thresholds. I am still in the process of exploring this approach and will provide a comprehensive update sometime in the near future.

In the meantime, I have been learning about using combinations of digital and analog inputs and outputs with the Arduino, and using various input sensors to control motors, servos, solenoids, and RGB LEDs:

In Introduction to Computational Media, we are learning about object oriented programming, and Dan Shiffman asked us to create a Processing sketch using classes and objects this week. As I prepare to create a physical chessboard, I thought it would be appropriate to make a software version to perform tests. Che55 (which I named with 5’s as an homage to Processing’s original name, “Proce55ing“) was the result.

Che55 is a fully functional chess GUI, written in Processing. Only legal moves can be made, and special moves such as en passant, castling, and pawns reaching the end of the board have been accounted for. I plan to link Che55 with Stockfish in order to create chess visualizations and provide game analysis, and to prototype various elements of the Mechanical Turk’s Ghost, including the musical component. I left plenty of space around the board for additional GUI elements, which I’m currently working on implementing. All of the code is available on Github.

Unfortunately, I cannot claim credit for the chess piece designs. Rather, I was inspired by an installation I saw at the New York MoMA two weeks ago called Thinking Machine 4 by Martin Wattenberg and Marek Walczak (also written in Processing).

That’s all for now. Stay tuned for new posts about each of these projects. I will try to keep this blog more regularly updated so there (hopefully) will be no need for future multi-project megaposts like this one. Thanks for reading.