I have been working with video and animation at my job and on my own for the past four years, so it came as a surprise how difficult it was to come up with a project for Video and the Open Web. It stems from how limited my understanding of what video is; something that someone exports into a “final” product and is transportable but not mutable. In the past, I felt that programs like Nuke and After Effects were the keys to unleashing media and playing with their content. Online communities like YouTube came one step closer to what the web does so well: improvisation and sharing. However, both seemed similar to other forms of media creation, where the last exported file tends to be the last stop of the media and bringing new life to it is available to the few who have the expensive tools and the knowledge.
Now I realize, unlocking and re-mixing video media, where a community of everyone has the ability to alter video in new ways with HTML5, is the teleological evolution of the medium. The aspects of the web that makes all this possible is not just what HTML5 can do to video, but the explosion of available media on the web from sources like You-tube and Vimeo. There is a massive library of assets from TV, movies, news, advertisements and the very fertile world of our own lives.
The idea for the project, code name TubeyLoops, takes video from the internet, creates fragments from in out cue points of that media (using some EDL solution), and lays out those fragments into an 8 or 16 beat pattern sequencer well known from drum machines. In addition to controlling the audio of the clips, the sequencer would play their accompanying video as well. Video displayed on the screen can be altered with the transform and animation toolkit now available with CSS3. Effects, like hue change, posterization, or glow, can also be applied to the clips in the sequencer. A final joined clip can be shared via link.
My inspiration stems from the classic midi beat sequencer FL studios (known as fruityloops when i first came across it more than a decade ago in my college dorm room), and the Audio Illusions of Diana Deutsch. Deutsch illustrates in her book the peculiarities of how we perceive language and what happens to spoken words when they are repeated over and over like in a sequencer:
In our final demonstration, speech is made to be heard as song, and this is achieved without transforming the sounds in any way, or by adding any musical context, but simply by repeating a phrase several times over. The demonstration is based on a sentence at the beginning of the CD Musical Illusions and Paradoxes. When you listen to this sentence in the usual way, it appears to be spoken normally – as indeed it is. However, when you play the phrase that is embedded in it: ‘sometimes behave so strangely’ over and over again, a curious thing happens. At some point, instead of appearing to be spoken, the words appear to be sung, rather as in the figure below.
Here is WNYC’s radiolab providing the stunning example and anecdote behind the illusion. It is worth a listen!
The phenomenon of remixing non-sung media into song is nothing new; Antoine Dodson and the Gregory Brothers can tell you that. The popular work by the brothers relies on the audio manipulation of autotune to create pop soundscapes, borrowing heavily from the conventions of hip hop sampling. What Deutsch demonstrates is that the spoken word has a melodic element to it that can be unlocked using repetition. That means that all video on the web is capable of being a layered song. With so many assets, the arrangements can be unique and widely varied. With the right media and arrangement, truly compelling and poignant media clips could be possible. Or better yet, who knows what could be possible? A looping Henry Kissinger and Ben Stein creation could be your highest rated track on your ipod.