Chroma-Temporal Surveillance Bot

According to the book series The Hitchhikers Guide to the Galaxy, written by Douglas Adams, 42 is “the answer to the ultimate question of life, the universe, and everything” as calculated, over 7.5 million years, by the mammoth supercomputer Deep Thought. When faced with the answer to the ultimate question, the beings whom constructed Deep Thought realized they did not know what the ultimate question actually is. Thus, the beings built a second even more powerful, planet-sized supercomputer to calculate the question. After millions of years, the program came to a conclusion and it is revealed that without any doubt, that the ultimate question is: “What do you get when you multiply six by nine?”. Everyone is baffled. 6 * 9 = 42 is named after this chunk of hitchhiker’s lore due to questions the installation invites about the role of digital information, its deceptive meaning, and condemning finality. 6 * 9 = 42 seeks to bring attention to the deceptive nature of data while providing critique on our trust in data and the menace of omnipresent surveillance. The installation presents an iMac computer, with no keyboard or mouse, along with what looks like an unpainted guitar pedal with a button and a single knob. As the viewer approaches, they can see a temporally distorted video feed of their actions over the last few seconds. By pressing the button, or by turning the knob, they can modify the underlying slit-scan algorithm which displays its output on the iMac’s screen. After the user is satisfied with their experience playing with the system, they leave and go on with their day. Unbeknownst to them, the underlying algorithm, the Chroma-Temporal Surveillance Bot (CTSB), has a sinister agenda. Ten percent of the button presses randomly cause the program to perform a “political profiling” of its user. The program saves the currently displayed frame, determines the red, green, and blue color intensity for each pixel, throws out the green information, and determines if there is more red or blue overall in the image. Based on the results of this chroma analysis, the program designates a political orientation to the interactee and proceeds to tweet the analyzed image along with a proclamation of the users suspected political orientation. None of this activity is explicitly told to the user who only experiences a temporarily dropped framerate. In fact, the user needs a computer, or internet connected smart device, to visit the CTSB twitter page to get any idea of the programs secret agenda. The Chroma-Temporal Surveillance Bot (CTSB) consists of a simple hardware interface consisting of only two controls housed in a small metal box connected to a iMac computer running a few hundred lines of Processing code running on an iMac computer. CTSB serves as the functional portion of the installation 6 * 9 = 42.


This slideshow requires JavaScript.

While there is simple firmware running on an Arduino Uno inside of the CTSB hardware interface, it only serves to report button presses and switch positions to the computational core of the system which is written in Processing and is running on an iMac host computer. Besides using the state of the hardware interface to determine its internal state as expanded upon below, CTSB gains control over the iMac’s webcam which it turns on. The images captured from the webcam provide the source material for the remainder of the program. Although the true purpose of the CTSB is to run its political profiling subroutine, this can only be done when someone is interacting with the CTSB hardware. Thus, the first thing CTSB does after initializing the webcam is to create some interesting “art” with the hope of drawing a crowd of users it can proceed to tweet about. This is done by means of a specialized type of slit-scan video processing algorithm which operates on the red, green, and blue color channels with separate images masks providing color independent slit-scan processing. Upon initialization, the system creates three contrasting greyscale “master color masks” – one for each of the primary colors. Next, the program begins storing the 256 most recently captured images from the host computer’s webcam in a FIFO buffer. After updating the FIFO CTSB calculates the color information for each pixel it will display on the iMac screen one at a time. The program references the value stored in each of the three master image mask’s corresponding pixel which is used to look up a corresponding image stored in the FIFO buffer. For instance, if the first pixel returns 0, 50, and 200 from the three master color masks it will take its red value from the first frame in the FIFO buffer, its green value from the 51st frame, and its blue value from the 201st frame. This process is repeated for every single pixel of each image displayed on the iMac. The switch and button give the user control over the master color masks. When the switch is turned to a new position, the program chooses a different algorithm for drawing the master color masks and redraws the masks, resulting in a different aesthetic quality. When the pushbutton is pressed, the program reconstructs the masks using the rule set dictated by the rotary switch creating a different variation of the currently selected aesthetic. While the video produced by the CTSB is entertaining its enjoyment comes at a cost to the users privacy and possible reputation. The hidden purpose of the CTSB is not to create beautifully psychedelic videos, but instead to gather data about the political affiliation of the gallery visitors. Ten percent of the time, chosen randomly, the pushbutton is pressed the CTSB executes its surveillance subroutine. First, the program analyzes the color content of the frame it is about to display on screen. It adds up all of the red and blue values for all of the pixels and determines if the image overall contains more red or blue. Next, according to the results of this “chroma analysis”, the CTSB determines the interactee is either Republican (if there is more red) or Democrat (if there is more blue). Although the system for determining the users political affiliation is obviously naive and erroneous, the CTSB proclaims the results of its findings along with its evidence (the image) on Twitter in the form of a tweet announcing the users allegiance to the affiliated political party.


The CTSB has gone through two major hardware revisions. The first version of the interface comprised of six N.O. push-buttons on the top of a BB aluminum project enclosure box. Inside the enclosure, an Arduino Nano recognizes button presses and reports the events to a Processing sketch running on a host computer. The Processing program handles communication with the hardware, reading images from the computers webcam, computing the slit-scan processing, and displaying the video output. Each of the buttons on the controller gave the user an axis of control over the slit-scan processing algorithm. This includes: changing the buffer length; reconstructing a mask; cycling through the mask categories; toggling between red, green, and blue channel separation; inverting the color processing; tweeting a photo; and cycling through different frame rates. This multitude of control results in millions of unique optional program states. After some testing, it became apparent that the interfaces large number of controls distracted attention away from the software system which the user is intended to focus on. As interactees spent more time figuring out what each of the button do than playing with the system they controlled, the need for a simplified interface motivated the design and construction of the second version of the CTSB. The second, and final, version of the interface features only two controls instead of the six found on the prototype. The first control is a button which redraws the color masks when pressed. The second control is a twelve-position, continuous rotary switch which allows users to choose between the different algorithms used to build the color masks. The simplified design provided users a more consistent experience in a curated, yet dynamic, manner. This was done partly by removing many of the axis of control once available to the user. For instance, by removing user control over the oversampling rate of the FIFO buffer, which is interesting with some effects but can destroy others if set too low or high, and hard coding its value for each of the program states reduces the chance the user enters into a “dead state” where nothing interesting is happening. This additionally reduces confusion about each controls’ effect on the displayed image, while still inviting exploration and allowing for countless variations through the generation of new masks. As the interface needs to attract participants to collect its data, it is important that the device appears friendly, innocent, and easy to use. The hardware interface is straightforward, familiar, and easy to understand to hide the software’s more sinister purpose.