The parts, gathered in the basement of Runnals by computer science major JP Perales ’21: one Feather Huzzah ESP8266 for sending wireless data, one BNO055 for detecting movement and orientation, an on/off switch for managing battery life, a 10K resistor, and a lot of cables—yellow for data, red for power, black for ground. Forty minutes later, and voila! The sensor was ready for rehearsal.

Welcome to Colby theater, 2019.

On this stage, an arm swing was enough to change the course of the show. Through the sensor wrapped around their wrist, each performer could trigger sounds or manipulate projections in real time. This was a new kind of artistic freedom and one that explored the interaction between humans and technology. Blending together theater, music, and technology, Strings set a new precedent for interdisciplinary production.

“This kind of experimental theater production is not something that you typically see in a small college in Maine,” said Associate Professor of Composition and Theory Jon Hallstrom. “The fact that the students were willing to sort of hurl themselves off the beaten path was pretty exciting, and very unusual, too.”

Strings came to life as the capstone project of Colleen Wright ’19, a music interdisciplinary computation major, and Jay Huskins ’19, a theater interdisciplinary computation major. “What I like about the interdisciplinary major,” Wright said, “is that it lets me have a lot of freedom to explore different ways that music and computer science can coexist.”

In Strings, they did just that.

 

“This kind of experimental theater production is not something that you typically see in a small college in Maine.”
—Associate Professor of Composition and Theory Jon Hallstrom

Student performers on stage

Top: A moment of pause after the performers collectively build a ramp on stage. All are present, except for the Shadow, Cole Walsh ’19, who, as his character archetype suggests, remains mostly in the shadows. Bottom: The stage is flooded with a violent red light and imagery after the ramp is torn apart. Sam Barry ’20 (left), the Spotlight, and Sakina Mustafa ’22, the Hustler, descend upon the ramp pieces scattered around the stage.

 
But how to do it? The two pondered over the summer and decided to go beyond creating a piece that “looked cool and did cool things,” Huskins said. “We knew we wanted to use technology as a representation of the real world,” Wright said. “We were hoping to examine and ask questions about how people interact with each other and with their environment.”

They would set out to design a work in which performers could impact their environment through technology and consequently be affected by their actions.

Rather than a written script, there would be character archetypes: Spotlight (Sam Barry ’20); Shadow (Cole Walsh ’19); Seeker (Brianna LaValle ’22); Activist (John Baker ’19); Accepting (Will Sideri ’20); and Hustler (Sakina Mustafa ’22). All had a super objective and power level (either high or low) determining how they could engage with their environment.

“I assigned them based on the feeling that I wanted for each character,” said Wright. She wanted Walsh’s character, the Shadow, to be overpowering and therefore gave him low pitched, long-lasting sounds. Baker, as the Activist, got very short and punctuated sounds to fit his sharp actions. Spotlight could send a static noise to the projections while others could alter the contrast, blur images out, or change the hue.

 

“So far, for me, computer science had been numbers and things that come out on a screen, whereas here it taught me that it can be more than just that.”—JP Perales ’21, computer science major

Top: Here is the Activist, John Baker ’19, just before the system “breaks down.” In the performance, Baker was assigned very short and punctuated sounds to fit his sharp actions. He used the Colby-made sensor, wrapped around his wrist, to activate the sounds. Bottom: The Hustler, Sakina Mustafa ’22 (left), and the Spotlight, Sam Barry ’20, are fighting. In the process, they also trigger sounds via the sensors. Colleen Wright ’19 assigned each character a set of personalized sounds, matching their super objective and power level.

 

To trigger these personalized effects, there needed to be a device. That’s when Perales’s sensors came into play. “I never expected to work on something like this,” Perales said. “So far, for me, computer science has been numbers and things that come out on screen, whereas here it taught me that it can be more than just that.”

Besides the sensors, there was another piece of key technology: the laser. “JP’s motion sensor is an individual interaction system about how one’s motion can make a difference,” said computer science and art double major Heidi He ’19, adding, “And the laser is corresponding to how the world responds to us.”

To do so, He programmed the laser to scan the empty stage before each performance. The system then compared these initial readings against the new data coming in during the performance. When the performers entered the stage, their presence was detected by the system as they broke the laser’s infrared lights. Consequently, the computer triggered a sound to be played at that given moment. Proud of her work, He plans to share this code on GitHub—an open-source coding platform. Not just her code, but she hopes all of the technology in Strings could be used by other students in the future.

“They ran into all the technical challenges that you run into when you try and make something that’s practical, that works in real time and is robust,” said Professor of Computer Science Bruce Maxwell. Sensors’ batteries died during rehearsals. Programs needed to be debugged. Computers slowed down just before the show. “This was I think the biggest, most extensive thing that’s happened so far with this major. And it was definitely the most ambitious in combining projection, sensors, sound, and choreography,” he said. “I think it turned out really quite well.”

 

“This was the biggest, most extensive thing that’s happened so far with this major. And it was definitely the most ambitious in combining projection, sensors, sound, and choreography.”—Professor of Computer Science Bruce Maxwell


Top: The Activist, John Baker ’19, finally reaches to the top of the ramp, after moving back and forth on it for a while—an act directed and controlled by live music wound forward and back by the Shadow, Cole Walsh ’19. Bottom: The final scene where the system “breaks down.” Leading up to that moment, each performer gets stuck in their own loop, moving faster and faster until the system breaks down. Thus, the performers are “frozen” in place, and the scene shuts down.

 

To develop the show’s structure, Magnus Pind Bjerre, a part-time faculty at Parsons School of Design at the New School, joined the team. With years of experience in professional multimedia performance arts, he said most productions usually plug into existing systems in theater houses. “Here it’s quite different because the whole way of making the work and showing the work has been developed around custom-built devices and custom-built technology,” he said. “We have used existing software and existing software protocols, but in a custom-made way.”

The level of connectivity between all the systems running together in the piece was unusual too, he noted.

“When you take our use of interactivity and then you take the fact that we have students engineering and creating devices to trigger the software—that’s where I think it gets pretty unique, especially at a liberal arts level,” said Associate Professor of Theater and Dance Jim Thurston. “The fact that we just imagined as a whole collective to do this work and achieved it makes us competitive even in that professional context.”

Looking ahead to Colby’s plans for a new arts and innovation center—and its emphasis on interdisciplinary work—Thurston said Strings was an intentional test to see what that could be like. “With a work like this, an original work, you keep moving big building blocks into place, but you’re not one-hundred percent sure where you’re going to end up,” he said. “I think this surpassed all of our dreams.”