ZETA
Named after the sixth star in a constellation, Zeta is a touch-based interactive audio-visual installation artwork made for the custom 3D touch interface at the Immersive Lab of the Institute for Computer Music and Sound Technologies (ICST), Zurich University of the Arts. The artwork creates an immersive, multi-user environment that encourages users to participate in the generation of an abstract audio-visual space. The custom touch interface combines surround audio (16 speaker distributed to 3D space) and panoramic video with full-screen touch interaction. Zeta explores the opportunities of this custom interface by focusing on an interactive artwork that employs applied artificial intelligence technologies.
The visuals of Zeta put a particle engine with physics simulation into practice. The user interaction affects the particle environment by creating audio-reactive force fields where the user place or move their hands or fingers. Each force field is also influenced by the features of the generated audio as well as other force fields present at a given moment. A rich and complex interactive emerges from this audio-visual behaviours.
The artwork allows up to six simultaneous hand interactions corresponding to six musical artificial intelligence. The touch interaction in Zeta awakes a musical agent. Musical agents are software generating music in real time. The specific agent architecture in Zeta, Musical Agent based on Self-Organizing Maps (MASOM), is an audio-based architecture that learns by listening to audio files such as recordings of performances or compositions.
The artificial musical agents in Zeta have been trained on a corpus of electroacoustic music produced at ICST. The agents memorized both the sounds and textures that were used there and learned some predictive machine learning models over these compositional structures. Guided by audience interventions, these agents generate novel musical content while trying their best to imitate their human masters.
The touch interaction initiates both a musical agent that generates a composition on the fly and a force field. The generated audio output is spatialized at the spatial location of the touch interaction. The properties and the strength of this force field depend on the audio features of the musical agent’s output. This mapping, along with the non-deterministic nature of both the agents and force fields materializes an interactive space in which the user is both an actor and a spectator. Zeta was developed during Kıvanç Tatar’s and Philippe Pasquier’s two-weeks-long residency at ICST, ZHdK in December 2018.