Using Computational Models to Create Perceptually Relevant User Interfaces for Nonvisual Artifacts

Liebman, Noah

doi:https://doi.org/10.21985/n2-xb2c-fh45

Work

Using Computational Models to Create Perceptually Relevant User Interfaces for Nonvisual Artifacts

Public

Download PDF

Download All Files (.zip)

Our experience of the physical world is mediated by our senses, but while most people have five senses, interactions with computer systems are largely limited to the visual sense. When working with nonvisual artifacts, like sound, on computers, such artifacts are typically transformed, or re-encoded, into something visual. Determining how to generate visual representations of nonvisual artifacts to support people's goals is a key design challenge. In many cases, simple encodings work well, but in other cases they fail to account for the complexities of how people's nonvisual senses work. This dissertation uses computational models of human perception to introduce a new class of inter-medium encodings that can better capture information that is relevant for users. I demonstrate this in the context of mixing multitrack audio. First, I show that current visual representations of multitrack audio lack semantic relevance that can be helpful in the mixing process. Next, I use a computational model of auditory perception to infer the perceived loudness of each track of audio, and apply it in the design of MaskerAid, a system that visualizes perceived loudness and frequency masking. I then describe and report on an empirical evaluation in which I found that MaskerAid meaningfully improved user performance toward the goal of producing mixes in which each track was clearly audible. I conclude with a discussion of the practical and theoretical implications of this work.

Creator