A 4D Diary Co-authored by Human and Machine: Interactively Visualize, Control, and Understand Data Collection and ML Computation within Built Environments

Ubiquitous data acquisition and machine learning (ML) systems, in particular, smart cameras, have proliferated throughout all kinds of built environments for surveillance or profit-oriented analysis. However, building occupants are rarely consciously aware of their presence and have no access to the data or the one-sided analysis. Transparency and privacy concerns may be aggravated when ML inferences extract more information from the collected data. Critically investigating ML-powered smart cameras, I first performed a self-surveillance experiment documenting my personal life. It generated a vivid 4D diary that captured lively details and recreated memories through a combination of machine recognitions and personal diaries. Recognizing the benefit of empowering those whose data is collected with active inputs, I proposed to transform the conventional unidirectional data acquisition and ML computation system into a bi-directional interactive one. Presented as a 4D diary co-authored by humans and machines, I designed and prototyped a digital platform featuring interactive visualization, control, and understanding. Through user studies and interviews, I explored the interaction dynamics between occupants and the system, exploring how the prototype mitigates transparency and privacy concerns. This research offers alternatives to the dominant paradigm of passive data acquisition and computation, empowers occupants with active voices and controls, and reintroduces their agency that can be overshadowed by automated and opaque processes. It demonstrates a methodology guiding technology development toward transparency and empathy.