Embodied Interaction: Bubbles
Concept and Execution
Proposed: an art installation visualizing the concept of a 'personal bubble'. People would walk around in a semi-dark room, and a circle of light, representing a normally invisible boundary, would be projected down around them. When they gesture by outstretching their arm towards another person, the circle shrinks, indicating that others can step closer. With respect to COVID-19 and social distancing, the barrier can only shrink to a 2 meter diameter, so as to force a 2 meter space between everyone.
Actual: a depth-based tracker that tracks 'person-blobs' at a certain distance away from the Kinect and assigns them a number, a lifespan, and a colored ellipse. The number is an ID, each one representing a new person-blob in the scene. The lifespan allows each person to briefly disappear/reappear without creating a new person-blob, to account for image processing inconsistencies. The colored ellipse represents each person-blob's 'personal bubble', and is size-dependent on the size of the person-blob.
Setting up the Kinect
The Kinect was taped to hang over the edge of the tallest cabinet in my room, to be directly over my head. Unfortunately, this was still too low for my purposes, due to too-low ceilings, so items taller than ~138cm causes the depth camera to return invalid values/black. I accounted for this by spending most of my trial time moving about on my knees, and by using a raised fist to represent another person.
Programming/Software Journey
I started this project without any experience with the Kinect or computer vision. I'd say at least half of the time dedicated to this project went to negotiating compatibility between the Kinect v1 (1414) and a laptop running Windows 10. I started with Processing, and first tried using the Kinect4WinSDK library. I found the documentation sparse and difficult to understand, so I changed to Daniel Shiffman's Open Kinect for Processing library, and used his tutorials on programming for the Kinect, blob tracking, and computer vision to come up with the majority of my code. I initially experienced errors when running my code, and so used an older version of Zadig (2.1) to install the libusbK (v3.0.6.0) drivers for the Kinect, replacing the ones installed by Kinect for Windows SDK v1.8. This fixed communications.
In one of his videos, Shiffman mentions using the point cloud from the Kinect to judge distances between blobs; however, I found that that made my program run too slowly. I tried switching over to openFrameworks, which would have been more powerful and efficient. Unfortunately, I was defeated by the learning curve for switching from Processing to openFrameworks, so after some exploration and difficulty with figuring out libraries (ofxCV is a popular library for computer vision), I went back to Processing.
I then tried using Isadora, which "provides a cost-effective and easy-to-program solution for fast media playback and cueing"; however, this proved to be a fruitless task of installing and uninstalling drivers via Zadig, OpenNI, and NI Mate, along with the natively supported plug-in for using OpenNI with Isadora in an attempt to get the Kinect and Isadora to communicate. This failed, I suspect due to driver issues but I never solved it.
In the end, I settled with a rather buggy implementation of a person-blob tracker, which detected blobs/heads a certain height away from the Kinect, and assigned them IDs and visual elliptical boundaries.
Demo:
The left half of this video displays what the blob detection is based on - the depth image from the Kinect has been thresholded to only track shapes a certain distance away from the Kinect. The right parallels the left side, demonstrating what may have been projected on the people, had this project succeeded. I attempted to demonstrate the head tracking capabilities; however I am very hampered by the problem of the physical space I currently am limited to - I am unable to mount the Kinect overhead and so have to 'fake' the appearance of bodies.
Going Forward
I currently have two ideas for going forward, both of which involve switching to openFrameworks for both the advanced computer vision libraries, and the speedier processing capabilities:
Connections: a path is formed between each person-blob, determined by the proximity between people - if two people are within a certain distance between them, a visual connection, such as a line, will be drawn between the two bubbles. The implementation of this would involve changing how I iterate through the scene when searching for blobs, I believe, so as to check each blob against each other for closeness.
Come Hither: a person is able to point at another person and have their bubble shrink, ideally asymmetrically to reflect the direction of the gesture. This would involve figuring out how to detect abnormalities in blobs that would signify an outstretched arm, after which the end/hand could be used as a vector to determine where to modify the encircling bubble.
Either way, given a better setup (appropriate mounting locations for the Kinect and projector), I would have to delve a lot deeper into things like contours and blob point clouds to achieve my proposed goals.