Tracking and interaction

Learning objectives

You know the concepts outside-in tracking and inside-out tracking.
You know the three main phases of interacting with an XDI system.

Tracking of devices and users

Fluent transitions between devices requires precise tracking of the devices and their users in order to support data exchange and interactions. Data used for tracking may include, but is not limited to, geographic location, orientation, proximity to others, temperature and biometric signals.

Tracking can be categorized into outside-in tracking (sensors external to the device) and inside-out tracking (sensors within the device).

Outside-in tracking

Outside-in tracking refers to using external sensors and/or devices to track the movements and actions of a user (of a device). Examples of outside-in tracking include smart home systems that adjust e.g. lights and heating based on room occupancy, gesture control systems that use cameras to track users' gestures to allow hands-free control of devices (e.g. PlayStation Move and Nintendo Wii Remote), capacity-based device or touch recognition systems (e.g. DiamondTouch), and camera-based or person-based detection systems (see e.g. Fig 1.).

Figure 1 - Detecting persons with from a camera feed.

While there are many benefits to outside-in tracking, including the possibility to track the whole user, outside-in tracking requires additional sensors which can be constly.

Inside-out tracking

Inside-out tracking refers to using sensors (and cameras) within a device to track the movements and actions of a user (and the device). Examples of inside-out tracking include using sensors of a device for detecting device location and orientation (see e.g. Meta Quest 2), which is used in virtual reality and augmented reality applications.

Interaction techniques

Interaction with XDI systems can be divided into three main phases: (1) configuration, (2) content engagement, and (3) disengagement. During the configuration phase, the system is setup and the devices are made aware of each other. During the content engagement phase, the devices fluently perform joint tasks. Finally, in the disengagement phase, the devices of the XDI system are disconnected or dismantled.

Fig 2. lists a few examples of interaction grouped by interaction modalities (adapted). As an example, during the configuration phase, a device could be shaked to inform the start of a XDI session. This could be followed by the use of the device for, for example, mid-air pointing. Finally, a session could be finished by another shake of the device.