Cross-Device Interaction
Learning Objectives
- You know what Cross-Device Interaction means.
- You know some examples of developments of Cross-Device Interaction.
- You know examples of application domains of Cross-Device Interaction.
In this chapter, we strongly draw on the article Cross-Device Taxonomy: Survey, Opportunities and Challenges of Interactions Spanning Across Multiple Devices.
Multiple interconnected devices
Cross-Device Interaction (XDI) refers to using multiple interconnected devices to accomplish a task. Research in the area focuses on enabling seamless communication and data sharing between devices with the purpose of transcending beyond individual device and user. The work encompasses multi-monitor workstations, multi-device environments and spaces, and ad hoc mobile cross-device use.
Multi-monitor workstations
The history of XDI started in late 1980s when programmers commonly used a separate monochrome display for debugging and high-end video games enabled playing with several displays. As an example, the first Doom game allowed connecting three computers on a network for additional left and right views whereby the player could look left and right and see the view turned 90 degrees on a separate screen (connected to a separate computer). For personal computers, the first one to support multiple displays was the Macintosh II. Common x86 OS support for multi-displays was introduced in Windows 98.
First non-engineering professionals to adopt multiple displays may have been stock brokers with their multi-display setups (Fig. 1), factory control rooms, and emergency response centres (Fig. 2). The common denominator in these is a single person using a computer desktop system comprising of several (large) displays.
Multi-device environments and spaces
Sometimes adding more displays to a single computer is not enough. In this case, there is the need to enable fluent co-operation of several devices. One of the earliest examples of cross-device interaction was the Xerox ParcTab, which is a prototype tablet computer that was introduced in 1988. Xerox ParcTab was designed to communicate wirelessly with other devices and to provide access to context information such as the location of the user (see e.g. An Overview of the ParcTab Ubiquitous Computing Experiment).
Research in multi-device environments and spaces includes developing techniques for synchronizing data between devices, such as email and calendars. In part, this work also led to the development of technologies such as Bluetooth and Wi-Fi, which allowed devices to communicate with each other wirelessly.
One example of a multi-device environment is the Digital War Room for Design, which is a group work facility for collocated design teams allowing fluent sharing of documents, interaction with mobile devices and inbuilt memory function for past group activities (see Fig 3.). See also “Tila ryhmätyön tukena - DiWa-projektin loppuraportti” (unfortunately mostly in Finnish).
Ad hoc, mobile cross-device use
The increasing availability and use of smartphones and tablets has led to a significant increase in cross-device interaction research.
Overall, this area is perhaps best illustrated by the modern always-connected bring-your-own-device culture prominent everywhere. Fast ubiquitous networks allow portable devices to connect and interact with local services and systems in an intuitive manner. Portable devices can seamlessly access and use device-agnostic implementations of offered web services either as is, as simplified web applications, or through native apps designed for specific operating systems.
Further, recent advances have included the development of techniques for enabling communication between a wide range of devices, such as wearable devices and more broadly Internet of Things (IoT) devices.
Application domains
In a nutshell, XDI “spans across the research areas of distributed user interfaces, second-screen and multi-device research, and mobile multi-device ecosystems.” and builds ”… upon the scale of multi-display systems, display switching for multi-displays, interaction techniques for spontaneous device association, or characteristics of devices’ ownership, access, and distance” — Cross-Device Taxonomy: Survey, Opportunities and Challenges of Interactions Spanning Across Multiple Devices
Thus, it is not perhaps surprising that there exists a myriad of application domains of XDI. These include:
- Knowledge work (e.g. using multi-display environments)
- Home Computing (e.g. interconnected home entertainment systems, shared calendars/planning tools, IoT devices such as washing machines and robot vacuum cleaners)
- Data Exploration (e.g. scientific visualization in a wide variety of domains)
- Mobile Computing (e.g. multi-factor authentication)
- Games and Installations (e.g. interactive museum installations, cross-play)
- Health (e.g. patient registration, patient workflow, activity monitoring)
- Software Development (e.g. debugging and programming)
Taxonomy of cross-device design space dimensions
The article Cross-Device Taxonomy: Survey, Opportunities and Challenges of Interactions Spanning Across Multiple Devices presents a taxonomy of cross-device design space using six dimensions, which are (1) temporal, (2) configuration, (3) relationship, (4) scale, (5) dynamics, and (6) space.
Temporal dimension
Cross-Device Interaction can be either synchronous (i.e. real-time, taking place in same time) or asynchronous (i.e. a sequential flow of activities between devices).
Synchronous XDI includes activities such as controlling a home entertainment system with a mobile phone, displaying messages sent to a mobile phone in a smart watch, and editing an online document collaboratively using devices such as laptops, tablets, and mobile phones.
Asynchronous XDI involves a sequential flow of activities between devices. Examples of this include taking a photo with a mobile phone and editing it on a laptop, using a laptop to create a hotel reservation and then displaying the reservation information with a mobile phone when signing in, and using a smart speaker such as Amazon Echo to set an alarm and cancelling the alarm with a mobile phone. In the case of Asynchronous XDI, the devices are not being used simultaneously, but the actions taken on one device have an impact on the subsequent actions taken on another device.
Many instances of XDI are based on synchronous use, as it allows a seamless and intuitive interaction between devices. However, asynchronous XDI is also useful (and mandatory) in certain situations, such as when a person needs to complete a task on a device that is not readily accessible at the time.
Configuration dimension
Configuration dimension defines the actual setup of the system including the input and output devices. The configuration dimension requires considering the temporal dimension discussed previously. For synchronous configuration, there are mirrored and distributed interfaces, while for asynchronous configuration, there are migratory and cross-platform interfaces.
Mirrored and distributed interfaces
Mirrored interfaces display the same content on multiple screens and displays. Use cases of mirrored interfaces include projecting a presentation from a laptop to a larger screen and casting an image from a mobile phone to a smart TV (see Fig. 4).
Distributed interfaces include spatially distributing information over multiple devices (e.g. large display walls consisting of multiple displays that collectively display information), and logically distributing the information over multiple devices (e.g. a smart TV control application on a mobile phone). For example, content streamers commonly use logically distributed interfaces with multiple screens, where one screen might contain the streamed content such as a game, another screen a chat, a third screen the streaming software, and so on.
Migratory and cross-platform interfaces
Migratory interfaces allow starting a task on one device and continuing with the task on another device, possibly adapting the shown information to match the form factor (size, shape, etc) of the used device. Migratory interfaces provide the benefit of not having to (manually) transfer data between used devices, allowing using the device that is best suited for a particular step in the task for each step. An example of a migratory interface could be a task management application where a mobile phone would be used for easily marking tasks done, while a laptop could be used to document tasks.
Cross-platform interfaces on the other hand mostly focus on native implementation of services that seek to maintain consistency between various devices. An example of cross-platform would be cross-platform play, where players using different video game hardware can play the same games. Realizing this is not always easy, however, as illustrated by Viva La Dirt League’s YouTube video on trying to aim with a controller (note that the link contains some profanity).
Relationship dimension
The relationship dimension describes the number of users and devices partaking in the cross-device activity. As illustrated in Fig 5., a single user using a single device does not fall into the XDI categorization, while a single user with multiple devices could e.g. relate to a XDI workstation. Multiple users where each would have a single device could relate to a group activity, while multiple users each with multiple devices could relate to e.g. a collaborative setting.
Scale dimension
The scale dimension outlines the physical dimensions of the XDI and include near-scale (e.g. wearables), personal (e.g. a workstation that is within the reach of a user), social (e.g. meeting table and larger group spaces with multiple users), and public (e.g. interactive gaming setups in public spaces).
Dynamics dimension
The dynamics dimension relate to whether the setup moves. This includes fixed setups (e.g. large wall displays that cannot be moved at easy), semi-fixed setups (e.g. displays with wheels that can be connectable), and ad-hoc and mobile setups (e.g. laptops with screens). The dynamics dimension naturally also relates to the configuration dimension, outlining in what ways the setup would be used.
Space dimension
XDI can be co-located (i.e. take place in the same place) or remote (i.e. with physical separation). XDI can also share the properties of both — for example, Fig 6. outlines an information kiosk for The Finnish Science Centre Heureka that consisted of separate displays for the active user and observing larger group/audience.
Summary
The six outlined space dimensions are summarized in Fig 7., which illustrates the “Taxonomy of Cross-Device Design Space Dimensions” from the article “Cross-Device Taxonomy: Survey, Opportunities and Challenges of Interactions Spanning Across Multiple Devices”.
Tracking of devices and users
Fluent transitions between devices requires precise tracking of the devices and their users in order to support data exchange and interactions. Data used for tracking may include, but is not limited to, geographic location, orientation, proximity to others, temperature and biometric signals.
Tracking can be categorized into outside-in tracking (sensors external to the device) and inside-out tracking (sensors within the device).
Outside-in tracking
Outside-in tracking refers to using external sensors and/or devices to track the movements and actions of a user (of a device). Examples of outside-in tracking include smart home systems that adjust e.g. lights and heating based on room occupancy, gesture control systems that use cameras to track users’ gestures to allow hands-free control of devices (e.g. PlayStation Move and Nintendo Wii Remote), capacity-based device or touch recognition systems (e.g. DiamondTouch), and camera-based or person-based detection systems (see e.g. Fig 8.).
While there are many benefits to outside-in tracking, including the possibility to track the whole user, outside-in tracking requires additional sensors which can be constly.
Inside-out tracking
Inside-out tracking refers to using sensors (and cameras) within a device to track the movements and actions of a user (and the device). Examples of inside-out tracking include using sensors of a device for detecting device location and orientation (see e.g. Meta Quest 2), which is used in virtual reality and augmented reality applications.
Interaction techniques
Interaction with XDI systems can be divided into three main phases: (1) configuration, (2) content engagement, and (3) disengagement. During the configuration phase, the system is setup and the devices are made aware of each other. During the content engagement phase, the devices fluently perform joint tasks. Finally, in the disengagement phase, the devices of the XDI system are disconnected or dismantled.
Fig 9. lists a few examples of interaction grouped by interaction modalities (adapted). As an example, during the configuration phase, a device could be shaked to inform the start of a XDI session. This could be followed by the use of the device for, for example, mid-air pointing. Finally, a session could be finished by another shake of the device.
