Research (on-going)

Visualization of complex situations to strengthen human-automation collaboration in real-time systems

Funded by the Swedish Research Council

Keeping humans in control of highly automated high stakes environments is one of the greatest emerging challenges for the combined areas of human-computer interaction and information visualization in modern time. High stakes domains are characterized by uncertain and sometimes unforeseen situations that may escalate rapidly. In industries such as nuclear power management and air traffic control, information rich high stakes environments have emerged through automation of human work together with centralization of control and coordination. This pattern is now also emerging in other parts of society, such as home health care, emergency response, and ambulance dispatch. This transformation is enabled by growing possibilities to gather real-time data about distributed dynamic activities, for instance through the Internet of Things and wearable devices.

Due to uncertainty and situation dynamics, planning in this general type of work environments has a critical element of being situated. Overarching goals and preliminary plans must be adjusted to events as they unfold. Since it has proven practically impossible in high stakes domains to plan for every contingency in detail and automate all responses, humans must stay in the loop and ultimately stay in control in the most challenging kind of situations – those that were too difficult or impractical to automate. To stay in control, situations and plans must simultaneously be considered on several levels, ranging from the status of physical items or actions, to overarching, and sometimes conflicting, qualities and goals (e.g. safety vs. efficiency). High stakes domains are also characterized by a lack of time to evaluate complex information displays and suggestions of plans for action from automated support systems. There is a substantial lack in existing methods in that the expertise of the user is not adequately exploited in the system. This is a vital step for developing efficient tools for high stakes environments.

The goal of this project is to research generic methods for visualization of complex situations to strengthen human-automation collaboration in real-time systems.
To this end we will conduct research on:

• visualization and interaction techniques specifically targeting this area.
• integrating the visualization techniques in existing automated systems.
• evaluating the effectiveness and efficiency of the proposed methods.

Amplify Teamwork With Automation

In this project portfolio the main aim is to strengthen the ability of humans to collaborate with automated systems, in particular through decision support systems.

In close collaboration with LFV (the Air Traffic Management of Sweden), we conduct research in information and interaction design, visualization, and human factors, with high relevance to Air Traffic Managment. The application area includes current and future work environments and practices in Air Traffic Management, e.g. conflict detection in area control (COOPANS), Multiple Remote Towers, and management of unmanned traffic (UTM).

Projects (examples)

• Studies on human-automation collaboration (2014-)
• PhD student project: Human performance and design of automation in Air Traffic Management (2015-)
• ViPlan - Design for Visual Planning (2016-)
• Air Sketch, PhD Student Project (2016-)
• Sound ATM (partial funding from VIPLAN) (2016)
• Transmodal UTM (2017-2018)
• Methods and tools for systematic safety work (2016)
• DIGIT - digitalization of towers (2017-2918)
• Resilient HR - resilience stratiges and maturityassessment (2017)

UTM50 - Visualization of unmanned traffic (2016-2018)

This project explores traffic management of high-volume unnmanned traffic, for cities. Our focus is traffic management concepts. As a case study, we will apply the concepts to the city of Norrköping, in an interactive visualzation of the city with simulated traffic.

Unmanned Aerial Vehicle (UAV) is today common in many places with an increasing number of applications. UAV is considered to be disruptive as such and no one knows where the market drives us, what regulator will approve, what society requires and what society will accept. One potential future is heavy UAV tra c over cities.

To go beyond line-of-sight drone operations, continous tracking and visualization of operations is a necesity.

Unmanned Traffic Management UTM is the means for managing UAV trafic.
Even though the main UTM roadmap progresses from line-of sight operations in unpopulated areas , taking a closer look at dense traffic in cities now paves the way for planning this difficult step. Studying the future admittedly comes with uncertaintly,but it is nevertheless important to develop a vision of this future, to be able to foresee some of the challenges and opportunities that lies ahead

An interactive visualization of traffic, regulatory aspects, and UTM concepts
An interactive visualization (map) of norrköping is developed that also will display simulated traffic, aspects central for regulation, and UTM concepts. It can be used for stakeholders to envision the speci c impact on key kinds of areas (e.g. residential areas, airports, logistics hubs) by UAV traffc and services.

Key areas are scanned by drones for the map, to provide a view from ground, a “giants” perspective on the city. This is to be combined with regular satellite images of the city and surrounding countryside, to envision the “big picture” of the UAV tra c, and key UTM concepts. We will then visualize services (traffic), regulation, and UTM concepts.

Concepts for managing unmanned traffic (UTM)
We conduct research through design of concepts for managing unmanned traffic (UTM) in cities, in high-traffic scenarios. We outline overarching traffic management concepts, which are then mixed and narrowed down to concepts that we evaluate through our interactive visualization, using Norrköping as the case. Traffic scenarios (based on a portfolio of envisioned services, that give rise to start point, end points, operating areas) are designed in a workshop series where we also discuss regulation of unmanned traffic (based on traffic and services from the service portfolio).

RESKILL Self-explanatory Automation through Interactive Visualization (2016-2021)

The main goal is to uncover core principles for building systems for reskilling for operators and tools for instructors. We conduct research on

• core principles for visualization of how automation works (for operators in training), RESKILL-A
• core principles for how operators work (for instructors), RESKILL-I
• how to integrate the principles in real-time systems (simulatorns).
  

With systems for reskilling to use during training, operators will be better equipped to use system later, in live operational settings, where the principles for reskilling are not used (at least not to the same extent). This issue is similar for sea and air traffic management, with applicability in other areas, in particular in the transport sector, facing similar challenges. This addresses societal goals for transportation systems, regarding safety, robustness, and efficiency.

The project addresses a main problem with automation, that humans get a weakened understanding of emerging critical situations, and that they get a weakened understanding of the basis for automated situation descriptions (what does low preparedness really mean?) or suggested actions. This is of particular importance when working during degraded (automation) modes, or when working in unexpected or unplanned situations (versus automation capabilities).

In the project, we will conduct research and development that gives a basis for developing systems, by research addressing particularly hard/important challenges, through developing prototypes, concepts, evaluation of concepts through studies of usage (human-in-the-loop).

The research team is leading in the areas of information visualization, human factors, safety, with extensive experience of human-in-the loop studies in real-time systems. The Swedish Civil Aviation Authority and the Swedish Maritime Administration will provide access to their training environments. In addition capabilities for human-in-the-lopp studies will be built up at LiU with research simulators for digital towers and ship bridges. Capabilities for mobile and stationary eye tracking in multi-display environments will also be established.

DomePres - Beyond Powerpoint

Funded by the Norrköping Fund for Research and Development

Emerging display formats, e.g. full dome, and circular 360 views, generates new opportunities for new presentation forms, that goes beyond the traditional square, single presenter, linear "powerpoint" format. It is a format that in turn builds on the ancient technique of proecting content of square transparent films on screens. To go beoynd this format limited by technology and tradition, design research is required to both uncover new formats, but also corresponding tools. New tools and formats will open the doors of these new environments to the creative industries, enabling creative and efficient processes. In particular, this project focuses on the Dome environment in the Norrköping Visualization Centre C.

In this project (2015-2017), the aim is to:

• Conduct information design research for complex display envrionments
• Conduct research on new tools and work processes for complex display environments, primarily applied to the Dome in the Visualization Centre.
• Collaborate with local companies in co-creation of tools and presentation formats.

Previous projects

Dynamic Planning of Municipality Reponse Systems

Funded by the Swedish Rescue Services Agency (MSB)

In this projcet (2011-2013) my the aim was to conduct research on interactive visualization that supports dynamic planning of resource deployment and preparedness for new events. This research aims at solving the complex preparedness situations that emerges from having resources deployed at various locations, doing tasks such as informing the public about risks, while being available to deploy in real emergenices.

• An experimental microworld environment (c3rescue) was extended to work with strategic information and decision problems, in emergency response, and medical crisis response. It was integrated with new systems for logistics calculations of preparedness and response.
• An experiment series was conducted with experienced emergency responders, across Sweden, through a mobile experiment setup.
• New techniques for interactive visualization to strengthen human-automation collaboration were elvauated, as well as limitations of the state-of-the art.

Hastily Formed Networks

Funded by the Swedish Rescue Services Agency (MSB)

In this project (2009-2011), my focus was on resilience and communications technology in rescue service operations. These are often characterized by consisting of hastily formed networks of people and technology, who need to achieve control over unexpected situations. My main contributions were:

• An activity analysis framework for analysing crisis exercises (and crisis management), regarding resilience and sensemaking: The Resilient Sensemaking and Control Model (RESCOM).
• An evaluation of the ability to create converstation spaces in hastily formed networks, through a case study of the extensive 2006 Bodträskfors forest fire.

Assumptions on accidents - generalizability for practical investigation work in different domains

Funded by the Swedish Rescue Services Agency (MSB)

In this project 2010-2011, the aim wasto explore the relevance of the notions safety culture and resilience for accident investigation practice. We outlined a tentative method for analysing resilience, focusing on risky situations where human interventions created safety margins outside of the design envelope. The aim of the method is to capture reslience-intervetions at different levels of organizations through systematic safety work. The intervetions can then be spread through the organization, and being taken into account during organizational change.

We also focussed on resistane to recommendations from accident investigations, and the strategies that investigators use to get their recommendations implemented by the recipient(s).

Assumptions on accidents and consequences for investigation and remedial action

Funded by the Swedish Rescue Services Agency (MSB)

This three-year project (2006-2009) addressed the methodology used in Sweidsh accident investigations.

The research focuses were as follows:

• What activities are given most attention and time in accident investigation?
• What causes are sought in accident investigation, as seen by investigators and as evident by accident investigation manuals?
• What does the notion "Human Factor" mean?
• How can the same kinds of factors that causes accidents also cause accident investigations to go wrong?
  

Web publishing system

To show how the findings in my thesis can be realized, I developed the Shape CMS web publishing system during 2006. It is a parallel publishing system, based on Apache Cocoon and the eXist XML database. The system can publish a set of contents in several edited channels, with different output formats, such as PDF and HTML. It differs from traditional publishing systems in that you can have as many page layouts for a content set as you want. From a research perspective, this means that a set of contents, with a given site structure, can be rendered as different genres, at the same time, which gives a concrete demonstration of the genre perspective on interaction design, which I describe in my doctoral thesis. The system is available as open source from shapecms.sourceforge.net. You can find more information about the system there. Currently, you find examples of rendering a set of contents as a traditional discussion forum, and as a blog, at the same time. You also find two online newspaper layouts, rendering the most central news-carrying page elements, found in online newspapers. One uses a short front-page, and the other a longer front page (characteristic of many Swedish online newspapers). From the perspective of design, that allows you to test two site designs at the same time, and compare them. From a research perspective, experiments could be carried out on different designs.

The Electronic Newspaper Initiative (ELIN)

Funded by the European Commission

The aim of the Electronic Newspaper Initiative project (2001-2005) was the ELIN toolkit, an add-on to the pubishing systems of media companies. The core technologies in the project were interactive video, personalisation, and automatic content adaption for mobile devices and newtwork capabilities. Also a 3D chat environment for news and advertisements was included in the toolkit. These technologies support the use and production of interactive news and advertisement on demand. Interoperability was achieved through the use of the MPEG 4,7, and 21 standards.

In the project, I acted as lead interaction designer, coordinating and carrying out desing work. This project was the main source of data for my doctoral thesis work.

Visual Interaction in the Cockpit (VINTHEC)

Funded by the European Commission

My masters thesis was made during 1997-1998 within the VINTHEC (Visual interaction in the cockpit) project, sponsored by the European Commission. The VINTHEC project was a joint research effort between companies in several countries, namely the National Aerospace Laboratory (NL) British Aerospace (UK), Defence Evaluation & Research Agency (UK), National Defence Research Establishment (SE), luleå University of Technology (SE), Saab AB (SE), H.A. Mooij Holding B.V (NL)

The project aim was to clarify the concept of situational awareness and to test and evaluate measurement tools for situational awareness, pilot workload and performance in simulation environments.

My work in the VINTHEC project was to conduct research on the eye-point-of gaze data from the simulation studies done at the National Aerospace Laboratory in Amsterdam. My work was done at the reserach site in Amsterdam, as a project employee of luleå University of Technology. My work was also included in one of the project deliverables.