Stage Human Action Data Acquisition for Digital Twin Systems (LINEACT CESI)

Automated Human Action Data Acquisition and Synchronization Tool for Digital Twin Systems

Keywords: Virtual Reality Training System (VRTS), Data collection and analysis, Unity, ROS, Computer vision.

Context

As part of Industry 5.0, the manufacturing process is centered around the human factor. Meticulous focus is placed on operator actions and motions, all the while ensuring their holistic well-being. Previous work has been made at CESI LINEACT on Human motion analysis (Slama et al., 2023) (Dallel et al., 2022). Within this framework, acquiring a comprehensive dataset for action recognition assumes paramount significance, given its multifaceted applications in enhancing human ergonomics and manufacturing efficiency. Acquiring such dataset has ever been made at CESI LINEACT (Dallel et al., 2020) and can be time consuming.

In parallel, digital twin and virtual reality represent technologies that can deal with several industrial issues like design, simulation and optimisation of industrial systems. Moreover, they represents tools that can acquire datasets with the ability to setup specific parameters (Dallel et al., 2023). In this context, the use of VR to acquire labelled datasets representing operator performing their activities become very interesting solution. In fact, it helps not only acquiring data and labelling actions instantaneously but also simulate different lightening conditions and camera point of views.

Work

During this internship, the focus will be on developing an automated tool with the primary objectives of:

• Acquiring and synchronizing various types of data, including dynamic human skeleton poses, discrete events such as 'grasping a tool,' 'walking,' 'assembling,' and object positions.

• Utilizing necessary sensors to track operator movements, such as Mocap Suit, Hand Mocap Gloves, VR/Real Camera or Real Camera View, Egocentric View, VR/Real Camera IR (Depth Camera), and VR/Real Point Cloud.

• Replaying the acquired data using a suitable tool to generate data with diverse acquisition parameters, including various camera settings.

• Establishing an acquisition protocol for a dataset specifically designed for the assembly system using the flexible manufacturing system (FMS) provided by LINEACT laboratory.

• Organizing the acquired dataset and implementing pre-processing techniques while validating the setup through a visualization code.

• Presenting and publishing both the developed tool and the dataset in either an indexed conference or a reputable scientific journal paper.

Technically, the main features to develop in the acquisition tool:

• Develop connectors to send data to acquisition tool.

• Develop the virtual acquisition environment in Unity, replicating an existing real environment.
  • Integrate JENII's 3D models with grab/teleport functionality.
  • Manage VR interactions to send discrete events (grab, release).
  • Develop virtual camera for acquiring image.
• Save data with synchronization using the appropriate process.
  • Define the data acquisition protocol.
  • Constraints:
    • Acquisition with ordered tasks (well-defined by the activity's protocol).
    • Acquisition with unordered tasks (based on the operator's perspective on a given activity).

CESI LINEACT laboratory

CESI LINEACT (UR 7527), Digital Innovation Laboratory for Companies and Apprenticeships for the Competitiveness of Territories, LINEACT CESI anticipates and accompanies technological changes in sectors and services related to industry, construction and digital technology. CESI‘s historical proximity to companies is a determining factor for our research activities, and has led us to focus our efforts on applied research close to the company and in partnership with them. A human-centered approach coupled with the use of technologies, as well as the territorial network and the links with training, have made it possible to build a transversal research; it puts the human, its needs and its uses, at the center of its problems and approaches the technological angle through these contributions.

Research is positioned on two complementary and unifying scientific themes and two major areas of application:

• Team 1: “Learning and Innovating”, which brings together cognitive, social and management sciences, as well as training and innovation sciences and techniques; Theme 1 can address application areas broader than those of the City or the Industry of the Future. The main scientific objectives targeted by this theme are the understanding of the effects of the environment, and more particularly of situations instrumented by technical objects (platforms, prototyping workshops, immersive systems…) on learning and creativity processes.

• Team 2: “Engineering and digital tools” which brings together skills in the fields of digital sciences and engineering sciences; it is mostly adapted to the two application areas below. The main scientific objectives of this theme relate to the modeling, simulation, optimization and data analysis of industrial or urban systems. The research work also focuses on the associated decision support tools and on human-machine interaction through the study of digital twins coupled with virtual or augmented environments and applied robotics.

The two application domains of the City of the Future and the Industry of the Future enable us to combine the business skills essential to the implementation of our projects around our scientific themes. They respond to the two major challenges that are the energy and digital transitions of our societies.

Skills

• A master level in computer sciences, with a speciality in Virtual Reality

• Basic knowledge in Deep learning

• Skills in Unity - C# and Virtual Reality, Python

• Knowledge in ROS or RTmaps would be appreciated

• Human skills
  • Good interpersonnal skills
  • English writing ability

Contacts

• Vincent HAVARD vhavard AT cesi.fr, lecturer, CESI LINEACT Rouen.

• Rim SLAMA SALMI rsalmi AT cesi.fr, lecturer, CESI LINEACT Lyon.

• Vincent VAUCHEY vvauchey AT cesi.fr, Senior research engineer,

How to apply

Submit you application to Vincent Havard vhavard AT cesi.fr and Rim SLAMA SALMI rsalmi AT cesi.fr

Please, fill the email object as: “[Internship] VR-HAR-Dataset”

The application must contain:

• CV ;

• A cover letter for the subject ;