Postdoc position (m/f/d) : Volumetric Video Capture with RGB-D cameras
Summary of the research themes
- Edge computing system,
- Graphic Quality Improvement,
- Volumetric Video Data encoding and compression
French Touch Factory, ITHACA Project : Immersion and Training with Holograms for Art, Craft and Audiovisual
Keywords : Volumetric Video, virtual reality, 3D interaction, edge computing, optimization algorithms, gesture training,
1. Context
The production of volumetric videos opens up new markets in various fields such as immersive storytelling, video games, digital training in industry, crafts, and healthcare. Despite being very costly, it remains limited to large-scale productions. The goal of ITHACA is to develop capture software based on RGB-D technology (depth cameras) and a post-production platform for affordable volumetric video production.
2. Research
A European consortium of four complementary entities:
- French Touch Factory, XR and Volumetric Video specialist,
- Research laboratory Center for Robotics Mines Paris - Virtual and Augmented Reality,
- Fraunhofer IPK (Germany),
- and Sergio Ochoa VFX Studio (Spain).
For the past year, the consortium has been developing a series of capture, post-production, fusion, and data export tools for volumetric video at 30 frames per second (fps). The success of the project relies on the use of algorithms leveraging the GPU of various devices (PCs, PC VR, standalone MR headsets, Androids, etc.).
Summary: The processing steps for volumetric data are as follows:
1. Capture: Utilize capture devices to record volumetric video data. Transitioning from a wired camera system connected to one or more master PCs to an edge computing approach based on Jetson Nanos. (By using for instance the new version of cameras ORBBEC Femto Mega or Bolt) [1]
2. Registration: Align and synchronize multiple data sources or viewpoints to ensure spatial coherence. Transitioning from traditional registration algorithms like Iterative Closest Point (ICP) to adaptive algorithms or those utilizing machine learning. [2]
3. Post-production: Apply editing and enhancement techniques to the registered data. Examples of Point Cloud Filters can be found in the point cloud library (PCL) [3]. AI can also be used to reach a better level of enhancement.
4. Fusion: Integrate the registered data sources to create a cohesive volumetric scene. Transitioning from simple point cloud fusion algorithms to more advanced methods. [4]
5. Display: Present the volumetric content to users through suitable display technologies. Taking advantage of new display methods such as Neural Radiance Fields (NeRF) or Gaussian splats for real-time hologram display. [5]
6. Export: Prepare the processed data for use in various applications or platforms. Implementing compression formats for meshes and textures sequences. [6]
7. Live Streaming: Enable real-time transmission of volumetric content for immediate viewing or interaction. Improving the encoding of volumetric data for live streaming and storage. [7]
The R&D position involves at least one of these steps based on the doctor's interests and specific skills. Each point has already been studied in the state of the art, and various approaches tailored to specific use cases have been experimented with in the past, both by teams at French Touch Factory or its competitors worldwide, as well as by scientists.
The candidate with a PhD will conduct their research in close collaboration with the teams at French Touch Factory, applying it to real-world cases in sectors such as narrative XR, live performance, video gaming, digital training in crafts, industry, and healthcare – many of which involve partners of ITHACA.
The goal is to test and enhance approaches already identified in the state of the art, ensuring they meet genuine business needs and specific use cases.
3. Requirements
• PhD in Computer Science, Virtual or Augmented Reality, Spatial Computing or similar relevant fields
• Strong knowledge of programming languages, C/C++, C# on Ubuntu environment and Windows, Python, Pytorch
• Motivation for working in a multidisciplinary research project at the crossroad between optimization, computer science and 3D user interfaces
• A good level in English language (written and spoken)
• Additional knowledge in the following areas would be appreciated: game engine programming (Unity, Unreal), machine learning, shaders, conducting user studies, UX design
4. Hiring conditions
• 24-month full-time post-doctoral contract
• The successful candidate will be offered a salary commensurate to CV
• Start date: Between April 2024 and January 1st 2025
• Location : Paris&Co startup incubator, 157 boulevard Macdonald, Paris 75019
Other advantages:
• Telecommuting up to 2 days a week
• 30 days of vacation from the first year of collaboration
• Refill of 75% of the subscription to the public transport
• Stimulating innovation ecosystem (startups, students, research, companies)
5. How to apply
The position is currently open, we welcome applications as of now, until filled, and no later than May 15th 2024. Candidates are required to send their applications including a CV with a publication list, a brief motivation letter explaining interests and qualifications regarding the position, prior research/work experience - all within a single pdf file - and contact information of two professional references to f.bouille@frenchtouchfactory.com. Feel free to contact us for any questions regarding the position.
References
[1] https://shop.orbbec3d.com/Femto-Mega
[2] Colored Point Clouds Registration by Park 2017: https://openaccess.thecvf.com/content_ICCV_2017/papers/Park_Colored_Point_Cloud_ICCV_2017_paper.pdf
RoCNet: 3D Robust Registration of Point-Clouds using Deep Learning : https://arxiv.org/pdf/2303.07963
[3] Point Cloud Library filtering: https://pointclouds.org/documentation/group__filters.html
[4] Function 4D approach: http://www.liuyebin.com/Function4D/assets/Function4D.pdf
[5] Dynamic Gaussian splats: https://dynamic3dgaussians.github.io/?fbclid=IwAR0fTRPm5qYrgO77THyMZximTZwGqpZBpsktL1-jkk4lM6HUx9ueOYeWm1w
Bullet time with NeRF: https://labs.laan.com/blogs/slow-motion-bullet-time-with-nerf-neural-radiance-fields/
[6] WaveLet-based mesh sequences compression: https://webusers.i3s.unice.fr/~fpayan/data/publications/Payan_ICMI_2005.pdf
[7] Mpeg Immersive Video - MIV: https://mpeg-miv.org/
Contact : François Bouille, R&D Director at French Touch Factory, f.bouille@frenchtouchfactory.com, +33 663 039 629
Poste Doctorant F/H Immersive and Situated Visualizations of Personal Data
Informations généralesThème/Domaine : Interaction et visualisationInstrumentation et...
Thèse : « «Dynamique d’interactions tactiles et cognition sociale » à l’UTC
Poursuivez ici selon votre inspiration...Type de financement : Demi-bourse région + demi-bourse...
Internship on Human Robot Interaction evaluation in Virtual Reality
Keywords:Human Machine Interaction, Human Robot Collaboration, Virtual Reality, Virtual Reality...
Internship on Intelligent Tutoring System in Virtual Reality
Keywords: Virtual Reality Training System (VRTS), Data collection and analysis, Machine...
Postdoctoral researcher / Research engineer (AR/VR)
Keywords: Human Machine Interaction, Virtual Reality, Virtual Reality Training Systems (VRTS),...
Thèse: Prototypage rapide d’interactions en RA/RV
Mots-clés : Interaction homme – machine (IHM), réalités virtuelle et augmentée (RV&A), génie...