Projects / research topics

Audio Based Motion Solver For Live Performances

Xsens & Metropool

With the continues digitization of the entertainment industry in combination with the recent Corona crisis, live venues are more and more pressured the explore new technologies to bring entertainment to people's homes. For this purpose all kinds of technology is available, including motion capture, VR and live streaming. However one of the problems that arises when doing live motion capture with musicians is that of correct finger and hand placements on instruments. To solve this problem the concert venue Metropool and Xsens would like your help by exploring similar technology as that of Jali research (http://jaliresearch.com). Based on a 2016 Siggraph paper this start-up utilized speech audio to synthesize blend shapes for mouth movements. Our questions would be if you could utilize these methods for musical instruments as well.

Research Goals

Research and develop a prototype application the can take audio of a music instrument (e.g. guitar, drums, etc.) and convert it to realistic and precise motions in 3D. Combine your findings with live motion capture from a Xsens suits to create a believable digital live performance experience.

Guidance

During the process you will be guided by experts from Saxion, Metropool and Xsens. You will work in the Saxion XR Lab in the Saxion Alphons Ariëns building (Ariënsplein).

Topics

Virtual reality, live performance industry, artificial intelligence, motion capture, animation, game development.

Links

Historical Storytelling Enschede

1Twente & Jan Astrego

Enschede's recent history is one of industrialization and deindustrialization. Due to rising and disappearance of these factories, there is a lot of hidden history, both positive and negative. On one hand there was extreme poverty, child labor and environmental damages. On the other hand did it bring growth, jobs and new technological developments. Although the industrialization was significant for how we view Enschede today, much of its industrial past has been demolished and forgotten, with only a few buildings still testament to its past. Here is where your help is requested. 1Twente, a local news organization, and Jan Astrego, who leads an initiative to bring the historical streetcar back in Enschede, would like to explore the possibility for series of short animations/films about the history of the city.

Research Goals

Within this project we would like you to research and create a concept for a series of short animations/films about the industrial past of Enschede. For this concept then create two productions/trailers. One should focus on De Krim neighbourhood, considered one of the first Dutch slums. The second production should be about the disappeared Enschede streetcar/tram which was one of the technical innovations brought to Enschede due to the textile manufacturing.

For your this project you will have to look into:

1. A suitable target group for the concept
2. Defining a good and fitting media format and tone of voice
3. A suitable production method and visual style

Project results

The outcome of your project should be:

1. A media concept for a series of animations/film productions around Enschede's historical past.
2. A short animation/movie about the Enschede streetcar.
3. A short animation/movie about poverty in the disappeared Enschede neighborhood De Krim.

Guidance

During the process you will be guided by experts from Saxion, 1Twente and Jan Astrego. You will work in the Saxion XR Lab in the Saxion Alphons Ariëns building (Ariënsplein).

Topics

Animation, film, VFX, visual storytelling, history.

Links

Nuanced Lessons of History

Nederlands Openluchtmuseum

The Nederlands Openluchtmuseum (Dutch Open Air Museum) in Arnhem is the place to visit when you want to learn about how the Dutch people lived in recent history. In the museum park you can visit historic houses, see old trades being practiced and get a general impression how people used to live.

Most of the museums visitors consist of groups of people, usually families with children. When visiting the park these visitors are informed through actors/employees and written signs. While this conveys basic information, a lot of people leave the park with a somewhat skewed and romantic idea of the past where people lived more slowly, purely, and in better contact with the surroundings. Although some of this might be true, live in those times was usually also much harder, with problems like bad healthcare, child labour, low quality and cramped housing, and extreme poverty. In order to give visitors a more nuanced view of the past the Nederlands Openluchtmuseum would like to request your help by researching and prototyping a next generation of immersive experience in the museum park.

Research Goals

Create and (interactive) immersive experience that let groups of museum visitor experience how harsh life could be in past times. Here you can take one or more existing buildings in the museum park as a place for you concept. When you have a good concept designed, you should build a working prototype that can be tested in the park. For your this project you will have to look into:

1. Research and the current state of immersive technologies for museums and theme parks
2. Research how people lived in the past. The up- and downsides
3. Research the current attractions in the museum park on how a new concept could fit it
4. Create a compelling technological concept where people can immersive themselves in how life used to be.
5. Build and test a prototypes in the museum park (if possible under future Covid restrictions)

Guidance

During the process you will be guided by experts from Saxion and the Nederlands Openluchtmuseum. You will work in the Saxion XR Lab in the Saxion Alphons Ariëns building (Ariënsplein).

Topics

Museum, education, interactive experiences.

Links

Realistic Plant Simulation

Mechatronica

Simulation can significantly speed up the development of new robots. Simulation can for example be used to test hardware that is physically not available yet. It can also be used to generate sensor data that can be used during development. In this project we primarily focus on generating sensor data.

Eventually, we want to test agricultural robots in simulation. The robots need to recognize plants and pull them out of the ground or harvest them. In this project we focus on the first aspect: having a realistic simulation of plants that move and look realistically. Ideally, these simulations can run in real-time with 100s of plants. In parallel, we need to explore how to combine the simulation in Unity with the robotic simulation in ROS 2 and Gazebo.

Research Goals

At the end of this project we would like to see a field of crops waving in the wind. The simulation can (virtually) be observed by a camera and a lidar that is used in ROS 2. Some required functionalities for the prototype are:

1. Model of plants that looks realistically, but can also run in real-time for a field of crops
2. Interfacing with virtual sensors, so that the simulation in Unity can be used in a robot simulation with ROS and Gazebo.

Tips: For this project it is worthwile to see if you can use the NVIDIA Omniverse enviroment.

Guidance

During the process you will be guided by experts from Saxion and Aeres hogeschool. You will work in the Saxion XR Lab in the Saxion Alphons Ariëns building (Ariënsplein).

Topics

Realistic simulation, plants, crops, agriculture, robotic simulation, Unity, ROS 2, Gazebo

Links

Unified Interaction Methods for Virtual Reality

The Virtual Dutch Men

Since several years, virtual reality (VR) is being used more and more as a communication platform where users can meet, work and train in a virtual world. Part of this development is the request for more complex and interactive tooling that allows the layman to control and build their virtual environments in VR.

The development of these tool pose their own problem when it comes to interaction. Due to imperfect hand tracking natural interactions are not possible. Combine this with the need for many interaction options, and lots of information, and a variety of hardware input devices, one can very quickly come to the conclusion that new insight into interaction methods for VR are needed.

Research Goals

Research and design a unified interaction framework for all kinds of interactions in VR that allow users to create and build complex 3D worlds.

Within this project you will have to:

1. Find and analyse existing applications and papers on their VR interaction methods.
2. Brainstorm a test different unified interaction methods that can be used in VR. Possibly combining controllers and hand tracking.
3. Build a prototype application for one platform of choice where you test your findings.

Guidance

During the process you will be guided by experts from Saxion and The Virtual Dutch Men. You will work in the Saxion XR Lab in the Saxion Alphons Ariëns building (Ariënsplein).

Topics

VR, interaction design, user experience.

Links

Virtual Reality Operating Room

Leiden University Medical Center (LUMC)

In medical setting doctors encounter various clinical situations where not only technical skills have to be mastered, also decision making, communication skills and clinical medical reason. Especially for anesthesiologists; crisis resource management and patient safety are fundamental, and simulation training allows to bridge the gap between theory and practice. In the operation room anesthesiologists may encounter live threatening situations where they have to act fast without much room for error. In a stressful situation the brain doesn't always function the way we want it too, therefore it's essential the simulation training feels ‘real' enough to create a completely immersive and realistic scenario.

Currently training is done in various ways, including a complex and expensive mechanical human manikin used to scenario training. Giving the limiting visuals of the manikin and limited accessibility (party due to the covid pandemic) the Leiden University Medical Center would like to explore how VR could be used as an accessible and visual realistic addition to the training of anesthesiology residents.

Research Goals

Within this project we would like you to research and build a virtual reality operating room including a prototype digital human to operate on. You will have to research the needed functionality as well as technologies to create a visual and physical believable simulation. It's very important a trainee feels as if he was actually exposed to a potential dangerous situation.

Some required functionalities for the prototype are:

1. Visual realistic operating room and digital human (patient).
2. Realistic soft body physics and fluids
3. Correct auditive and visual feedback based on decisions of the trainee
4. Multi-user training capabilities (networking)
5. Interactive physiology of the patient adaptable by the trainer.
6. Possibility to perform a list of technical skills
7. Possible face tracking for multi-user interactions.

Guidance

During the process you will be guided by experts from Saxion and the Leiden University Medical Center. You will work in the Saxion XR Lab in the Saxion Alphons Ariëns building (Ariënsplein).

Topics

Virtual reality, training applications, photo realism, digital humans, networking, game development, real-time soft-bodies and fluids.

Links

Virtual Reality Weapons Room

Royal Dutch Army

One of the benefits of virtual reality (VR) is that it allows for training of situations that are difficult, expensive or even impossible to train in real-life. For this purpose the education and training affairs section of the Royal Netherlands Army is looking into utilizing this technology for various training situations. One of these is that of training of logistical personal in the setup and construction of weapon rooms. For this purpose we would like to ask for your help.

Research Goals

Within this project we would like you to research and build virtual reality experience where you can train how to configure and run a realistic weapons room as it would be used by the military. Within this assignment you will have to research what kind of functionality is needed in order to prepare logistical personal for the setup of weapons rooms, here in the Netherlands as well as abroad on missions.

Some of the prerequisites for this project are:

1. The application should contain different scenarios that require different kind of preparation in logistics and inventory.
2. A trainee should be able to spatially lay-out equipment in a weapons room following the right procedures and rules.
3. The application should be near photorealistic and involve location based spatial VR (no teleporting).

Guidance

During the process you will be guided by experts from Saxion and the Royal Dutch Army. You will work in the Saxion XR Lab in the Saxion Alphons Ariëns building (Ariënsplein).

Topics

Virtual reality, training applications, photo realism, 3D modelling and materials, game development.

Virtual Reality Storm Project

Borculo Weather Museum

Extreme weather is potentially a frightening and dangerous experience often associated with the tropics and countries around the equator. However sometimes these events happen in more moderate weather systems as well. Such was the case in 1925 in the small town of Borculo, close to Enschede. A huge typhoon devasted most of the town, collapsing the church and causing wide spread destruction to houses and the environment. Nowadays, with the changing of weather systems globally, extreme weather will probably occur more often, also in places which usually experience more moderate weather. In order to create awareness about the potential impact of extreme weather the new to be created Borculo weather museum would like to request your help by creating a VR storm experience around the 1925 Borculo storm disaster.

Research Goals

Within this project we would like you to research and create a virtual reality storm experience around the Borculo storm disaster of 1925. Here you will have to look at:

For your this project you will have to look into

1. How to realistically create a believable storm in VR which feels frightening and dangerous
2. How to use hyper VR elements to create a more immersive experience (overlaying the virtual of the real world)
3. How to improve the existing VR environment to create more immersion and believability
4. How to use storytelling to add immersion

For the experience you can use the already existing prototype made XR lab interns as a base to expend upon.

Guidance

During the process you will be guided by experts from Saxion. You will work in the Saxion XR Lab in the Saxion Alphons Ariëns building (Ariënsplein).

Topics

Virtual reality, destruction simulations, environmental design, hyper reality, hardware.

Links

Cyber Security Game

ABN AMRO

Since the introduction of the internet more and more activities have moved online. This is also includes banking and other kind of important administrative tasks. While moving these activities online saves a lot of time, money and can make processes more accessible; it also makes them more prone to misuse and cybercrime. One methods of reducing the damages of these activities is through educating people about the safety concerns while being online. Exactly for this reason the ABN AMRO has requested the help of Saxion. Last semester a group of students did research, and designed a potential cyber security game, that could spread more awareness around this topic. However the game was not digitally prototyped and tested. To continue the work and to get a gaming perspective on this topics your help is requested.

Research Goals

For this project you will have to take the previous work done for the Cyber Security Game and convert to a fun, engaging and visual compelling game. Here you will have to look into:

1. What visuals fit with the target audience and create them accordingly.
2. How to current gameplay can de converted to a game and what you can add to make it ever more negaging.
3. How the game could be expanded with new levels or ideas.
4. Which additional spin-offs ideas can contribute to more awareness around cyber security.

Guidance

During the process you will be guided by experts from Saxion and ABN AMRO. You will work in the Saxion XR Lab in the Saxion Alphons Ariëns building (Ariënsplein).

Topics

Virtual reality, optimization, interface and interaction design, modeling, texturing, animation, sound

Virtual Reality Nursing Trainer

Lectorates Ambient Intelligence and Technology Health and Care, Solis, Tendr & The Virtual Dutchmen

In today's Corona times, nurses have become our new superheroes. Just like normal people, superheroes need training. However, the current nursing skill training means are non-optimal. With the use of virtual reality, several organizations (Zorggroep Solis, Medisch Spectrum Twente and Saxion HBO-v) want to enable nurses to train hands-on anywhere, anytime in a fun and educationally powerful way. A concrete lists of steps the nurse has to go through for the insertion of a tube through the nose into the stomach has already been worked out and an initial proof of concept has been developed. However, the current application is not yet suitable for training nurses. For this we need you to help professionalize this VR training! We are looking for artists, designers and engineers to make it suitable for use in actual training practice.

Research Goals

During the project it is important to evaluate designs and implementations with the care/training organizations (Saxion, Solis, MST) and VR professionals (Tendr, TVDM) to make sure the end result is indeed fit for training nurses. The main challenge is to make it realistic enough for training purposes and keep it engaging, yet also easy to use for people with little experience with games and VR in particular. We ask you to look into:

1. How to design/improve the interactions and interface for inexperienced VR users .
2. How to design and animate the various objects in a realistic, recognizable way for the nurses
3. How to optimize the application to render smoothly on a standalone VR headset (Oculus Quest 2)
4. How to optimize the scenario and user experience for training
5. How to apply spatial sound in VR to enhance realism, immersion and feedback

Guidance

During the process you will be guided by experts from Saxion Lectorate Ambient Intelligence. There will also be access to and feedback the other partners in the project on the domains of healthcare and VR. You will work in the Saxion XR Lab in the Saxion Alphons Ariëns building (Ariënsplein).

Topics

Virtual reality, optimization, interface and interaction design, modeling, texturing, animation, sound