COSMO stands for COgnitive Support in Manufacturing Operations, and was a two-year imec.icon-project that brought together partners from research (itec, IPI, Flanders Make and ID Lab), industry (Mariasteen, Rhinox, CNHi and Azumuta) and education (GO! Technisch Atheneum Keerbergen, GO! Eureka Torhout, Dominiek Savio Gits and Hoge Kouter Kortrijk). COSMO was co-funded by the Flemish Government through imec (the Flemish strategic research centre for nanoelectronics and digital technologies) and Flanders Innovation & Entrepreneurship.
In the COSMO-project, (spatial) Augmented Reality as well as Virtual Reality were used to assist students in performing assembly tasks by providing cognitive support. By developing and investigating personalized AR and VR instructions, we tried to figure out the full learning potential of these technologies.
We had two experimental phases. During the first phase, the goal was the development and investigation of different support levels to guide operators through an assembly task.
During the second phase, a personalised system was developed in which the support level was adapted to the ability of the user. We created an AI-system (VR) and a rule-based algorithm (AR), that use previous assembly actions as input (such as errors, assembly time, …) to estimate the user’s ability. The impact of adaptivity on the objective task performance (such as errors, assembly time and help inquiries) and on the subjective well-being (such as stress, frustration, mental effort, perceived complexity) was investigated.
During this investigation we aim to answers the following research questions
- How can we automatically personalize the learning experience when training assembly tasks in AR and VR?
- Does automatically adapting instructional support have a beneficial impact on performance and well-being?
Augmented Reality, an inclusive technology
Guided by Augmented Reality, students performed short (+/- 5-10 minutes) assembly tasks such as building a wooden birdhouse, assembling a furniture color sample, and assembling and filling a teabox. In our COSMO-project, Augmented Reality was specifically used to support operators and students with special needs. The system provided efficient and detailed instructional support. A video-projector highlighted the part to pick and presented instructions on the worktable. A depth-camera detected correct and incorrect actions and offered immediate feedback to the user.
Virtual Reality
In manufacturing settings, the training of assembly operators to perform new and complex tasks is costly and time consuming. Virtual Reality (VR) enables a wide variety of low-cost and remote pre-training in a safe way.
Supported by instructions in VR, operators and students got to know assembly tools, parts and workflow during our research project. Students built parts of an agriculture vehicle, for example, where they virtually assembled a motor block and a fuel tank that took about 5 to 10 minutes to complete.
Adaptivity: algorithm and experimental design during the second phase
Three support levels were used with AR and VR: High support, Medium support and Low support. Participants performed tasks with adaptive support or with non-adaptive support, so that we would be able to compare both types of support.
Adaptive support was offered on task-level (the same support during the whole task) or on step-level (where support could vary in between steps of the same task). During non-adaptive tasks, task-level support was pre-set.
In high support, users were offered different forms of guidance at the same time:
During AR-tasks, users got pictures of the part they needed to pick or they got pictures of how they needed to assemble parts. Also, while picking a part, a flashing light was guiding students towards the right object(s). Next to that, written instructions were projected on the working table.
In VR-tasks, students got spoken instructions, they got a virtual compass that showed them the right direction towards parts and the part that needed to be picked also lighted up. During assembly steps, arrows and ghosting supported the user.
When lowering the level to Medium and to Low after that, less guidance was given.
Augmented Reality
The AR-adaptivity algorithm was rule-based, automated support selection was based on previous behavior: picking errors and help inquiries.
There were three conditions:
- Without adaptivity
- Task-level adaptivity
- Step-level adaptivity
The participants were students from special needs schools (N = 50) who performed three assembly tasks (building a birdhouse, folding a carton box and filling it with tea & assembling furniture samples), four repetitions per task.
Virtual Reality
The VR-adaptivity algorithm was AI-based by means of a recommender system in which automated support selection was based on users’ previous assembly times.
Two conditions were used:
- Without adaptivity
- Step-level adaptivity
The participants were students from technical and professional secondary education schools (N = 40), who performed three assembly tasks (assembling a fuel tank, a motor block and a handrail), four repetitions per task.
Results
Augmented Reality
No clear evidence was found to say that the adaptive version was better than the non-adaptive version. Seemingly, the pace at which the non-adaptive version decreased the instructional support was already quite good. We must take into account that there weren’t that many participants in phase II to really find differences and that the tasks were perhaps not complex enough to really make the adaptive version stand out. Results from earlier research (Ginns, 2006, Vanneste et al., 2020) indicate that an adaptive version comes out stronger when the tasks are more complex. We would suggest to use adaptivity for complex tasks that students find difficult to complete.
The gradual drop of instructions could still be positive for most students. When students get more details (higher level of support), generally they found the task somewhat less complex. Therefore, we consider this approach still as promising to foster learning.
Most of the students declared that they were satisfied with the instructions and they enjoyed working with Augmented Reality. For a few (more difficult) steps, they felt they could have used a higher support level.
Virtual Reality
In the non-adaptive version, the level of support faded out in the same way, regardless of the performance of the student. In the adaptive version, the individual performance of the student was taken into account in an aim to provide tailored support. However, when a step was still sensed as too hard, a help-button could be used to increase the support level. Results indicated that students completed a task somewhat faster in the adaptive version and results also showed that adaptive version also required a little less mental effort from the students. However, the differences between the adaptive and the non-adaptive version were still small.
We need to take into account that the experiments were taken with a rather small, heterogeny group. Also, the non-adaptive version was already ‘smart’ because it decreased the support step by step, which was positive for most students.
The adaptive version will definitely offer advantages for (very) fast or (very) slow workers since they are in need of tailored support.
Furthermore, results from interviews indicated that the majority of students was positive or rather positive about the VR training. Participants mentioned a variety of advantages: VR is simple, safe, it’s fun to learn while being active, you can train a wide variety of technical skills, it requires a low physical effort, etc. One third of the participants thinks VR is rather a complement to a real training but cannot replace it, since mainly the physical aspect (weight of objects, handling tooling, etc.) cannot really be simulated in the virtual world.
Are you interested in more results or do you want more information about the project. Please check out http://cosmo.kuleuven-kulak.be or https://www.imec-int.com/en/what-we-offer/research-portfolio/cosmo
References
Ginns, P. (2006). Integrating information: a meta-analysis of the spatial contiguity and temporal contiguity effects. Learning and Instruction, 16(6), 511–525. https://doi.org/10.1016/j.learninstruc.2006.10.001
Vanneste, P., Huang, Y., Park, J., Cornillie, F., Decloedt, B., & Van Den Noortgate, W. (2020). Cognitive support for assembly operations by means of augmented reality: an exploratory study. International Journal of Human-computer Studies, 143, November 2020. https://doi.org/10.1016/j.ijhcs.2020.102480
