VR ROBOT
TELEOPERATION

Collaborators

Robotics (RPAD) Lab
Lucas Ochoa
Tiffany Chen
Mary Safy
2 others

Project Space

2019: Our client was the Robots Perceiving and Doing Lab at Carnegie Mellon University. Their long term goal was to investigate the technologies (models, algorithms, interfaces) that would go towards a future where people could teach robots various tasks. Our short term goal in this project was to design and build a virtual reality (VR) environment where users control a remote robot to demonstrate how a task should be done. The interface would capture the "demonstration" data for later learnings.

My Role

Designer
Developer

The App

Tiffany and I coded out the Unity + websocket app for the Vive headset that interfaced with the lab's robot operating system. A Kinect (3D depth camera) above the physical robot station streamed a live point-cloud to the headset

Design Questions
Abridged Process

Storyboards drawn on my iPad, then compiled/dicussed with team in Figma over the course of literature review

  • trajectory_control
    Position or trajectory control? How granular? How explicit? What sort of signal filtering?
  • Existing softwares all have the precedent of gimbal controls for precision movement.
  • Sketching in VR to plan physical setup
  • Trying and experiencing ideas in 3D and at scale
  • Direct manipulation for control engagement
  • If engaging remote control used the metaphor of car ignition...

A video version of a 1st person point of view in a VR experience with a moderator prompting tasks. Animated in Blender (by Mary Safy)

  • Trying out different shaders to visually convey "non physical" controls
  • Need a visual cue for whether an object is actually between the two claws (depth perception alone isn't enough)
  • Putting the prompt for the task (the white example table of stacked cubes) on the table was an option that got cut out
  • Even taking the prompt out of reaching distance wasn't enough to discourage users from trying to interact
  • Path drawing for the robot. We found a nice compromise between direct and indirect manipulation of the end effector