Interaction in Virtual Reality

"to afford" = "to offer" / "to make possible"

Relationship between the properties of an object and the capabilities of an agent (typically a human user).

They don’t have to be perceivable to exist.

Informs the user what is possible before they interact.

Examples: signs, labels, images, feel of a button on a controller, etc.

Misleading signifiers can be ambiguous or represent an affordance that doesn’t actually exist.

Can be unintentional but useful. For example, garbage on a beach represents unhealthy conditions.

Do not have to be attached to an object. Can signify general information, such as current interaction mode.

Signifiers should be used to make constraints easy to understand.

If communicated well, they can improve accuracy, precision, and user efficiency.

Should be consistent, so that learning can be transferred across tasks.

Experts should be able to disable constraints.

In VR, they are typically physical or mathematical, but in general can be also be semantic or even cultural.

Degrees of Freedom (DoF) — Number of independent dimensions available for motion of an entity.

Constraints can add realism (e.g., preventing users from clipping through walls) but can also make interactions more difficult.

Lack of immediate visual feedback when the user moves can result in break-in-presence or even motion sickness.

Haptic feedback is especially difficult to implement (e.g., real forces caused by walls) but sensory substitution can be used.

Should be prioritized based on importance to avoid overwhelming the user.

Instead of putting stuff on a heads-up display (or near the head), consider putting it near the waist. There, it can be easily accessed but isn’t obtrusive.

Users should be allowed to configure or disable it.

Ghosting — Second rendering of an object in a different pose.

Highlighting — Visually outlining an object.

Audio cues — For example, sounds that signify collision.

Passive haptics — Static objects IRL that can be touched.

Rumble — Vibration of input devices.

Mappings are useful even when the thing cannot be manipulated with directly (e.g., using a broom to flip a switch).

Input devices have a mapping natural for some interaction pattern but poor for another.

Non-spatial mappings transform spatial input to non-spatial and vice versa.

Spatial compliance — Direct mappings in space lead immediately to understanding.

Temporal compliance — Different types of feedback (e.g., proprioceptive and visual) to the same action should be synced.

Current mode can be changed by pressing a button, selecting it in a UI panel, or it may be order-dependent (i.e., activate automatically based on previous actions).

The current mode should always be made clear to the user.

Example: An object should be selected first, before an action can be done upon it.

Sequence of interactions should occur without distractions.

Users should not have to physically (e.g., eyes, hands, head) or cognitively move between tasks.

Lightweight mode switching, physical props, and multimodal techniques help maintain flow.

In natural language, action-object is more common (e.g., "Pick up the book.")

Object is concrete an easier to think about.

Action is abstract. It’s easier to imagine the action with an object already in mind.

It’s better the use a specialized modality when it’s clearly the best (e.g., selection by pointing for your app).

When user preference is divided, it may be better to implement both options.

When selection needs to be (somewhat) realistic.

Limited by what the user can physically reach. They may need to travel to the object first.

Realistic hands — As close to reality as possible. Arm pose may be computed using inverse kinematics. Measuring the user’s arms and adjustments to the environment may be necessary.

Non-realistic hands — Focus on usability rather than realism. Often floating hands without arms or abstract cursors. Reduce visual occlusion.

Go-go technique — When extended beyond 2/3 of reach, the arm grows non-linearly, allowing to pick up far-away objects.

One of the most fundamental and used selection patterns.

Typically more effective than hand selection unless realism is required. When selecting objects beyond personal space, it’s faster and more precise.

Naive implementation can be imprecise due to hand tremor.

It can also be imprecise of the hardware of the input device is imprecise or loses tracking.

Dwell selection (pointing for a period of time) can be frustrating (especially with eye gaze). Don’t use unless necessary.

Shooting rays from hands is actually quite natural because of laser pointers.

Hand pointing — Ray from a hand/finger.

Head pointing — Ray from the center of vision.

Eye gaze selection — Selecting by looking with the eyes. Problematic because people expect to be able to look at stuff without it triggering something (Midas touch problem).

Object snapping — Fixes hand tremor imprecision by snapping/bending the ray towards the closest/most important object.

Precision mode pointing — "Slowed down" cursor. Can be combined with a zoom lense.

Two-handed pointing — Ray starts from the hand closer to the body and extends through the farther hand.

Simulates touch at long distances.

Works well at any distance as long as the object can be seen.

Don’t use when selection is frequent, because of gorilla arm.

Works for a single eye, so users should close the other.

The hands occlude the scene unless transparent.

Head crusher — Users positions their thumb and forefinger around the desired object.

Sticky finger — The object underneath the user’s finger gets selected.

Lifting palm — The user positions their palm so that it looks like the object is lying on it.

Framing hands — Using both hands, the user makes a frame around the object.

Good for selecting data when there are no geometrical surfaces (e.g., CT datasets).

Great for making selections inside point clouds, voxels, subsets of surfaces, or even empty space.

Can be difficult than selecting a single object.

Cone-casting flashlight — Pointing but with a cone instead of a ray. Aperture is a modification that allows the user to specify the size of the cone by bringinng the closer or further.

Two-handed box — Using both hands, the user positions and orients a box with snapping and nudging. Snap brings the box to the hand and allows setting its initial pose. Nudge enables precise adjustments; motion of the box after the initial position is locked to the hand; it can be reshaped with the other hand as well.21

More efficient and satisfying than other manipulation patterns.

Like hand selection, it is limited by the reach of the user.

For high realism, use in combination with hand selection.

Non-isomorphic rotations — Allow more rotation with smaller wrist movements. Reduces clutching.

Go-go technique — Just like with hand selection. It allows manipulation as well.

Useful for remote manipulation or when the object or the user themself is scaled.

Can be difficult if there is no directional compliance (e.g., when there is a rotational offset between the proxy and the remote object).

Use tracked physical props for tactile feedback and realism.

Tracked physical props — The user manipulates a physical object. Natural two-handed interactions and tactile feedback (passive haptics). Extremely easy to use without training.

Enhances manipulation capability of the hands.

Can take more effort if the user must first travel and maneuver to an appropriate angle to apply the tool.

Hand-held tools — Virtual objects attached to the hand. Can be used from afar (e.g., TV remote) or directly (e.g., a paintbrush). Often easier to use and more direct than widgets.

Object-attached tools — The tool is attached to the object being manipulated (not the hand). Signifier representing the affordance between the object, tool, and user (e.g., vertex handles in the corners, or a color icon that triggers a color cube tool). The signifier should make obvious what the tool does.

Jigs — Similar to real-world physical guides used by carpenters and machinists. Grids, rulers, other shapes that the user attaches to object vertices, edges, and faces. Allows customizable constraints and snapping. Jigs can be combined into jig kits.

Provides situational awareness, user-defined proxies, and quick movement.

Features a "doll", an avatar for the user that matches their movements. A transparent frustum may show what the user currently sees.

Use maps where forward is always up so that the orientation of the map matches the orientation of the world.

Voodoo doll — Combines WIM with image-plane selection. The user has a partial WIM in non-dominant hand and a proxy for the manipulated object in the dominant hand. By moving the proxy doll in relation to the world doll, the user can position and orient the manipulated world object. The world doll does not move.

Moving into one’s own avatar — The user can move their own doll in the WIM. Don’t map the user’s doll’s movement to the user’s perspective (causes motion sickness), teleport instead. Using multiple WIMs can act as a portal between worlds.

Viewbox — The user creates the WIM using volume selection. It acts as a reference: it and its insides can be manipulated. Make sure it’s not recursive.