Pointman (user interface)

Pointman is a seated user interface for controlling one's avatar in a 3D virtual environment. It combines head tracking, a gamepad, and sliding foot pedals to provide positional control over many aspects of the avatar's posture.^[1][2] Pointman was developed by the US Naval Research Laboratory (NRL) to support the use of dismounted infantry simulation for USMC training and mission rehearsal.^[1][2][3] NRL's goal in developing Pointman was to extend the range and precision of actions supported by virtual simulators, to better represent what infantrymen can do.^[4]

Pointman seeks to enhance the level of control provided by conventional desktop and console game controllers by engaging the user's whole body to control corresponding segments of the avatar's body.^[1][2][3][5] The user employs his head and upper body to control looking and aiming, as well as leaning to duck and peek around cover. He uses his hands to operate virtual weapons and direct tactical movement, and he uses his feet for stepping and controlling his avatar's postural height.^[1][2]

Pointman uses a set of three consumer grade input devices: a Natural Point TrackIR 5 head tracker, a Sony DualShock 3 gamepad, and a pair of flight simulator foot pedals from CH Products.^[1] The additional input from the head and feet offloads the hands from having to control the entire avatar and allows for a more natural assignment of control. Together, the three input devices offer twelve independent channels of control over the avatar's posture.^[4]

The head tracker registers the translation and rotation of the user's head. Pointman uses these inputs to map the movements of the user's head and torso one-for-one to those of his avatar.^[1][2] The virtual view changes as the user turns his head to look around or leans his torso in any direction. When the weapon is raised into an aim position its sights remain centered in the field of view, so that turning the head also adjusts the aim.^[1] The user can aim as precisely as he can hold his head on target.^[2][6] Hunching the head down by flexing the spine is also registered by the head's translation, and the avatar adopts a matching posture. Leaning forward and hunching are used to duck behind cover. Rising up and leaning to the side are used to look out and shoot from behind cover.^[1][2]

The gamepad includes dual thumb sticks and a pair of tilt sensors. Pointman uses the thumb sticks to turn the avatar's body and set the stepping direction.^[1][2][3] The tilt of the gamepad is mapped to control how the virtual rifle is held. The user tilts the gamepad down to lower the rifle, and tilts the gamepad up to continuously raise the rifle up through a low ready into an aim and then to a high ready.^[1][2] This allows users to practice muzzle discipline, by lowering the rifle to avoid muzzle sweeping friendlies, minimizing collisions when moving through tight spaces, or leading with the rifle when moving around cover.^[2][6] Once the rifle is raised into an aim position, the user's head motion aligns the sight picture.^[1][2] The user rolls the gamepad (tilting it side to side) to cant the weapon.^[1] Gamepad buttons are mapped to control various weapon operations (including firing and reloading) and aiming functions (such as the optic zoom level).^[7] The foot pedals slide back and forth and also move up and down like accelerator pedals. Pointman uses these inputs to control the avatar's lower body.^[1][2][3] The translational sliding (apart then together) is mapped to control the separation of the avatar's legs, simulating stepping when the avatar is upright and crawling when the avatar is prone.^[2] This allows users to take precise, measured steps when moving around obstacles or cover, and to continuously vary their speed over a realistic range of walking, running and crawling gaits.^[1][2][6] The up-down movement of the pedals is mapped to control the avatar's postural height via the flexing of the avatar's legs. This allows the avatar to continuously transition from standing tall to a low crouch (or kneel when the legs are apart), and when prone from hands-and-knees to belly-on-the-ground.^[1][2] The ability to precisely control their avatar's postural height allows users to make better use of cover and concealment, and to look and shoot out from behind cover while minimizing their exposure.^[2][3][6]

The Virtual Battlespace combined arms simulators (VBS2, VBS3, and their successor VBS4), from Bohemia Interactive Simulations (BIS), are used for training by the USMC, the US Army, and a number of the NATO armed forces. BIS worked closely with NRL to tightly integrate the Pointman interface with VBS, and to allow Pointman to control the posture of the user's avatar on a continuous basis.^[1][2] The detailed articulation of the user's avatar is made visible to other squad members running in a networked simulation. Pointman-enhanced VBS (VBS-Pointman) supports the operation of a wide range of small arms and additional forms of mobility, including climbing, swimming, and mounted roles (driver, passenger and gunner) using the full complement of manned vehicles.^[8]

Pointman was designed by Dr. Jim Templeman of NRL and implemented by Patricia Denbrook of DCS, Inc. NRL's Base Funding Program supported the initial development of Pointman, and the USMC Program Manager Training Systems (PM_TRASYS) together with the Office of Naval Research (ONR) Rapid Technology Transition program office supported its integration with VBS2. ONR's Human Performance, Training, and Education Thrust Area added its support in refining and demonstrating Pointman.^[1]

A formal Military Utility Assessment (MUA) of Pointman integrated with VBS2 was performed by the MarForPac Experimentation Center at MCB Hawaii in September 2011.^[1][2][5][6] The squad of Marines that participated in the study (Golf Company, 2nd Battalion, 34d Marine Regiment) gave Pointman high marks for realism and usability. In response to a series of survey questions, the Marines felt Pointman allowed them to realistically: control viewing, perform tactical movements, control the virtual rifle, utilize cover, and control the avatar’s posture. They found it comfortable, easy to use, and that it enhanced the simulation. The primary recommendation of the MUA report was: “Transition the Pointman DISI (dismounted infantry simulation interface) enhancements into VBS2 to increase realism and efficacy as a virtual training aid.”^[6]

NRL is continuing^[when?] to develop the Pointman interface as part of its ongoing research in expressive interaction for desktop simulation. This involves extending Pointman to include non-verbal communications (such as eye movements, facial expression, and arm gestures) needed to support team and cross-cultural interaction, without limiting tactical mobility.^[4] A driving application is the training of cultural interaction skills alongside warfighting skills, using training scenarios which pose a mix of tactical, cultural and ethical challenges.^[9]

• Pointman Demonstration Video
• Enhancing Realism in Desktop Interfaces for Dismounted Infantry Simulation
• User Interfaces for Dismounted Infantry Simulation
• VBS-Pointman Quick Reference Cards
• VBS-Pointman Training Guide
• VBS-Pointman API
• CH Products Pro Pedals
• NRL Immersive Simulation Lab (ISL)
• VRWiki Walking Simulators
• Pointman in BIS Showcase