EXPLORATION OF DEPTH PERCEPTION AND NON-PICTORIAL CUES

 

IN HAPTICALLY REINFORCED INTERACTION WITH 3D IMAGES

 

 

Leader:

Grigori Evreinov, Ph.D., Senior Researcher

Department of Computer Sciences

33014 University of Tampere

Finland

 

Suomen Akatemia grant 127774 (project completed)

1.01.2009-31.12.2012.

 

*        *        *

3D computer graphics was introduced by William Fetter in 1960. Nowadays we are able to manipulate virtual 3D objects in a much more flexible way than before. As soon as 3D Display technologies will become generally available they will present new challenges in human-computer interaction and interesting opportunities for applications. Further more, 3D interaction represents a fundamental aspect of the multimodal research.

Still, in many cases the visual system cannot integrate multiple visual features preattentively or/and in the absence of other perceptual information. Pictorial and non-pictorial cues may be dependent of the focus of interest, while the latter is usually stimulus driven and goal-directed. A specific way of interaction can modify the interpretation of visual information and the integration of visual cues with signals of other modalities (haptic and auditory). When the user interacts through stylus a sense of the planar surface significantly worsens 3D pictorial cues.

We aim to explore in detail a wide range of non-pictorial visual/haptic cues (objects, patterns and scenes) and their influence on 3D rendering in the context of the human-computer interaction. A performance of the user with novel approach, techniques and metaphors that can reinforce visual-haptic integration in mobile/desktop applications will be under investigation.

We need deeper knowledge on and understanding of cross-modal perception and signal processing. In particular, the following questions have to be answered:

How could 3D visual information be combined with haptic cues to optimize/facilitate the image/scene understanding?

How to avoid ambiguousness and misunderstanding of depth perception?

How to find the appropriate mapping between perceptual cues of two modalities (visual and haptic) and the data variables to be visualized?

In our approach the task model, the display state and the user input will manage the haptic output (proprioception sense) from the system as a complex function of the information content.

New perceptual models for haptically augmented interaction with 3D images will be explored. Understanding of perceptual mechanisms of the interference and integration of visual-haptic signals will contribute significantly to the development direct and intuitive paradigms of advanced interaction with the computer.

 

*        *        *

 

Challenges

3D computer graphics was introduced by William Fetter in 1960 [1]. However, direct interaction with 3D images for a long time was only considered as artifacts due to a poor usability of the techniques and special hardware needed e.g., polarized/shutter glasses [2, 9] and volumetric display [3] to support 3D visual rendering. In the later 80’s development of the head mounted and peripheral displays and virtual reality applications for scientific imaging, simulators and games had a great impact on realizing the potential of 3D technology in HCI.

Significant investments in Europe into 3D optical technology based on liquid crystal displays (LCD) were made by Philips and Sharp Laboratories. Besides TV and desktop monitors, the manufacturers were oriented on the fastest-growing segment of the market of different mobile devices (phones, PDAs, ultra-mobile pc, GPS personal navigators and other gadgets).

The Sharp Actius RD3D was the first 3D laptop computer, equipped with a 15 inch 1024x768 resolution 2D/3D switchable display. The Sharp Actius AL3DU is the latest 3D laptop having a similar display. The Sharp LL-151-3D is a 15-inch 1024x768 resolution 2D/3D switchable desktop monitor we have and will use in our further experiments. Recently Philips developed IC3D display signal processing chip, which should open the new possibility for 3D-vision on mobile. Samsung has developed AMOLED [3] display panels that will display 3D images on mobile phones. Nowadays, the world-leading manufacturers are members of 3D Consortium [5] and3D market will grow enormously in various fields such as medicine, education, museums, games, e-commerce etc.” [5].

There are also several models of mobile phones allowing 3D imaging. The first Sharp 3D product was the SH251iS mobile phone, featuring a 2D/3D switchable TFT display at 176x220 resolution. This product is becoming popular and was sold over 1.5 million units in the first 6 months. The second 3D product was the SH505i mobile phone with a continuous grain silicon (CGS) display at 240x320 resolution. Amex Digital (Hong Kong) announced their new GPS phone with 2.4-inch 3D display called G-Plus [6]. iPods/iPhones likely soon will be equipped with new 3D displays and the first 3D game (Corridor - 3D labyrinth) for the iPhone was already released by Johnson [10], [7].

Nowadays we are able to manipulate virtual 3D objects in a much more flexible way than before. Several virtual environments and augmented reality-based systems are successfully used in medicine for surgeons training and rehabilitation. For these applications, 3D interaction represents a fundamental aspect of the multimodal research.

However, perceptual deviations, which can take place in surgical simulation, could have negative consequences in clinical practice [11, 12]. Moreover, new paradigms are necessary for designers in order to enable new sense and visualization features for advanced human-computer interaction. As soon as 3D Display [8] technologies will become ubiquitous it will present new challenges in human-computer interaction. This 3D breakthrough display technology and coupled problems of cross-modal perception should be tackled in the advanced HCI research and specifically with mobile devices in mind [8].

 

Evreinova, T.V., Evreinov, G., Raisamo, R. An Exploration of Volumetric Data in Auditory Space. J. Audio Eng. Soc., Vol. 62, No. 3, 20 March 2014, pp. 172-187

The reduced behavior for the exploration of volumetric data based on the virtual sectioning concept was examined using three sonification techniques in the absence of visual feedback. Using a 3-alternative forced choice procedure, the subjects explored five experimental sets of the virtual top projection of volumetric shapes with different levels of complexity (gradientbased surface discontinuity). The subjects had to explore auditory space and identify the target - the modified shape's projection, which was different from two others (non-rotated and rotated images). An analysis of the performance of 12 subjects demonstrated that the strongly synchronized merged information of blended chords does not allow reliance on inherent mechanisms to focus on and retrieve separate components of an auditory image in the sound flow. The redundancy of free scanning in auditory space, even enhanced by a selfguided strategy, overloaded the listeners and did not allow the establishment of the right relations between successive sound samples separated in space and time. The high sensitivity of the cross-sectional scanning technique to modify and change the orientation of the virtual objects was achieved by optimizing the interaction with data in auditory space. The method was developed for the alternative visualization of spatially distributed data collected by sensors.

 

Evreinova, T.G., Evreinov, G., Raisamo, R. A Case Study on Non-Visual Pen-Based Interaction with the Numerical Data, Int. Journal of Human Computer Interaction (IJHCI), 2013, vol. 4, issue 2, pp. 70-87. http://cscjournals.org/csc/manuscript/Journals/IJHCI/volume4/Issue2/IJHCI-79.pdf

A widespread tabular format still poses great problems for screen readers because of diversity and complexity of the cells’ content. How to access the numerical data presented in tabular form in a quick and intuitive way in the absence of visual feedback? We have implemented and assessed the algorithm supporting an exploration of the tabular data in the absence of visual feedback. This algorithm helps to solve the most commonly encountered problems: retrieving the position of extreme values and the target value that can also be linked to the specific content of the virtual table. The performance of 11 blindfolded subjects was evaluated when they used the StickGrip kinesthetic display and when they relied on the Wacom pen and auditory signals. The results of the comparative study are reported.

 

Evreinova, T.V., Evreinov, G., Raisamo, R. Virtual Sectioning and Haptic Exploration of Volumetric Shapes in the Absence of Visual Feedback. Advances in Human-Computer Interaction, Hindawi, Volume 2013 (2013), Article ID 740324, 15 pages, http://dx.doi.org/10.1155/2013/740324 http://www.hindawi.com/journals/ahci/aip/740324/

The reduced behavior for exploration of volumetric data based on the virtual sectioning concept was compared with the free scanning at the use of the StickGrip linkage-free haptic device. Profiles of the virtual surface were simulated through the penholder displacements in relation to the pen tip of the stylus. One or two geometric shapes (cylinder, trapezoidal prism, ball, torus) or their halves and the ripple surface were explored in the absence of visual feedback. In the free scanning, the person physically moved the stylus. In the parallel scanning, cross-sectional profiles were generated automatically starting from the location indicated by the stylus. Analysis of the performance of 18 subjects demonstrated that the new haptic visualization and exploration technique allowed to create accurate mental images, to recognize and identify virtual shapes. The mean number of errors was about 2.5% in the free scanning mode, 1.9% and 1.5% in the parallel scanning mode at the playback velocity of 28 mm/s and 42 mm/s respectively. All the participants agreed that haptic visualization of the 3D virtual surface presented as the cross-sectional slices of the workspace was robust and easy to use. The method was developed for visualization of spatially distributed data collected by sensors.

 

Evreinova, T.V., Evreinov, G., Raisamo, R. Evaluation of Effectiveness of the Stickgrip Device for Detecting the Topographic Heights on Digital Maps. Int. Journal of Computer Science and Applications 2012, 9(3) 61 – 76. http://www.tmrfindia.org/ijcsa/v9i35.pdf

Multimodal visualization of landscapes and cityscapes requires new concepts and metaphors for intuitive operations with multidimensional data in fully immersive virtual environments. Variations in lighting conditions and perceptual interpretation of the reduced topographic colors can significantly modify an assessment of the true elevation profiles on a digital map. This paper describes the results of the empirical evaluation of the new interaction technique that has potential to enhance the imaging functionality of the two-dimensional maps. It was demonstrated that untrained subjects become much more accurate at detecting the altitudes in a range of 0-4000 m assigned within the palette when values of the light intensity had been associated with the haptic information. The results confirmed that the accuracy of the height estimation with the StickGrip haptic device appeared to be higher by about 32% in comparison to visual assessment.

 

Evreinova, T.V., Evreinov, G., Raisamo, R. Haptic Visualization of Bathymetric Data. Haptics Symposium HS2012, 4-7 March Vancouver, BC, Canada.

Visualization of water depth in geographical maps is limited by contour line density and by human ability to distinguish a subtle difference of the color gradient at a specific map scale. We were interested in whether it is it possible to increase the accuracy of subjective assessment of the bathymetric information coded by color intensity when visual observation would be complemented with haptic feedback presented as a function of the water depth. This paper describes the results of an evaluation of the new interaction technique that has potential to increase the estimation accuracy of color-coded information presented in a two-dimensional space of a topographic map. In particular, it was demonstrated that untrained subjects could accurately navigate between two geographic locations on the map of the lake by providing the necessary depth when values of the color intensity were associated with haptic feedback presented as a function of the lake floor. A comparative evaluation of the accuracy of navigation was carried out visually, using a regular mouse, and instrumentally with the StickGrip haptic device. The accuracy of navigation with the StickGrip haptic device appears to be higher by 14.25% to 23.5% in a range of bathymetric data of 40-140 m. We confirmed that a kinesthetic sense of distance to the surface of interaction (tablet) and self-perception of the finger joint-angle positions enhance the accuracy in distinguishing the color intensity of the digital map. The new mobile technique can be used as an alternative to the earlier non-mobile force-feedback devices for interaction with geospatial data.

 

Evreinova, T.V., Evreinov, G., Raisamo, R. An Evaluation of the Virtual Curvature with the StickGrip Haptic Device: A Case Study. UAIS Journal 12(2) 2013, pp. 161-173 DOI 10.1007/s10209-012-0273-0 http://www.springerlink.com/content/y617148415702184/.

Dynamic simulation of distance to the physical surface could promote the development of new inexpensive tools for blind and visually impaired users. The StickGrip is a haptic device comprised of the Wacom pen input device added with a motorized penholder. The goal of the research presented in this paper was to assess an accuracy and usefulness of the new pen-based interaction technique when the position and displacement of the penholder in relation to the pen tip provided the haptic feedback to the user about the distance to the physical or virtual surface of interaction. We aimed to examine how accurately people are able 1) to align the randomly deformed virtual surfaces to the flat surface and 2) to adjust the number of surface samples having a randomly assigned curvature to the template having the given curvature and kept fixed. We approached these questions by measuring both the values of the adjusted parameters and the parameters of the human performance such as a ratio between inspection time and control time spent by the participants to complete the matching task with the use of the StickGrip device. The test of the pen-based interaction technique was conducted in the absence of visual feedback when the subject could rely on the proprioception and kinesthetic sense. The results are expected to be useful for alternative visualization and interaction with complex topographic and mathematical surfaces, artwork, and modeling.

 

Evreinova, T.V., Evreinov G. and Raisamo R. Integrating discrete events and continuous head movements for video-based interaction techniques. Behaviour & Information Technology. Taylor&Francis Group. 2011, vol. 30 no. 6, pp. 739-746.

Human head gestures can potentially trigger different commands from the list of available options in graphical user interfaces or in virtual and smart environments. However, continuous tracking techniques are limited in generating discrete events which could be used to execute a predefined set of commands. In this article, we discuss a possibility to encode a set of discrete events by integrating continuous head movements and crossing-based interaction paradigm. A set of commands can be encoded through specific sequences of crossing points when a head-mouse cursor such as a scaled pointer interacts with a graphical object. The goal of the present experiment was testing the perceptual-motor performance of novices in target acquisition tasks using a subset of round head gestures and symbolic icons designating eight types of directional head movements. We have demonstrated that the novices can equally well execute round head gestures in clockwise and counter-clockwise directions by making two crossings for about 2 s or three crossings for about 3 s. None of the participants reported neck strain or other problems after 360 trials performed during a 40-min test in each of 5 days.

 

Evreinova, T.V., Evreinov, G., Raisamo, R. Interpretation of Ambiguous Images Inspected by the StickGrip Device. IADIS Int. Interfaces and Human Computer Interaction 2011, IADIS IHCI (Rome, Italy, July 2011), pp. 209-217. CD-ROM Proc. July 2011.

Interpretation of ambiguous images perceived visually and relying on supplementary information coordinated with pictorial cues was selected to evaluate the usefulness of the StickGrip device. The ambiguous visual models were achromatic images composed from only two overlapping ellipses with various brightness gradients and relative position of the components. Inspection of images by the tablet pen enhanced with the pencil-like visual pointer decreased discrepancy between their actual interpretation and expected decision by only about 2.6 for concave and by about 1.3 for convex models. Interpretation of the convex images ambiguous with their inverted concave counterparts inspected by the StickGrip device achieved three times less discrepancy between decisions made and expected. Interpretation of the concave images versus inverted convex counterparts was five times more accurate with the use of the StickGrip device. We conclude that the kinesthetic and proprioceptive cues delivered by the StickGrip device had a positive influence on the decision-making under ambiguous conditions.

 

Evreinova, T.G., Evreinov, G. and Raisamo, R. Camera-Based Target Acquisition Augmented with Phosphene Sensations. In Proc. of the 12th Int. Conf. on Computers Helping People with Special Needs. ICCHP 2010 Austria, Linz, 12-16 July 2010, LNCS 6180, Springer-Verlag Berlin Heidelberg 2010, pp. 282-289

This paper presents the results of evaluation of the user performance in the target acquisition task using camera-mouse real time face tracking technique augmented with phosphene-based guiding signals. The underlying assumption was that during non-visual inspection of the virtual workspace (screen area), the transcutaneous electrical stimulation of the optic nerve can be considered as alternative feedback when the visual ability is low or absent. The performance of the eight blindfolded subjects was evaluated. The experimental findings show that the camera-based target acquisition augmented with phosphene sensations is an efficient input technique when visual information is not available.

 

Evreinova, T.V., Evreinov, G., Raisamo, R. Estimating Topographic Heights with the StickGrip Haptic Device. International Symposium on Multimedia Applications and Processing FedCSIS 2011, (Szczecin, Poland, Sept. 18–21, 2011), pp. 699 – 705. ISBN 978-83-60810-39-2, IEEE Catalog Number CFP1185N-ART, http://fedcsis.eucip.pl/proceedings/index.html

This paper presents an experimental study aimed to investigate the impact of haptic feedback when trying to evaluate quantitatively the topographic heights depicted by height tints. In particular, the accuracy of detecting the heights has been evaluated visually and instrumentally by using the new StickGrip haptic device. The participants were able to discriminate the required heights specified in the scale bar palette and to detect these values within an assigned map region. It was demonstrated that the complementary haptic feedback increased the accuracy of visual estimation of the topographic heights by about 32%.

 

Evreinov G.E., Evreinova T.V., Raisamo R. Method, Computer Program and Device for Interacting with a Computer. Finland Patent Application, G06F ID 20090434. 2009, WO/2011/061395, PCT/FI2010/050924 available at: http://www.wipo.int/patentscope/search/en/WO2011061395 .

The invention discloses a device for interacting with a computer and a user. The device comprises a first part comprising a tip for contacting a surface, a second part for contacting the user and a signaling means for interacting with the computer. The device further comprises moving means for moving the second part in relation to the tip while the tip is arranged to maintain contact with the surface. The movement of the second part in relation to the tip corresponds to a change in at least one image parameter, the image being projected on a display, and wherein the movement of the second part in relation to the tip provides the user with a haptic sense.

 

 

 

References

[1] http://en.wikipedia.org/wiki/William_Fetter

[2] http://en.wikipedia.org/wiki/Stereoscopy 

[3] http://en.wikipedia.org/wiki/Volumetric_display

[4] http://mobilementalism.com/2006/06/05/samsung-develop-3d-display-for-mobile-phones/ 

[5] 3D Consortium 2007 http://www.3dc.gr.jp/english/

[6] http://www.amexdigital.com/news/Press_Release-E_ADR_GPlus_GPS_Phone.htm

[7] http://www.mobile3dworld.com/

[8]  E.g., 3D LCD Displays are electrically switchable between 3D (stereoscopic) images and 2D (flat-screen, planar) images, and are capable of 3D display without the need for special glasses. Cited on: http://www.sle.sharp.co.uk/research/optical_imaging/3d_research.php

[9] Evreinov G.E., Agranovski A.V. Stereo spectacles. Int. Cl. 7, G 02 C 1/00, G 02 B 27/22, Patent RU, No 2152066 C1, 2000.

[10] Johnson, M. The first 3D game for the iPhone! 2007 http://sho.rt-b.us/archive/2007/07/corridor-the-first-3d-game-for.html

[11] Battu, B., Kappers, A.M.L., Koenderink, J.J. Ambiguity in pictorial depth. Perception, 2007 36(1): 1290-1304.

[12] Bedford, F.L. Can a space-perception conflict be solved with three sense modalities? Perception, 2007 36(1): 508-515.

 

 

 

contents

 

Links

 

Grigori Evreinov’s Home Page

Tampere University

Faculty of Information Technology and Communication Sciences (ITC)

 

This page maintained by Grigori Evreinov (Grigori.Evreinov@tuni.fi)

 

December 2020