A "Scanpath" down Memory (for pronunciation) Lane: The Eyes have it!

"Eye catching" new study on the function of eye scanning movements in memory creation and access, Eye-movement replay supports episodic remembering, by Johansson, Nystrom, Dewhurst and Johansson in the recent Proceedings of the Royal Society of Biological Sciences. (Also summarized informally in Neurosciencenews.com). The general concept is that eye movement patterns ("scanpaths") that accompany creation of a memory are virtually the same as those used in recalling features of the memory later. 

From the abstract:

Our findings provide direct evidence that such scanpaths are replayed to assemble and reconstruct spatio-temporal relations as we remember and further suggest that distinct scanpath properties differentially contribute depending on the nature of the goal-relevant memory.

In other words, at some level the eyes, at least in episodic memory creation, as part of the overall visuo-spatial processing system of the brain, move in minute scan patterns of millisecond duration, which are not only accessible in recall of that event/episode, but are integral to it. How that relates to internal "events" inside the brain is not entirely clear, of course, but the research again confirms the correlation between path and event-in-time. As the research literature review in the research report relates, those eye movements and related phenomena have been the object of research and various therapeutic applications for over 70 years. 

And how does this relate to remembering or learning pronunciation? The key idea is what constitutes an "episode" in this context " . . . that the sequential replay of eye movements serves to facilitate pure episodic reconstruction in the absence of visual input." (from the article) According to that model, when a gesture is used, for example, associated with a new sound or perhaps an intonation contour, the two are "stored" together in a sense. Either may, in principle, then prompt recall of the other. If the pair are practiced together, the components of the two, the sound, itself, the kinesthetic "track" of the gesture, the "trail" in the visual field can be further joined and strengthened. 

That is, in effect, the basis of the KINETIK Method, as well having gotten its early inspiration from what is known as Observed Experiential Integration Therapy.  

From about 30 years of experience in working with gesture and sound change, the connection between what we call the "movement, tone and touch technique," the MT3, the complex of sound and gesture (and touch), when mapped onto a sound or word or phrase or clause or passage, is incredibly powerful. At times the eyes are actually tracking a gesture across the visual field; at others, the MT3 is out of direct line of sight, but, as the research reveals the "episode" is still embodied, in part, by one or more eye movements that are associated with it. How to exploit those complexes effectively is the question, of course. To learn more about how that works, go to www.actonhaptic.com/kinetik. 

Source: 
Eye-movement replay supports episodic remembering
Roger Johansson, Marcus Nyström,Richard Dewhurst and
Mikael Johansson
Published:15 June 2022 https://doi.org/10.1098/rspb.2022.0964

Proceedings of the Royal Society of Biological Sciences

Fixing your eyes on better pronunciation--or before it!

ClipArt by

Early on in the development of haptic pronunciation teaching, we began by borrowing a number of techniques from Observed Experiential Integration therapy, developed by Rick Bradshaw and colleagues about 20 years ago. OEI has proved to be particularly effective in the treatment of PTSD.  In OEI one of the basic techniques is the use of eye tracking, that is therapists carefully control the eye movements of patients, in some cases stopping at places in the visual field to "massage" points through various loops and depth of field tracking.

Clker.com

We discovered that attempting to control students' eye movement, having them follow with their eyes the track of the gestures across the visual field being used to anchor sounds during pronunciation work, that although memory for sounds seemed better, the holding of attention for such extended lengths of time could be really counterproductive. In some cases, students even became slightly dizzy or disoriented after only a few minutes. (And, in retrospect, we were WAY out of our league . . . )

Consequently, attention shifted to visual focus on only the terminal point in the gestural movement where the stressed syllable of the word or phrase was located, where the hands touched. We have been using that protocol for about a decade.

Now comes a fascinating study by Badde et al., "Oculomotor freezing reflects tactile temporal expectation and aids tactile perception" summarized by ScienceDaily.com, that helps refine our understanding of the relationship between eye movement and touch in focusing attention. In essence, what the research demonstrated was that by stopping or holding eye movement just prior to a when subject was to touch a targeted object, the intensity of the tactile sensation was significantly enhanced. Or, the converse: random eye movement prior to touch tended to diffuse or undermine the impact of touch. That helps explain something . . .

The rationale for haptic pronunciation teaching is, essentially, that the strategic use of touch both successfully manages gesture and focuses much more effectively the placement of stressed syllables in words accompanying the gesture in gesture synchronized speech. In almost all cases, the eyes focus in on the hand about to be touched--just prior to what we term the: TAG (touch-activated ganglia) where touch literally "brings together" or assembles the sound, body movement, vocal resonance and iwth graphic visual schema and meaning of the word or phoneme, itself.

In other words, the momentary freezing of eye movement an instant before the touch event should greatly intensify the resulting impact and later recall produced by the pedagogical strategy. We knew it worked, just didn't really understand why. Now we do.

Put your current pronunciation system on hold for bit . . . and get (at least a bit) haptic!

Original source:
Stephanie Badde, Caroline F. Myers, Shlomit Yuval-Greenberg, Marisa Carrasco. Oculomotor freezing reflects tactile temporal expectation and aids tactile perception. Nature Communications, 2020; 11 (1) DOI: 10.1038/s41467-020-17160-1

Getting a feel for pronunciation: What our pupils can tell us!

Clker.com

What do you do with your eyes when you are struggling to understand something that you are listening to? (Quick: Write that down.) Now some of that, of course, depends on your personal wiring, but this new study “Asymmetrical characteristics of emotional responses to pictures and sounds: Evidence from pupillometry” by Nakakoga, Higashi, Muramatsu, Nakauchi, and Minami of Toyohashi University of Technology, as reported in neuroscience.com, sheds some new "light" on how the emotions may exert influence on our ongoing perception and learning. Using eye tracking and emotion measuring technology, a striking pattern emerges.

From the summary (boldface, mine):
"It suggests that visual perception elicits emotions in all attentional states, whereas auditory perception elicits emotions only when attention is paid to sounds, thus showing the differences in the relationships between attentional states and emotions in response to visual and auditory stimuli."

So, what does that imply for the pronunciation teacher? Several things, including the importance of what is going on in the visual field of learners when they are attempting to learn or change sounds. It has been long established that the process of learning pronunciation is especially susceptible to emotion. It can be an extraordinarily stressful experience for some learners. Even when there are no obvious stressors present, techniques such as relaxation or warm ups have been shown to facilitate learning of various aspects of pronunciation.

Consequently, any emotional trigger in the visual field of the learner can have either "pronounced" positive or negative impact, regardless of what the instructor is attempting to direct the learners' attention to. If, on the other hand, learners' attention is focused narrowly on auditory input and the emotional impact, you have a better chance of managing emotional impact FOR GOOD if you can successfully manage or restrict what is going on in the visual field of the learner that could be counterproductive emotionally (Think: Hypnosis 101. . . or a good warm up . . . or a mesmerizing lecture!)

That doesn’t mean we teach pronunciation with our eyes closed . . . when  it comes to the potential impact of the visual field on our work. Quite the contrary! How does the “front” of the room (or the scenes on screen) feel to your pupils? Can you enhance that? 

To learn more about one good (haptic) way to do that, join us at the next webinars!

Original Research: Open access
 “Asymmetrical characteristics of emotional responses to pictures and sounds: Evidence from pupillometry”.by Nakakoga, S., Higashi, H., Muramatsu, J., Nakauchi, S.,  and Minami, T.
PLOS ONE doi:10.1371/journal.pone.0230775

Good listening (and pronunciation teaching) is in the EYE of the beholder (not just the ear)!

clker.com

Here is some research well worth gazing at and listening to by Pomper and Chait of University College London: The impact of visual gaze direction on auditory object tracking, summarized by Neurosciencenews.com:

In the study, subjects "sat facing three loudspeakers arranged in front of them in a darkened, soundproof room. They were instructed to follow sounds from one of the loudspeakers while ignoring sounds from the other two loudspeakers. . . . instructed to look away from the attended loudspeaker" in an aural comprehension task. What they found was that " . . . participants’ reaction times were slower when they were instructed to look away from the attended loudspeaker . . .  this was also accompanied by an increase in oscillatory neural activity . . . " 

 Look . .  I realize that the connection to (haptic) pronunciation teaching may not be immediately obvious, but it is potentially significant. For example, we know from several research studies (e.g., Molloy et al. 2015) that visual tends to override or "trump" audio--in "head to head" competition in the brain. In addition, auditory generally trumps kinesthetic, but the two together may override visual in some contexts. Touch seems to be able to complement the strength or impact of the other three or serve to unite them or integrate them in various ways. (See the two or three dozen earlier blog posts on those and related issues.)

In this study, you have three competing auditory sources with the eyes tracking to one as opposed to the others. Being done in a dark room probably helped to mitigate the effect of other possible visual distraction. It is not uncommon at all for a student to chose to close her eyes when listening or look away from a speaker (a person, not an audio loudspeaker as in the study). So this is not about simply paying attention visually. It has more to do with eyes either being focused or NOT. 

Had the researchers asked subjects to gaze at their navels--or any other specific object--the results might have been very different. In my view the study is not valid just on those grounds alone, but still interesting in that subjects' gaze was fixed at all.) Likewise, there should have been a control group that did the same protocols with the lights on, etc. In effect, to tell subjects to look away was equivalent to having them try to ignore the target sound and attend to it at the same time. No wonder there was " . . .  an increase in oscillatory neural activity"! Really!

In other words, the EYEs have it--the ability to radically focus attention, in this case to sound, but to images as well. That is, in effect, the basis of most hypnosis and good public speaking, and well-established in brain research. In haptic pronunciation teaching, the pedagogical movement patterns by the instructor alone should capture the eyes of the students temporarily, linking back to earlier student experience or orientation to those patterns. 

So try this: Have students fix their eyes on something reasonable or relevant, like a picture or neutral, like an area on the wall in front of them--and not look away--during a listening task. Their eyes should not wander, at least not much. Don't do it for a very long period of time , maybe 30 seconds, max at the start. You should explain to them this research so they understand why you are doing it. (As often as I hammer popular "Near-ol'-science", this is one case where I think the general findings of the research are useful and help to explain a very common sense experience.)

 I have been using some form of this technique for years; it is basic to haptic work except we do not specifically call attention to the eye tracking since the gestural work naturally accomplishes that to some degree. (If you have, too, let us know!)

This is particularly effective if you work in a teaching environment that has a lot of ambient noise in the background. You can also, of course, add music or white noise to help cancel out competing noise or maybe even turn down the lights, too, as in the research. See what I mean?

Good listening to you!

References:
UCL (2017, July 5). Gaze Direction Affects Sound Sensitivity. NeuroscienceNew. Retrieved July 5, 2017 from http://neurosciencenews.com/sound-sensitivity-gaze-direction-7029/
Molloy, K, Griffiths, D.,  Chait, Lavie, N. Inattentional Deafness: Visual Load Leads to Time-Specific Suppression of Auditory Evoked Responses. Journal of Neuroscience, 2015; 35 (49): 16046 DOI: 10.1523/JNEUROSCI.2931-15.2015

Killing pronunciation improvement: better heard (and felt) but not seen!

Clker.com

Fascinating study, Visual Distractors Disrupt Audiovisual Integration Regardless of Stimulus Complexity, by Gibney, et al. Department of Neuroscience, Oberlin College.

Tigger warning: This is a thick, technical read, but the conclusions of the study have potentially important implications for pronunciation teaching, especially attempts to enhance uptake of new and corrected sounds or patterns that rely on effective integration of sounds, images, movement and vocal resonance. 

In essence, what the research examined was, as the title suggests, how distractions in the visual field affected subjects attention and ability to learn and recall audio-visual stimuli (images on a computer screen accompanied by sounds). What was striking (again as evident in the title) was that no matter how complex the task of associating the targeted sound with the visual image or object in focus, with even the slightest distraction created on the screen, e.g., a object briefly appearing in a corner, the subject's ability to integrate and recall the complex target later . . .was compromised.

The implications for pronunciation teaching?  Not surprisingly, attention is critical in integrating sensory information. We know that, of course. What is more interesting is the idea that any visual distraction whatsoever that occurs when sound, movement and visual imagery (such as the orthography or phonetic representation of a word or phrase) are being "integrated" may seriously  undermine the process. In other words, visual attention and eye tracking during the process may have dramatic impact. That is a "variable" that can, in principle, be managed in the classroom, although most do not consider visual distraction to be potentially that disruptive of pronunciation instruction. But it certainly can be.

We discovered early on that in haptic pronunciation work, where not only sound, visual imagery, movement and vocal resonance are involved--but touch as well, visual distraction can seriously derail the process. This research suggests, for example, that the same effect during general pronunciation work as well, especially oral work, may be a significant impediment in some contexts. 

The sterile, featureless language laboratory booth of old may have had more going for it than we thought! In early haptic work we experimented with controlling eye tracking. Perhaps it is time we revisited that idea. It certainly deserves our undivided attention.

Original research article: Front. Integr. Neurosci., 20 January 2017 | https://doi.org/10.3389/fnint.2017.00001

Back to the future of pronunciation teaching (and the "Goldfish" standard for attention management)

You apparently have a bit more than 8 seconds to read this post. So you may want to just scroll down to the conclusion and start there . . .

Clip art: 

Capturing and holding attention, if only for a few seconds, is the key to effective change in pronunciation work, especially for "mechanical" adjustments--and most other things in life. In earlier blog posts, the "gold standard" or is sine qua non of haptic pronunciation work has been seen to be about 3 seconds. In other words, for a learner to adequately experience the totality of a new sound or word, physically, auditorily, visually and conceptually--connecting things together, before moving on to practice or at least noticing or any chance at "uptake"-- takes complete, undivided attention for at least that long or longer.

Even that is often an unrealistic requirement with all the other potential distractions in the classroom or visual field. Research on the effectiveness of recasting learner utterances by instructors, for example, (Loewen and Philip, 2006) suggests that most of the time that strategy is relatively ineffective. One critical variable is always the quality or intentionality of learner attention, both in term of what the function the instructor is attempting to carry out and general learner receptivity.

Clker.com

Recall that Microsoft claims that our collective attention span, in part due to the impact of technology, has now dropped to about 8 seconds, just below that of the goldfish. (The UK Telegraph report is much more entertaining than that from the techies.

A new study by Moher, Anderson and Song of Brown University, summarized by Science Daily.com, adds a fascinating piece to the puzzle and may suggest how to begin to maintain attention better in class. What they discovered in an experimental study was that their subjects were, in effect, better able to "block" obvious distractions than they were more subtle ones. Backgrounded images in the visual field had more effect on subsequent action than did foregrounded, more striking elements which appeared to be easier for the brain to manage or ignore. They seem to have "discovered" one possible path into the mind by subliminal stimuli, evading first line conceptual or perceptual defences.

What is the obvious "subtle, unobtrusive, yet potent" application to pronunciation teaching? If you don't have "full body, mind and visual field" attention, there is no telling what is interfering with anchoring of sound change in the brain and subsequent total or partial recall.

Early on in EHIEP (Essential Haptic-integrated English Pronunciation) work I experimented extensively with controlling eye movement, in part to maintain concentration and attention, based primarily on the research underlying the therapeutic model of "Observed experiential integration" (See citation below) developed by  Bradshaw and Cook (2011). The effect was dramatic in working with individuals but applying those techniques to the classroom proved at least impractical. In part because the haptic pedagogical system was just developing, I backed off from eye patterning techniques in pronunciation work in 2009.

Based on Moher et al's research, however, it is perhaps time to again give directed eye movement management a "second look" in our work, going back to what I believe is the (haptic) future of pronunciation instruction, especially in virtual, computer-mediated applications.

Will report back on an in progress exploratory study with one learner using some eye movement management later this summer. Not surprisingly I am already "seeing" some promising results, attending to features of the teaching session that I would normally not have noticed!

Full citations:

Brown University. "Surprise: Subtle distractors may divert action more than overt ones." ScienceDaily. ScienceDaily, 16 July 2015, www.sciencedaily.com/releases/2015/07/150716123831.htm. (Jeff Moher, Brian A. Anderson , Joo-Hyun Song. Dissociable Effects of Salience on Attention and Goal-Directed Action. Current Biology, 2015 DOI: 10.1016/j.cub.2015.06.029)

Bradshaw, R. A., Cook, A., McDonald, M. J. (2011). Observed experiential integration (OEI): Discovery and development of a new set of trauma therapy techniques. Journal of Psychotherapy Integration, 21(2), 104-171.

Loewen, S., and Philip, J. (2006). Recasts in the adult English L2 classroom: Characteristics, explicitness, and effectiveness. The Modern Language Journal, 90, 536-556.

Haptic cinema and EHIEP-tic pronunciation training

My discovery of "haptic cinema" and that approach to experiential entertainment and teaching about 6 years ago was a game changer. The integration of the senses, especially the place of perceived texture in that media became the phenomenological model for "haptic-integrated clinical pronunciation," and still is. Here is a great example, "Haptic cinema: a sensory interface to the city."  It is about 11 minutes long. Put on some earphones, sit someplace where you'll have no visual distractions and experience it. 

Clip art: Clker

That is what it should feel like, the felt sense of haptic anchoring in EHIEP instruction, when the learner articulates a sound or word with rich vocal resonance as hands move across the visual field (with some degree of eye tracking) and the hands touch on the stressed vowel--possibly followed by a short continued movement completing an intonation "denouement." 

To prepare for watching it, you might go outside and hug a tree first . . . 

Haptic anchoring with eye tracking: for right handers only?

Clip art: Clker

Early in the development of HICP, I explored using eye tracking to enhance anchoring, a technique used in many disciplines and therapies. (See earlier posts.) What I discovered quickly was that that technique seemed to work well for some but, for others, it could be disconcerting or even disorienting to the point of feelings of vertigo and nausea! This 2008 research summarized by Munger of Cognitive Daily by Logan and Roediger, helps to explain why.

For the strongly right hand dominant, those procedures, in some contexts (such as recalling lists), can be highly effective. For others, the eye saccade exercise seems to downgrade performance. When working with individuals I still may use significant eye tracking of hand movement, but not if they are ambidextrous (as I am) or occasionally left handed, depending on their real eye dominance. In class work, likewise, we have learned that even excessive repetitions of basic haptic anchoring, which do not require explicit eye engagement (which should NEVER be necessary if done correctly anyway) should be avoided.

What is interesting is that a similar "follow the windshield wiper" technique is used to desensitize emotionally traumatized (left or right handed) patients in psychotherapy. You might try that version on "haptically or methodologically challenged" colleagues or students, in fact. It works "both ways," so to speak . . .

Future pronunciation: Hands-free "optic" anchoring (with eye tracking)

Clip art: Clker

Ever since seeing the Clint Eastwood 1982 film, Firefox, I have been intrigued with the idea of using eye tracking for teaching pronunciation. In the film, Eastwood flies a stolen, high tech Russian fighter--controlled by his eyes and a few spoken Russian commands-- back to the US. As noted in the right column, the work of Bradshaw and Cook in developing "Observed Experiential Integration,", which involves extensive, therapeutic use of eye tracking, was fundamental to much of the early development of the HICP/EHIEP framework. Here is, basically, a marketing piece for a company that has developed (from my perspective) an amazing range of eye tracking-based software applications. As I read the product list, it would take only two or three applications to allow a learner to do almost all of the protocols or procedures that we have developed, hands free.

In fact, the "optic anchoring" created simply by tracking the eyes across the visual field in roughly the same patterns that we do with arms and hands would be at least as effective, if not more so. Although we do use some eye tracking techniques in working with accent reduction, in general EHIEP work, no explicit eye tracking is used, in part because of the inherent potency of eye tracking procedures and the absolute necessity of being formally trained in working with it. This technology is certainly worth "taking a look at" now. It is clearly integral to the future of virtual reality-based language instruction.

Remembering: Just close your eyes to block out that distracting sound in the background or pronunciation

Clip art: Clker

Here is (an abstract only of) a study by Perfect, Andrade and Eagan (2011) which demonstrates that under certain conditions, closing the eyes effectively blocks out background/environmental  auditory clutter (in this case.) The technique seemed to enhance or at least protect both visual and auditory recall. Earlier posts have reported on research related to the impact of eye closure (or purely haptic, nonvisual engagement) on encoding, showing a parallel effect. It is as if once the eyes have done their part in establishing an object's properties or pattern, in some contexts or stages, it may be better to carefully limit further, potentially distracting gaze.

Most disciplines deal with directed eye engagement in some form, especially in expert practice and performance. That can be done any number of ways from full closure to fixed positions in the visual field. (I'm sure you have students who, themselves, use similar strategies at times. Regardless, the implication for pronunciation instruction is intriguing: To get or recall the optimal auditory felt sense of a sound, simply cut out external auditory interference. That appears to be relatively easy . . . you can  do it with your eyes closed!

Pronunciation modalities: out of sight--but IN mind!.

clip art: Clker

In this 2009 study of modality dominance, by Hecht and Reiner, when visual is paired one-on-one with either haptic or auditory competing stimuli, visual consistently overpowers either of the two. When the three are presented simultaneously, however, the dominance of visual disappears. That may explain why having some learners focus on a visual schema (such as the orthography) while articulating or practicing a new sound may not turn out to be very efficient--or doing a kinesthetic "dance" of some kind to practice a rhythm pattern (without speaking at the same time) while looking at something in the visual field, may not work all that well either for some learners.

The presence of eye engagement may override or nullify information in the competing modality. In HICP, where all three modalities are usually engaged, the "distracting" influence of sight is at least lessened. In fact, the tri-modality "hexus" should only better  facilitate the integration of the graphic word, the felt (haptic) sense of producing it and the internal (auditory) bone- resonance and vibrations. Although a substantial amount of pronunciation learning may be better accomplished with eyes closed, tri-modal (haptic, visual and auditory) techniques probably come in a close second. We will "see" in forthcoming research!