What you see is what you forget: pronunciation feedback perturbations

Tigger warning* This blogpost concerns disturbing images, perturbations, during pronunciation
work.

In some sense, almost all pronunciation teaching involves some type of imitation and repetition of a model. A key variable in that process is always feedback on our own speech, how well it conforms to the model presented, whether coming to us through the air or perhaps via technology, such as headsets--in addition to the movement and resonance we feel in our vocal apparatus and bone structure in the head and upper body.  Likewise, choral repetition is probably the most common technique, used universally. There are, of course, an infinite number of reasons why it may or may not work, among them, of course, distraction or lack of attention.

Clker.com

We generally, however, do not take all that seriously what is going on in the visual field in front of the learner while engaged in repetition of L2 sounds and words. Perhaps we should. In a recent study by Liu et al, Auditory-Motor Control of Vocal Production during Divided Attention: Behavioral and ERP Correlates,  it was shown that differing amounts of random light flashes in the visual field  affected the ability of learners to adjust the pitch of their voice to the model being presented for imitation. The research was done in Chinese, with native Mandarin speakers, attempting to adjust the tone patterns of words presented to them, along with the "light show". They were instructed to produce the models they heard as accurately as possible.

What was surprising was the degree to which visual distraction (perturbation) seemed to directly impact subjects' ability to adjust their vocal production pitch in attempting to match the changing tone of the models they were to imitate. In other words, visual distraction was (cross-modally) affecting perception of change and/or subsequent ability to reproduce it. The key seems to be the multi-modal nature of working memory itself. From the conclusion: "Considering the involvement of working memory in divided attention for the storage and maintenance of multiple sensory information  . . .  our findings may reflect the contribution of working memory to auditory-vocal integration during divided attention."

The research was, of course, not looking at pronunciation teaching, but the concept of management of attention and the visual field is central to haptic instruction, in part because touch, movement and sound are so easily overridden by visual stimuli or distraction. Next time you do a little repetition or imitation work, figure out some way to insure that working memory perturbation by what is around learners is kept to a minimum. You'll SEE the difference. Guaranteed.

Citation:
Liu Y, Fan H, Li J, Jones JA, Liu P, Zhang B and Liu H (2018) Auditory-Motor Control of Vocal Production during Divided Attention: Behavioral and ERP Correlates. Front. Neurosci. 12:113. doi: 10.3389/fnins.2018.00113

*The term "Tigger warning" is used on this blog to indicate potentially mild or nonexistent emotional disruption that can easily be overrated. 

Ferreting out good pronunciation: 25% in the eye of the hearer!

Clker.com

Something of an "eye opening" study, Integration of Visual Information in Auditory Cortex Promotes Auditory Scene Analysis through Multisensory Binding, by Town, Wood, Jones, Maddox, Lee, and Bizley of University College London, published on Neuron. One of the implications of the study:

"Looking at someone when they're speaking doesn't just help us hear because of our ability to recognise lip movements – we've shown it's beneficial at a lower level than that, as the timing of the movements aligned with the timing of the sounds tells our auditory neurons which sounds to represent more strongly. If you're trying to pick someone's voice out of background noise, that could be really helpful," They go on to suggest that someone with hearing difficulties have their eyes tested as well.

I say "implications" because the research was actually carried out on ferrets, examining how sound and light combinations were processed by their auditory neurons in their auditory cortices. (We'll take their word that the ferret's wiring and ours are sufficiently alike there. . . )

The implications for language and pronunciation teaching are interesting, namely: strategic visual attention to the source of speech models and participants in conversation may make a significant impact on comprehension and learning how to articulate select sounds. In general, materials designers get it when it comes to creating vivid, even moving models. What is missing, however, is consistent, systematic, intentional manipulation of eye movement and fixation in the process. (There have been methods that dabbled in attempts at such explicit control, e.g., "Suggestopedia"?)

In haptic pronunciation teaching we generally control visual attention with gesture-synchronized speech which highlights stressed elements in speech, and something analogous with individual vowels and consonants. How much are your students really paying attention, visually? How much of your listening comprehension instruction is audio only, as opposed to video sourced? See what I mean?

Look. You can do better pronunciation work.


Citation: (Open access)

DOI: https://doi.org/10.1016/j.neuron.2017.12.034

Empathy for strangers: better heard and not seen? (and other teachable moments)

The technique of closing one's eyes to concentrate has both everyday sense and empirical research support. For many, it is common practice in pronunciation and listening comprehension instruction. Several studies of the practice under various conditions have been reported here in the past. A nice 2017 study by Kraus of Yale University, Voice-only communication enhances empathic accuracy, examines the effect from several perspectives.
😑
What the research establishes is that perception of the emotion encoded in the voice of a stranger is more accurately determined with eyes closed, as opposed to just looking at the video or watching the video with sound on. (Note: The researcher concedes in the conclusion that the same effect might not be as pronounced were one listening to the voice of someone we are familiar or intimate with, or were the same experiments to be carried out in some culture other than "North American".) In the study there is no unpacking of just which features of the strangers' speech are being attended to, whether linguistic or paralinguistic, the focus being:
 . . . paradoxically that understanding others’ mental states and emotions relies less on the amount of information provided, and more on the extent that people attend to the information being vocalized in interactions with others.
😑
The targeted effect is statistically significant, well established. The question is, to paraphrase the philosopher Bertrand Russell, does this "difference that makes a difference make a difference?"--especially to language and pronunciation teaching?
😑
How can we use that insight pedagogically? First, of course, is the question of how MUCH better will the closed eyes condition be in the classroom and even if it is initially, will it hold up with repeated listening to the voice sample or conversation? Second, in real life, when do we employ that strategy, either on purpose or by accident? Third, there was a time when radio or audio drama was a staple of popular media and instruction. In our contemporary visual media culture, as reflected in the previous blog post, the appeal of video/multimedia sources is near irresistible. But, maybe still worth resisting?
😑
Especially with certain learners and classes, in classrooms where multi-sensory distraction is a real problem, I have over the years worked successfully with explicit control of visual/auditory attention in teaching listening comprehension and pronunciation. (It is prescribed in certain phases of hapic pronunciation teaching.) My sense is that the "stranger" study actually is tapping into comprehension of new material or ideas, not simply new people/relationships and emotion. Stranger things have happened, eh!
😑
If this is a new concept to you in your teaching, close your eyes and visualize just how you could employ it next week. Start with little bits, for example when you have a spot in a passage of a listening exercise that is expressively very complex or intense. For many, it will be an eye opening experience, I promise!
😑

Source:
Kraus, M. (2017). Voice-only communication enhances empathic accuracy, American Psychologist 72(6)344-654.

Good listening (and pronunciation teaching) is in the EYE of the beholder (not just the ear)!

clker.com

Here is some research well worth gazing at and listening to by Pomper and Chait of University College London: The impact of visual gaze direction on auditory object tracking, summarized by Neurosciencenews.com:

In the study, subjects "sat facing three loudspeakers arranged in front of them in a darkened, soundproof room. They were instructed to follow sounds from one of the loudspeakers while ignoring sounds from the other two loudspeakers. . . . instructed to look away from the attended loudspeaker" in an aural comprehension task. What they found was that " . . . participants’ reaction times were slower when they were instructed to look away from the attended loudspeaker . . .  this was also accompanied by an increase in oscillatory neural activity . . . " 

 Look . .  I realize that the connection to (haptic) pronunciation teaching may not be immediately obvious, but it is potentially significant. For example, we know from several research studies (e.g., Molloy et al. 2015) that visual tends to override or "trump" audio--in "head to head" competition in the brain. In addition, auditory generally trumps kinesthetic, but the two together may override visual in some contexts. Touch seems to be able to complement the strength or impact of the other three or serve to unite them or integrate them in various ways. (See the two or three dozen earlier blog posts on those and related issues.)

In this study, you have three competing auditory sources with the eyes tracking to one as opposed to the others. Being done in a dark room probably helped to mitigate the effect of other possible visual distraction. It is not uncommon at all for a student to chose to close her eyes when listening or look away from a speaker (a person, not an audio loudspeaker as in the study). So this is not about simply paying attention visually. It has more to do with eyes either being focused or NOT. 

Had the researchers asked subjects to gaze at their navels--or any other specific object--the results might have been very different. In my view the study is not valid just on those grounds alone, but still interesting in that subjects' gaze was fixed at all.) Likewise, there should have been a control group that did the same protocols with the lights on, etc. In effect, to tell subjects to look away was equivalent to having them try to ignore the target sound and attend to it at the same time. No wonder there was " . . .  an increase in oscillatory neural activity"! Really!

In other words, the EYEs have it--the ability to radically focus attention, in this case to sound, but to images as well. That is, in effect, the basis of most hypnosis and good public speaking, and well-established in brain research. In haptic pronunciation teaching, the pedagogical movement patterns by the instructor alone should capture the eyes of the students temporarily, linking back to earlier student experience or orientation to those patterns. 

So try this: Have students fix their eyes on something reasonable or relevant, like a picture or neutral, like an area on the wall in front of them--and not look away--during a listening task. Their eyes should not wander, at least not much. Don't do it for a very long period of time , maybe 30 seconds, max at the start. You should explain to them this research so they understand why you are doing it. (As often as I hammer popular "Near-ol'-science", this is one case where I think the general findings of the research are useful and help to explain a very common sense experience.)

 I have been using some form of this technique for years; it is basic to haptic work except we do not specifically call attention to the eye tracking since the gestural work naturally accomplishes that to some degree. (If you have, too, let us know!)

This is particularly effective if you work in a teaching environment that has a lot of ambient noise in the background. You can also, of course, add music or white noise to help cancel out competing noise or maybe even turn down the lights, too, as in the research. See what I mean?

Good listening to you!

References:
UCL (2017, July 5). Gaze Direction Affects Sound Sensitivity. NeuroscienceNew. Retrieved July 5, 2017 from http://neurosciencenews.com/sound-sensitivity-gaze-direction-7029/
Molloy, K, Griffiths, D.,  Chait, Lavie, N. Inattentional Deafness: Visual Load Leads to Time-Specific Suppression of Auditory Evoked Responses. Journal of Neuroscience, 2015; 35 (49): 16046 DOI: 10.1523/JNEUROSCI.2931-15.2015

Great pronunciation teaching? (The "eyes" have it!)

Clker.com

Attention! Внимание!

Seeing the connection between two new studies, one on the use of gesture by trial lawyers in concluding arguments and one on how a "visual nudge" can seriously disrupt our ability to describe recalled visual properties of common objects--and by extension, pronunciation teaching--may seem a bit of a stretch, but the implications for instruction, especially systematic use of gesture in the classroom are fascinating.

The bottom line: what the eyes are doing during pronunciation work can be critical, at least to efficient learning. Have done dozens posts over the years on the role or impact of visual modality on pronunciation work; this adds a new perspective. 

The first, by Edmiston and Lupyan of  University of Wisconsin-Madison, Visual interference disrupts visual knowledge, summarized in a ScienceDaily summary:

"Many people, when they try to remember what someone or something looks like, stare off into space or onto a blank wall," says Lupyan. "These results provide a hint of why we might do this: By minimizing irrelevant visual information, we free our perceptual system to help us remember."

The "why" was essentially that visual distraction during recall (and conversely in learning, we assume), could undermine ability to describe visual properties of even common well-known objects, such as the color of a flower. That is a striking finding, countering the prevailing wisdom that such properties are stored in the brain more abstractly, not so closely tied to objects themselves in recall.

Study #2: Matoesian and Gilbert of the University of Illinois at Chicago, in an article published in Gesture entitled, Multifunctionality of hand gestures and material conduct during closing argument. The research looked at the potential contribution of gesture to the essential message and impact of the concluding argument to the jury. Not surprisingly, it was evident that the jury's visual attention to the "performance" could easily be decisive in whether the attorney's position came across as credible and persuasive. From the abstract:

This work demonstrates the role of multi-modal and material action in concert with speech and how an attorney employs hand movements, material objects, and speech to reinforce significant points of evidence for the jury. More theoretically, we demonstrate how beat gestures and material objects synchronize with speech to not only accentuate rhythm and foreground points of evidential significance but, at certain moments, invoke semantic imagery as well. 

The last point is key.  Combine that insight with the "Nudge" study. It doesn't take much to interfere with "getting" new visual/auditory/kinesthetic/tactile input. The dominance of visual over the other modalites is well established, especially when it comes to haptic (movement plus touch). These two studies add an important piece, that random VISUAL, itself, can seriously interfere with targeted visual constructs or imagery as well. In other words, what your student LOOK at and how effective their attention is during pronuncation work can make a difference--an enormous difference, as we have discovered in haptic pronunciation teaching.

Whether learners are attempting to connect the new sound to the script in the book or on the board, or are attempting to use a visually created or recalled script (which we often initiate in instruction) or are mirroring or coordinating their body movement/gesture with the pronunciation of a text of some size, the "main" effect is still there: what is at that time in their visual field in front of them or in their created visual space in their brain may strongly dictate how well things are integrated--and recalled later. (For a time I experimented with various system of eye tracking control, myself, but could not figure out how to develop that effectively--and safely, but emerging technologies offer us a new "look" at that methodology in several fields today.)

So, how do we appropriately manage "the eyes" in pronunciation instruction? Gestural work helps to some extent, but it requires more than that. I suspect that virtual reality pronunciation teaching systems will solve more of the problem. In the meantime, just as a point of departure and in the spirit of the earlier, relatively far out "suggestion-based" teaching methods, such as Suggestopedia, assume that you are responsible for everything that goes on during a pronunciation intervention (or interdiction, as we call it) in the classroom. (See even my 1997 "suggestions" in that regard as well!)

Now I mean . . . everything, which may even include temporarily suspending extreme notions of learner autonomy and metacognitive engagement . . .

See what I mean?

Sources: 
Matoesian, G. and Gilbert, K.  (2016). Multifunctionality of hand gestures and material conduct during closing argument. Gesture, Volume 15, Issue 1, 2016, pages: 79 –114
Edmiston, P. and  Gary Lupyan, G. (2017) Visual interference disrupts visual knowledge. Journal of Memory and Language, 2017; 92: 281 DOI: 10.1016/j.jml.2016.07.002

Invisible pronunciation: What you see is not necessarily what you get.

Nice new study by Smotrova "Making Pronunciation Visible: Gesture In Teaching Pronunciation", in
press, in TESOL Quarterly, examining in depth the pedagogical gestures used by a pronunciation teacher. She had devised an ingenious set of gestures to signal various aspects of pronunciation, such as stress placement, intonation contours, etc. Students (subjects) seemed to have engaged well with the process and there was evidence of both uptake and subsequent student-initiated use of the gestural system.

EHIEP

In the literature review, Essential Haptic-integrated Pronunciation (EHIEP) is described in some detail, for the most part accurately. What is missing, however, is any reference to the critical role of touch in contributing to the effectiveness of haptic pronunciation (HPT). EHIEP is, instead, characterized as a "kinesthetic" approach, meaning: movement and gesture-based. That is, of course, correct at face value, as far as it goes, but the application of touch to the system has been fundamental for over a decade, since 2005.

What we discovered very early on was that gesture used for such "signalling" by the instructor has valuable applications, such as pointing out problems or coordination of gross motor movements such as hand clapping or dancing. What was far more problematic, however, was attempting to use gesture systematically by conducting learner body movement to help them "embody" the new or corrected sounds. Only by using touch to anchor gesture, primarily by touch on the stressed syllable but also in many cases by assigning touch to the beginning and the terminus of the gestural movement, could we consistently work effectively with pedagogical gesture.

That is particularly the case when you want learners to use gesture spontaneously or with homework assignments. If not carefully controlled and applied, gesture use is often at best only marginally effective; at worst, threatening, intimidating and highly invasive.

In other words, the key is not just what you can see someone else doing,  but how well that gesture connects up in the body, or is "embodied" with the sound element or structure being taught, corrected or practiced. And that happens most consistently when the learner does the pedagogical movement pattern (gesture) with precision, the focus of EHIEP. Touch makes that process consistent and systematic, and generally quite acceptable and emotionally "safe" for learners as well. 

The general visual/cognitive bias in pronunciation teaching today is very problematic. Although it is understandable, given the often rigid and noncognitive nature of traditional drill and articulatory training models, it is simply too easy for learners and instructors to avoid the physical/kinesthetic side of the process which can be both inordinately time consuming and individualized.

At the basic instructional level, HPT is (simply) the answer.

3D pronunciation instruction: Ignore the other 3 quintuplets for the moment!

Clker.com

For a fascinating look at what the field may feel like--from a somewhat unlikely source, a 2015 book, 3D Cinema: Optical illusions and tactile experience, by Ross, provides a (phenomenal) look at how and why contemporary 3D special effects succeeds in conveying the "sensation of touch". In other words, as is so strikingly done in the new Star Wars epic, the technology tricks your brain into thinking that you are not only there flying that star fighter but that you can feel the ride throughout your hands and body as well.

This effect is not just tied in to current gimmicks, such as moving and vibrating theater seats or spray mist blown on you, or various odors and aromas being piped in, although it can be. Your mirror neurons respond more as if it is you who is doing the flying, that you are (literally) "in touch" with the actor. The neurological interconnectedness between the senses (or modalities) provides the bridge to greater and greater sense of the real or a least very "close encounter."

How does the experience in a good 3D movie compare to your best multi-sensory events or teachable moments in the classroom, focusing on pronunciation? 

It is easy to see, in principle, the potential for language teaching, creating one vivid teachable moment after another, "Wowing!" the brain of the learner with multi-sensory, multi-,modal experience. As noted in earlier blogposts on haptic cinema, based in part on Marks (2002), that concept, "the more multi-sensory, the better", by just stimulating more of the learner's (whole) brain virtually anything is teachable, is implicit in much of education and entertainment.

Although earlier euphoria has moderated, one reason it can still sound so convincing is our common experience of remembering the minutest detail from a deeply moving or captivating event or presentation. We all have had the experience of being present at a poetry reading or great speech where it was as if all our senses were alive, on overdrive. We could almost taste the peaches; we could almost smell the gun powder.

Part of the point of 3D cinema is that it becomes SO engaging that our tactile awareness is also heightened enormously. As that happens the associated connections to other modalities are "fired" as well. We experience the event more and more holistically. How that integration happens exactly can probably be described informally as something like: audio-visual-cognitive-affective-kinasethetic-tactile-olfactory and "6th sense!" experienced simultaneously.

At that point, apparently the brain is multitasking at such high speed that everything is perceived as "there" all at once. And that is the key notion. That would seem to imply that if all senses are strongly activated and recording "data" then, what came in on each sensory circuit will later still be equally retrievable. Not necessarily. As extensive research and countless commercially available systems have long established,  for acquisition of vocabulary, pragmatics, reading skills and aural comprehension, the possibilities of rich multi-sensory instruction seem limitless at this point.

Media can provide memorable context and secondary support, but why that often does not work as well for learning of some other skills, including pronunciation is still something of a mystery. (Caveat emptor: I am just completing a month-long "tour of duty" with seven, young grandchildren . . . ) In essence, our sensory modalities are not unlike infant octuplets, competing for our attention and storage space. Although it is "possible" to attend to a few at once, it is simply not efficient. Best case, you can do maybe two at a time, one on each knee.

The analogy is more than apt. In a truly "3D" lesson, consistent with Ross (2015), whether f2f or in media, where, for example, the 5 primary "senses" of pronunciation instruction (visual, auditory, kinaesthetic, tactile and meta-cognitive) are near equally competitive, that is vividly present in the lesson, overwhelmingly so. Tactile/kinaesthetic can be unusually prominent, accessible, in part, as noted in earlier blogposts, because it serves to "bind together" the other senses. In that context, consciously attending to any two or three simultaneously is feasible.

So how can we exploit such vivid, holistically experienced, 3D-like milieu, where movement and touch figure in more prominently? I never thought you'd ask! Because of the essentially physical, somatic experience of pronunciation--and this is critical, from our experience and field testing--two of the three MUST be kinaesthetic and tactile--a basic principle of haptic pronunciation teaching.(Take your pick of the other three!)

Consider "haptic" simply an essential "add on" to your current basic three (visual, auditory and meta-cognitive), and "do haptic" along with one or two of the other three. The standard haptic line-of march:

A. Visual-Meta-cognitive (very brief explanation of what, plus symbol, or key word/phrase)
B. Haptic-metacognitive (movement and touch with spoken symbol name or key word/phrase, typically 3x)
C. Haptic-auditory (movement and touch, plus basic sound, if the target is a vowel or consonant temporarily in isolation, or target word/phrase, typically 3x)
D. Haptic-Visual-Auditory (movement and touch, plus contextualized word or phrase, spoken with strong resonance, typically 3x)
E. Some type of written note made for further reference or practice
F. (Outside of class practice, for a fixed period of up to 2 weeks follows much the same pattern.)

Try to capture the learner's complete (whole body/mind) attention for just 3 seconds per repetition--if possible! Not only can that temporarily let you pull apart the various dimensions of the phonemic target for attention, but it can also serve to create a much more engaging (near 3D) holistic experience out of a potentially "senseless" presentation in the first place--with "haptic" in the mix from the outset.

Happy New Year!

Keep in touch.

Citation:
Ross, M. (2015). 3D Cinema: Optical Illusions and Tactile Experiences. London: Springer, ISBN: 978-1-349-47833-0 (Print) 978-1-137-37857-6 (Online)

Can't see teaching (or learning) pronunciation? Good idea!

Clker.com

A common strategy of many learners when attempting to "get" the sound of a word is to close their eyes. Does that work for you? My guess is that those are highly visual learners who can be more easily distracted. Being more auditory-kinesthetic and somewhat color "insensitive" myself, I'm instead more vulnerable to random background sounds, movement or vibration. Research by Molloy et al. (2015), summarized by Science Daily (full citation below) helps to explain why that happens.

In a study of what they term "inattentional deafness," using MEG (magnetoencephalography), the researchers were able to identify in the brain both the place and point at which auditory and visual processing in effect "compete" for prominence. As has been reported more informally in several earlier posts, visual consistently trumps auditory, which accounts for the common life-ending experience of  having been  oblivious to the sound of screeching tires while crossing the street fixated on a smartphone screen . . . The same applies, by the way, for haptic perception as well--except in some cases where movement, touch, and auditory team up to override visual. 

The classic "audio-lingual" method of language and pronunciation teaching, which made extensive use of repetition and drill, relied on a wide range of visual aids and color schemas, often with the rationale of maintaining learner attention. Even the sterile, visual isolation of the language lab's individual booth may have been especially advantageous for some--but obviously not for everybody!

What that research "points to" (pardon the visual-kinesthetic metaphor) is more systematic control of attention (or inattention) to the visual field in teaching and learning pronunciation. Computer mediated applications go to great lengths to manage attention but, ironically, forcing the learner's eyes to focus or concentrate on words and images, no matter how engaging, may, according to this research, also function to negate or at least lesson attention to the sounds and pronunciation. Hence, the intuitive response of many learners to shut their eyes when trying to capture or memorize sound. (There is, in fact, an "old" reading instruction system called the "Look up, say" method.)

The same underlying, temporary "inattention deafness" also probably applies to the use of color associated with phonemes --or even the IPA system of symbols in representing phonemes. Although such visual systems do illustrate important relationships between visual schemas and sound that help learners understand the inventory of phonemes and their connection to letters and words in general, in the actual process of anchoring and committing pronunciation to memory, they may in fact diminish the brain's ability to efficiently and effectively encode the sound and movement used to create it.

The haptic (pronunciation teaching) answer is to focus more on movement, touch and sound, integrating those modalities with visual.The conscious focus is on gesture terminating in touch, accompanied by articulating the target word, sound or phrase simultaneously with resonant voice. In many sets of procedures (what we term, protocols) learners are instructed to either close their eyes or  focus intently on a point in the visual field as the sound, word or phrase to be committed to memory is spoken aloud.

The key, however, may be just how you manage those modalities, depending on your immediate objectives. If it is phonics, then connecting letters/graphemes to sounds with visual schemas makes perfect sense. If it is, on the other hand, anchoring or encoding pronunciation (and possibly recall as well), the guiding principle seems to be that sound should be best heard (and experienced somatically, in the body) . . . but (to the extent possible) not seen!

See what I mean? (You heard it here!)

Full citation:
Molloy, K., Griffiths, T., Chait, M., and Lavie, N. 2015. Inattentional Deafness: Visual Load Leads to Time-Specific Suppression of Auditory Evoked Responses. Journal of Neuroscience 35 (49): 16-46.

Good looking, intelligible English pronunciation: Better seen (than just) heard

One of the less obvious shortcomings of virtually all empirical research in second language pronunciation intelligibility is that is generally done using only audio recordings of learner speech--where the judges cannot see the faces of the subjects. In addition, the more prominent studies were done either in laboratory settings or in specially designed pronunciation modules or courses.

In a fascinating, but common sense 2014 study by Kawase, Hannah and Wang it was found that being able to see the lip configuration of the subjects, as they produced the consonant 'r', for example, had a significant impact on how the perceived intelligibility of the word was rated. (Full citation below.) From a teaching perspective, providing visual support or schema for pronunciation work is a given. Many methods, especially those available on the web, strongly rely on learners mirroring visual models, many of them dynamic and very "colorful." Likewise, many, perhaps most f2f pronunciation teachers are very attentive to using lip configuration, their own or video models, in the classroom.

What is intriguing to me is the contribution of lip configuration and general appearance to f2f intelligibility. There are literally hundreds of studies that have established the impact of facial appearance on perceived speaker credibility and desirability. So why are there none that I can find on perceived intelligibility based on judges viewing of video recordings, as opposed to just audio? In general, the rationale is to isolate speech, not allowing the broader communicative abilities of the subjects to "contaminate" the study. That makes real sense on a theoretical level, bypassing racial and ethnic and "cosmetic" differences, but almost none on a practical, personal level.

There are an infinite number of ways to "fake" a consonant or vowel, coming off quite intelligibly, while at the same time doing something very much different than what a native speaker would do. So why shouldn't there be an established criterion for how mouth and face look as you speak, in addition to how the sounds come out? Turns out that there is, in some sense. In f2f interviews, being influenced by the way the mouth and eyes are "moving" is inescapable.

Should we be attending more to holistic pronunciation, that is what the learner both looks and sounds like as they speak? Indeed. There are a number of methods today that have learners working more from visual models and video self recordings. That is, I believe, the future of pronunciation teaching, with software systems that provide formative feedback on both motion and sound. Some of that is now available in speech pathology and rehabilitation.

There is more to this pronunciation work than what doesn't meet the eye! The key, however, is not just visual or video models, but principled "lip service", focused intervention by the instructor (or software system) to assist the learner in intelligibly "mouthing" the words as well.

This gives new meaning to the idea of "good looking" instruction!

Full citation:
Kawase S, Hannah B, Wang Y. (2014). The influence of visual speech information on the intelligibility of English consonants produced by non-native speakers. J Acoust Soc Am. 2014 Sep;136(3):1352. doi: 10.1121/1.4892770.

Why haptic (pronunciation) teaching and learning should be superior!

Wow. How about this "multi-sensory" conclusion from Max-Planck-Gesellschaft researchers Mayer, Yildiz, Macedonia, and von Kriegstein, Visual and motor cortices differentially support the translation of foreign language words (full citation below)--summarized by Science daily (boldface added for emphasis) :

"The motor system in the brain appears to be especially important: When someone not only hears vocabulary in a foreign language, but expresses it using gestures, they will be more likely to remember it. Also helpful, although to a slightly lesser extent, is learning with images that correspond to the word. Learning methods that involve several senses, and in particular those that use gestures, are therefore superior to those based only on listening or reading."

The basic "tools" of haptic pronunciation teaching, what we call "pedagogical movement patterns," are defined as follows:

As a word or phrase is visualized (visual) and spoken with resonant voice, a gesture moving across the visual field is preformed which culminates in hands touching on the stressed syllable of the word or phrase (cognitive/linguistic), as the sound of the word is experienced as articulatory muscle movement in the upper body and by vibrations in the body emanating from the vocal cords and (to some degree) sound waves returning to the ears (auditory). 

Clipart'
Clker.com

And what bonds that all together? A 2009 study by Fredembach,et al demonstrated just how haptic anchoring--and the PMP should work: in relative terms, the major contribution of touch may generally be exploratory and assembling of multi-sensory experiences. The key is to do as much as possible to ensure that learners keep as many senses in play during "teachable moments" when new word-sound complexes are being encountered and learned. 

Make sense? Keep in touch!

Citations:

Fredembach, B., Boisferon, A. & Gentaz, E. (2009) Learning of arbitrary association between visual and auditory novel stimuli in adults: The “Bond Effect” of haptic exploration. PLoS ONE, 2009, 4(3), 13-20.

Max-Planck-Gesellschaft. (2015, February 5). Learning with all the senses: Movement, images facilitate vocabulary learning. ScienceDaily. Retrieved February 7, 2015 from www.sciencedaily.com/releases/2015/02/150205123109.htm

Revenge of the multi-taskers: Distracted during motor (or pronunciation) learning or practice? No problem!

This is the second in a series of posts on creating and managing effective language or pronunciation practice, (analogically) based on Glyde's guitar practice framework. (See earlier post.) His

Clip art:
Clker.com

principle #5 was common-sensical: Failing to avoid distraction.

Earlier posts have looked at the interplay between haptic (movement and touch) and visual and auditory modalities. One general finding of research has been that visual stimuli or input tend to override auditory and haptic. In part for that reason, we have worked to restrict extraneous visual auditory distraction during haptic pronunciation work. In therapy, on the contrary, many times distraction is used quite strategically to draw the patient's attention away from a problematic experience or emotion.

Now comes a fascinating study by Song and Bedard of Boston University (summarized by Science Daily - See full citation below) demonstrating how visual distraction during motor learning may at least not be problematic. As long as subjects were subjected to relatively similar distraction on the recall task, the fact that they had been systematically distracted during the learning task seemed to have little or no effect. Furthermore, if the "distracted" subjects were later tested in the "non-distracting" condition, they did not perform as well as their "distracted" fellow subjects.

In other words, the visual context of motor learning was not a factor in recall--as long as it was reasonably consistent with the original learning milieu.

So, what does all that mean for effective pronunciation practice? Quite a bit, perhaps. Context, from many perspectives is critical. Establishing linguistic context has been a given for decades; managing the classroom environment (or the homework practice venue) so that new or changed sounds are recalled in a "relatively similar setting" to how they were learned is another question.

One of the principles of haptic pronunciation teaching is to use systematic gesture + touch across the visual field to anchor sound change--maintaining as much of learner attention as possible for at least 3 seconds. In practice, the same pedagogical movement patterns (PMP) are used--and, according to learners, even in spontaneous later recall of new material the PMPs often figure prominently in visual/auditory recall as well.

So, to paraphrase Glyde's 5th principle: Avoid inconsistent distraction (in pronunciation teaching), at least in those more motor-based work or phases. Or better yet, embrace it!

Citation:
Brown University. (2014, December 9). Distraction, if consistent, does not hinder learning. ScienceDaily. Retrieved December 18, 2014 from www.sciencedaily.com/releases/2014/12/141209120141.htm

Seeing before believing: key to (haptic) pronunciation teaching

We call it "haptic pronunciation teaching." That is actually shorthand for something like:
simultaneous haptic-integration of visual-auditory-kinesthetic-tactile modalities in anchoring pronunciation

Almost every functional system for teaching pronunciation includes graphics or videos of some kind, even if it just a black and while line drawing of the mouth. Some substitute extensive written or verbal explanation for visual models. Our basic approach has been to use touch to link the senses, but often without too much concern for the precise order in which learner attention is directed through the various sources of information on the sound.

A fascinating new study on sensory sequencing in dance instruction by Blitz from Bielefeld University reported in Science Daily (see complete reference below) suggests that our real time sequencing in training in the use of pedagogical movement patterns (gesture, plus touch) is probably much more critical than we have assumed. That is especially relevant to how we (hapticians) maintain attention in the process. In other words, in the classroom, in what order do we introduce and train learners to the parameters of sounds and sound processes? That is, of course, equally relevant to all teaching!

NOTE: Please accept for the moment the parallel between dance instruction and our haptic work, that is training learners to experience, through gesture/touch and placement in the visual field, L2 or L1 sounds associated with targeted words. Also, allow me to side step the question of whether dancers are, by nature probably a bit "hyper-kinesthetic!"  

The study discovered that first viewing a dance sequence without verbal explanation or instruction--and then hearing or reading instructions after that was significantly more effective than the converse in long term memory for the sequence. Both visual and "cognitive" sources were present but the order was the critical variable. The subjects were apparently free to repeat both the visual and verbal inputs a limited number of times, but not to "mix" the ordering of them.

In other words, insight into what had been experienced was far more effective than was verbal cognitive schema in setting up and productively exploiting the visual experience or model to come. For us, the pedagogical implications are relatively clear, something like: (1) Observation (video clip) then (2) brief verbal explanation, then (3) experiential training in doing the gestural pattern, then (4) practice, along with (5) focused explanation of the context of the targeted sound.

How might that perspective impact your (pronunciation) teaching?

AMPISys, Inc.

See what I mean?






Full reference: Bielefeld University. "Best sensory experience for learning a dance sequence." ScienceDaily. ScienceDaily, 7 November 2014. .

Face it . . . Your pronunciation could look better!

According to research by Lander and Caper at the University of Lancaster, a little  more lipstick and work on your speech style may be in order. (Watched yourself on video lately when you ask a student "look at my mouth" as you provide a model?) Their study demonstrated unequivocally that your listeners' ability to understand you if they can see you can be enhanced considerably with a little tweaking. One feature that made words more easily understood, not surprisingly, was backing off from conversation style toward more declarative articulation, especially in times of potentially disruptive background noise. In addition, although other movement of facial muscles does play a supporting role or is synchronized with mouth and lip movement, it was the mouth that carried the functional load primarily. 

Clip art: Clker

This is a particularly interesting problem in haptic work, in part because the eyes of the student are naturally drawn to the hand and arm movements. Consequently, you must be a bit more conscientious about how you articulate a model word, for example, as you do the corresponding pedagogical movement pattern, to be sure that students can also read you lip patterning as well. Record some of your work, turn off the sound and spend a little time trying to figure out what you were saying . . . 

Obviously nothing to just "pay lip service to!" 

Citation: Investigating the impact of lip visibility and talking style on speechreading performance - http://dx.doi.org/10.1016/j.specom.2013.01.003

Can't see how to say it right? (Self-reflective, visual-soma-kinaesthetic correction of mispronunciation)

So you try to demonstrate with your face and mouth how a learner should be pronouncing a vowel, for example--and it simply does not work. In fact, the mispronunciation may just get worse. New research by Cook of City University London, Johnston of University College London, and Heyes of the University of Oxford (Summarized by Science Daily) may suggest why: visual feedback of the difference between one's facial gesture and that of a model can be effective in promoting accommodation; simple proprioceptic feedback (i.e., trying to connect up the correct model with the movements of the muscles in your face, without seeing what you are doing simultaneously) generally does not work very well. Amen, eh.

I have had students whose brains are wired so that they can make that translation easily, but they are the exception. The solution? Sometimes a mirror works "mirror-cles;" some new software systems (noted in earlier blogs) actually does come up with a computer simulation that attempts to show the learner what is going on wrong inside the mouth and what should be instead--with apparently very modest, but expensive results.

Clip art: Clker

Clip art: Clker

The EHIEP approach is to early on anchor the positioning and movement of the jaw and tongue to pedagogical movement patterns of the arms and hands. From that perspective, it is relatively easy, at least on vowels, stress and intonation (and some consonants) to provide the learner with both visual, auditory and proprioceptic feedback simultaneously, showing both the appropriate model and how the learner's version deviates. (In fact, in some correction routines, it is better to anchor the incorrect articulation first, before going to the "correct" one.) In effect, "(Only if) Monkey see (him or her mis-speak), (can) Monkey do (anything about it!)"

Anchoring pronunciation: Do you see what you are saying?

Clip art: Clker

Clip art: Clker

You can, in fact--if you are pronouncing a sound, word or phrase using EHIEP-like pedagogical movement patterns, PMPs (gestures across the visual field terminating in some form of touch by both hands.) Not only CAN you, according to research by Xi and colleagues at Northwestern University, summarized by Science Daily, but your eyes strongly interpret for you the "feeling of how it happens." The visual "character" of the dynamic gesture (its positioning, fluidity, distance from the eyes and texture on contact with the other hand) may well override the actual tactile feedback from your hands and proprioceptic "coordinates" of movement from your arms.

In the study, subjects were simultaneously presented with video clips that slightly contradicted what their hands and arms were doing. It was clearly demonstrated that even though subjects were also instructed to ignore the video and concentrate on the actual positioning, movement and related information about touch and weight coming from the hands, the "eyes have it." What they were seeing reinterpreted the other incoming sensory data.

As noted in earlier posts, visual can often override other modalities. What is "new" here and contributes to our understanding of how and why haptic-integration works is that the subjects' perception of the EHIEP sound-touch-movement "event" would appear to be strongly influenced by the style or flair or precision and consistency of the PMP. That has been one of key problems in creating the video models: insufficient clarity and consistency in the execution of PMPs (by me!)

This is both good news and bad news. Good, in that the PMP is, indeed, a potentially a very powerful anchor--and that the visual "feel" of each can contribute substantially to anchoring effectiveness. Bad, in that for maximal effectiveness the video/visual model needs to be exceedingly precise and consistent. (I have explored the use of Avatars instead of me but there are even bigger potential issues there.) Preparing/getting in shape now to do a new set of videos after the holidays, based on this and simular research. Can't wait to see what those feel like!