Context, AI voice technology, haiku and (pronunciation) teaching

Two studies just published and summarized by ScienceDaily.com together illustrate how critical context is to understanding and (pronunciation) teaching. One is on the impact of voice technology; the other, on AI-assisted or created Haiku. 

In the first, How voice technology influences what we reveal about ourselves, by Melzner, Bonezzi, and Meyvis, published originally in the Journal of Marketing Research, it was revealed that customers, not surprisingly, will reveal more about themselves, directly or indirectly when responding by speaking to automated systems, rather than interacting on the keyboard with text messaging. Some of that "revelation" is actually paralinguistic (voice characteristics such as pitch and pacing) and background sounds. In other words . . . context. 

In the Haiku study, Basho in the machine: Humans find attributes of beauty and discomfort in algorithmic haiku, by Ueda at Kyoto University Institute for the Future of Human and Society, discovered, basically, that subjects rated haiku created by humans collaborating with AI higher, in general, than Haiku created just by AI or humans, but, if they suspected the engagement of AI in the process, the ratings went down. (Exactly how subjects were prompted to check for that is not indicated, but just the very suggestion of possible AI "meddling" had to have a pervasive effect, undermining the validity of the study, potentially skewing the perceptions and expectations of the subjects . . . )

(Full disclosure, having lived in Japan for a decade, I came away with a great appreciation for haiku, such that it gradually became both my preferred genre for reading pleasure and poetic expression.) 

How does this all tie together with pronunciation teaching? In both studies, context is critical. In fact, haiku only works when the context is either provided in advance and often explained in great detail OR where the subjects have grown up in . . . well . . .Japan, where the form is encountered from infancy. Great haiku as an art form, itself, generally recreates context (or possible contexts) in the mind of the devotee/reader. In both cases, comprehension is grounded in context, not in the form, itself. 

So it is with pronunciation teaching as well. It is possible to work with pronunciation out of explicit context and the sounds or patterns be later available in spontaneous language use, but the treatment has to have almost "haiku-like" in salience for the learner. On the other hand, the immediate context of let's say attention to a consonant, as in the AI voice study, is encoded along with the targeted sound, e.g., the voice characteristics, including stress and disruptive performance markers of student and model, the room ambience, and the neurology and biomes of learners and instructors all--and all either enabling or disabling recall later. 

In other words, unless context, in several senses, is working for  you or being proactively generated in pronunciation work, the odds are not with you. For that reason, in part, the KINETIK method approach and others like it are designed to consistently embed pronunciation in regular, good course content or personally memorable, engaging narrative of some kind, where the chances of the focus of the work being remembered should be at least better, or at least with less clutter. 

O AI, eh? Aye!
O the story is long but . . . 
The tail is longer

Sources: 

American Marketing Association. (2022, November 30). How voice technology influences what we reveal about ourselves. ScienceDaily. Retrieved December 20, 2022 from www.sciencedaily.com/releases/2022/11/221130114606.htm

Kyoto University. (2022, December 2). Basho in the machine: Humans find attributes of beauty and discomfort in algorithmic haiku. ScienceDaily. Retrieved December 20, 2022 from www.sciencedaily.com/releases/2022/12/221201122920.htm

I like the way you move there! (Why haptic pronunciation teaching is so attractive!)

Do you like your students? Really? If you do, can they tell? If you don't, do they know? Do you like
teaching pronunciation? Does it show?

Clker.com

If your answer to any of those 6 questions is "I don't know . . . ,"  A meta-analytic investigation of the relation between interpersonal attraction and enacted behavior, by Montoya, Matthew; Kershaw and Prosser, summarized by Neurosciencenews.com, may be of interest. What they did is look at a bunch of studies, done on "hundreds" of cultures, trying to find universally recognized human behaviors that signal attraction (e.g., I like you!) Those nonverbal behaviors that (they claim) are universal are: 

• Smiling
• Eye contact
• Proximity (getting close in space)
• Laughter

Now, of course, how those behaviors are actually conveyed in different cultures may be quite different, but it is a fascinating claim. The summary goes on: 

"Other behaviors showed no evidence of being related to liking, including when someone flips their hair, lifts their eyebrows, uses gestures, tilts their head, primps their clothes, maintains open body posture or leans in." (Some of those at least intuitively seem to be related to attraction, at least in North American or Northern European cultures.)

One of the other, most striking findings (to me, at least) is that mimicking (or mirroring) and head nods were only associated with attraction in English. In other words, if your nonverbal messaging or expectations of students in the classroom relies to any extent on mirroring (of you or of your mirroring of them) or head nodding--and for the native English speaking instructor it certainly will to some degree--there can be a very real affective mismatch. 

Any native English speaker who has taught in Japan, for instance, can easily have their perception of audience engagement scrambled initially, when those in the audience sit (apparently) very still, with less body movement or mirroring, and nod heads for reasons other than just understanding or attraction. 

The intriguing implication of that research, in terms of haptic pronunciation teaching and training, is that both head nods and mirroring figure in very prominently in the teaching methodology, in effect making it perhaps even more "English-centric" than we had imagined. In most instances of modeling or correction of pronunciation, for example, a student "invited" to synchronize his or her upper body movement with the instructor or other students, as they repeat the targeted word, phrase or clause together. Likewise, upper torso movement in English and in the haptic system accompanies or drives head nodding, often referred to as upper torso nods, in fact.

In other words, the basic pedagogical process of haptic pronunciation work is, itself, "attractive," involving nonverbal "synchronization" of head and body in ways that enable acquisition of at least English. The only other language that we have done some work in to date is Spanish, but its "body language" is, of course, closely related to English. 

Even if you are not entirely "attracted" to haptic yet, this research certainly lends more support for the use of mirroring in English language instruction, especially pronunciation. (Nod if you agree!)



Source:
“A meta-analytic investigation of the relation between interpersonal attraction and enacted behavior” by Montoya, R. Matthew; Kershaw, Christine; & Prosser, Julie L. in Psychological Bulleting. Published May 8 2018. doi:10.1037/bul0000148

Teaching EnglishL2 advanced conversation (with hand2hand prosodic and paralinguistic "comeback")

Clker.com

We'll be doing a new workshop: "Pronunciation across the 'spaces' between sentences and speakers."  At the 2018 BCTEAL Conference here in Vancouver in May. Here is the summary:

This workshop introduces a set of haptic (movement + touch) based techniques for working with English discourse-level prosodic and paralanguistic bridges between participants in conversation, including, key, volume and pace. Some familiarity with teaching of L2 prosodics (basically: rhythm, stress, juncture and intonation) is recommended.

The framework is based to some extent on Prosodic Orientation in English Conversation, by Szczepek-Reed, and new features of v5.0 of the haptic pronunciation teaching system: Essential Haptic-interated English Pronunciation (EHIEP), available by August, 2018. The innovation is the use of several pedagogical movement patterns (PMPs) that help learners attend to the matches and mismatches of prosodics and paralanguage between participants in conversation that create and maintain coherence and . . . empathy across conversational turns.

For a quick glimpse of just the basic prosodic PMPs, see the demo of the AH-EPS ExIT (Expressiveness) from EHIEP v2.0.

The session is only 45 minutes long, so it will just be an experiential overview or tour of the set of speech-synchronized-gesture-and-touch techniques. The video, along with handouts, will be linked here in late May.

Join us!

Empathy for strangers: better heard and not seen? (and other teachable moments)

The technique of closing one's eyes to concentrate has both everyday sense and empirical research support. For many, it is common practice in pronunciation and listening comprehension instruction. Several studies of the practice under various conditions have been reported here in the past. A nice 2017 study by Kraus of Yale University, Voice-only communication enhances empathic accuracy, examines the effect from several perspectives.
😑
What the research establishes is that perception of the emotion encoded in the voice of a stranger is more accurately determined with eyes closed, as opposed to just looking at the video or watching the video with sound on. (Note: The researcher concedes in the conclusion that the same effect might not be as pronounced were one listening to the voice of someone we are familiar or intimate with, or were the same experiments to be carried out in some culture other than "North American".) In the study there is no unpacking of just which features of the strangers' speech are being attended to, whether linguistic or paralinguistic, the focus being:
 . . . paradoxically that understanding others’ mental states and emotions relies less on the amount of information provided, and more on the extent that people attend to the information being vocalized in interactions with others.
😑
The targeted effect is statistically significant, well established. The question is, to paraphrase the philosopher Bertrand Russell, does this "difference that makes a difference make a difference?"--especially to language and pronunciation teaching?
😑
How can we use that insight pedagogically? First, of course, is the question of how MUCH better will the closed eyes condition be in the classroom and even if it is initially, will it hold up with repeated listening to the voice sample or conversation? Second, in real life, when do we employ that strategy, either on purpose or by accident? Third, there was a time when radio or audio drama was a staple of popular media and instruction. In our contemporary visual media culture, as reflected in the previous blog post, the appeal of video/multimedia sources is near irresistible. But, maybe still worth resisting?
😑
Especially with certain learners and classes, in classrooms where multi-sensory distraction is a real problem, I have over the years worked successfully with explicit control of visual/auditory attention in teaching listening comprehension and pronunciation. (It is prescribed in certain phases of hapic pronunciation teaching.) My sense is that the "stranger" study actually is tapping into comprehension of new material or ideas, not simply new people/relationships and emotion. Stranger things have happened, eh!
😑
If this is a new concept to you in your teaching, close your eyes and visualize just how you could employ it next week. Start with little bits, for example when you have a spot in a passage of a listening exercise that is expressively very complex or intense. For many, it will be an eye opening experience, I promise!
😑

Source:
Kraus, M. (2017). Voice-only communication enhances empathic accuracy, American Psychologist 72(6)344-654.

On developing excellent pronunciation and gesture--according to John Wesley,1770.

Have just rediscovered Wesley's delightful classic "Directions Concerning Pronunciation and Gesture", a short pamphlet published in 1770. The style  that Wesley was promoting was to become something of the hallmark of the Wesleyan movement: strong, persuasive public speaking. Although I highly recommend reading the entire piece, here are some of Wesley's  (slightly paraphrased) "rules" below well worth heeding, most of which are as relevant today as were they then.

 Pronunciation

• Study the art of speaking betimes and practice it as often as possible.
• Be governed in speaking by reason, rather than example, and take special care as to whom you imitate.
• Develop a clear, strong voice that will fill the place wherein you speak.
• To do that, read or speak something aloud every morning for at least 30 minutes.
• Take care not to strain your voice at first; start low and raise it by degrees to a height.
• If you falter in your speech, read something in private daily, and pronounce every word and syllable so distinctly that they may have all their full sound and proportion . . . (in that way) you may learn to pronounce them more fluently at your leisure.
• Should you tend to mumble, do as Demosthenes, who cured himself of this defect by repeating orations everyday with pebbles in his mouth. 
• To avoid all kinds of unnatural tones of voice, endeavor to speak in public just as you do in common conversation.
• Labour to avoid the odious custom of spitting and coughing while speaking.

Gesture

• There should be nothing in the dispositions and motions of your body to offend the eyes of the spectators.
• Use a large looking glass as Demosthenes (again) did; learn to avoid all disagreeable and "unhandsome" gestures.
• Have a skillful and faithful friend to observe all your motions and to inform you which are proper and which are not.
• Use the right hand most, and when you use the left let it only be to accompany the other.
• Seldom stretch out your hand sideways, more than half a foot from the trunk of your body.
•  . . . remember while you are actually speaking you are not be studying any other motions, but use those that naturally arise from the subject of your discourse.
• And when you observe an eminent speaker, observe with utmost attention what conformity there is between his action and utterance and these rules. (You may afterwards imitate him at home 'till you have made his graces your own.)

 Most of the "gesture" guidelines and several of those for pronunciation are employed explicitly in public speaking training--and in haptic pronunciation teaching. Even some of the more colorful ones are still worth mentioning to students in encouraging effective speaking of all sorts. 

Good looking, intelligible English pronunciation: Better seen (than just) heard

One of the less obvious shortcomings of virtually all empirical research in second language pronunciation intelligibility is that is generally done using only audio recordings of learner speech--where the judges cannot see the faces of the subjects. In addition, the more prominent studies were done either in laboratory settings or in specially designed pronunciation modules or courses.

In a fascinating, but common sense 2014 study by Kawase, Hannah and Wang it was found that being able to see the lip configuration of the subjects, as they produced the consonant 'r', for example, had a significant impact on how the perceived intelligibility of the word was rated. (Full citation below.) From a teaching perspective, providing visual support or schema for pronunciation work is a given. Many methods, especially those available on the web, strongly rely on learners mirroring visual models, many of them dynamic and very "colorful." Likewise, many, perhaps most f2f pronunciation teachers are very attentive to using lip configuration, their own or video models, in the classroom.

What is intriguing to me is the contribution of lip configuration and general appearance to f2f intelligibility. There are literally hundreds of studies that have established the impact of facial appearance on perceived speaker credibility and desirability. So why are there none that I can find on perceived intelligibility based on judges viewing of video recordings, as opposed to just audio? In general, the rationale is to isolate speech, not allowing the broader communicative abilities of the subjects to "contaminate" the study. That makes real sense on a theoretical level, bypassing racial and ethnic and "cosmetic" differences, but almost none on a practical, personal level.

There are an infinite number of ways to "fake" a consonant or vowel, coming off quite intelligibly, while at the same time doing something very much different than what a native speaker would do. So why shouldn't there be an established criterion for how mouth and face look as you speak, in addition to how the sounds come out? Turns out that there is, in some sense. In f2f interviews, being influenced by the way the mouth and eyes are "moving" is inescapable.

Should we be attending more to holistic pronunciation, that is what the learner both looks and sounds like as they speak? Indeed. There are a number of methods today that have learners working more from visual models and video self recordings. That is, I believe, the future of pronunciation teaching, with software systems that provide formative feedback on both motion and sound. Some of that is now available in speech pathology and rehabilitation.

There is more to this pronunciation work than what doesn't meet the eye! The key, however, is not just visual or video models, but principled "lip service", focused intervention by the instructor (or software system) to assist the learner in intelligibly "mouthing" the words as well.

This gives new meaning to the idea of "good looking" instruction!

Full citation:
Kawase S, Hannah B, Wang Y. (2014). The influence of visual speech information on the intelligibility of English consonants produced by non-native speakers. J Acoust Soc Am. 2014 Sep;136(3):1352. doi: 10.1121/1.4892770.

The posture of (haptic pronunciation) teaching and learning

Especially if you are new to language learning--or a robot, here is fascinating study by Morse, Benitez, Belpaeme, Cangelosi, and Smith of Indiana university, "Posture Affects How Robots and Infants Map Words to Objects," summarized by ScienceDaily (See full citation below).

Basically what the research demonstrates is the role of body attitude (or orientation) in space in name and concept learning. From the summary:

"Using both robots and infants, researchers examined the role bodily position played in the brain's ability to "map" names to objects. They found that consistency of the body's posture and spatial relationship to an object as an object's name was shown and spoken aloud were critical to successfully connecting the name to the object."

And a quote from the lead author as to the implications of this line of research: 

"These experiments may provide a new way to investigate the way cognition is connected to the body, as well as new evidence that mental entities, such as thoughts, words and representations of objects, which seem to have no spatial or bodily components, first take shape through spatial relationship of the body within the surrounding world," . . .

In haptic pronunciation teaching (Essential Haptic-integrated English Pronunciation, EHIEP) we basically associate the sounds, words and patterns of language (English in this case) with specially designed gestures across the visual field, what we call 'pedagogical movement patterns' (PMPs).  We realized almost a decade ago that, at least for some learners (those that are more visually eidetic), the precision with which those models are presented and practiced initially is critical.

Studies in any number of "physical" disciplines, such as athletic training, rehabilitation psychotherapy have long established that principle, that where the new learning occurs in the visual field--and in the body--is integral to efficiency and effectiveness of learning. 

Of course the relevance of those studies goes far beyond learning pronunciation. Depending on your agenda and method, the "context of learning" extends out from the body to the concepts to the words, to the social milieu--even to the room. 

Sit up and take notice! (And join us at the TESOL Convention in Toronto this week on the 28th!)

Full citation:
Indiana University. "Robot model for infant learning shows bodily posture may affect memory and learning." ScienceDaily. ScienceDaily, 18 March 2015. .

(Haptic) pair-a-linguistic pronunciation teaching

Clip art: Clker

Saw a recent discussion thread that (incorrectly) identified gesture as a paralinguistic feature of speech. That term, paralanguage, typically refers to pitch, loudness, rate and fluency. Gesture or body movement may be synchronized with speech in a way that it can reflect some aspect of paralanguage, as in when arm gesticulating is coordinated with the stress or rhythm pattern, such that a baton-like gesture comes down on key points for emphasis in a lecture, etc.

Actually, I like that idea, combing or pairing (haptic-anchored) gesture with paralanguage. In EHIEP work we do something of that with pitch, rate and fluency, using special gestures terminating in touch, what we refer to as pedagogical movement patterns (PMPs), that function to control those three features of speech in various ways (typically done in either modelling or error correction or expressive oral reading of fixed texts). To see a demonstration of each go to the Demo page on the AH-EPS website:

pitch - the Expressiveness PMP
rate - the Rhythm Fight Club PMP
fluency - the Tai Chi Fluency PMP

We have yet to figure out an effective PMP for loudness. If you can think of one . . . give us a shout!

Keep in touch.

Mimic movement and pronunciation? Not always your cup of coffee!

Clip art: Clker

Clip art: Clker

That's right. Observed a great example of that recently at a conference. According to research by Ondobaka, de Lange, Wiemers, and Bekkering of University of Nijmegen, and Newman-Norlund of the University of South Carolina, summarized by Science Daily, it only works if your students have the same goals that you do: "If you and I both want to drink coffee, it would be good for me to synchronize my movement with yours  . . . but if you're going for a walk and I need coffee, it wouldn't make sense to be coupled on this movement level." Hmm. I can see where not being "coupled" might not facilitate a walk, but how about the impact of the same effect in pronunciation instruction, especially kinaesthetic or haptic-integrated work?

Previous blogposts have looked at a range of factors that may affect effectiveness of mirroring of pedagogical movement patterns, from personality, cognitive preferences and clutter in the visual field, to lack of achievable objectives. Orienting learners (and instructors) to why they should consistently "dance along with" the EHIEP  model on the video--or even the usual practice of mirroring videotaped conversations for fluency, is critical.

As one participant at our workshop commented, "This stuff is paradigm shifting!" (A common response, of course!)  Another participant, however, one who came in late and missed hearing the theory and "goals" of the workshop and left early, had a very different take. Later he told me apologetically--in part by his clearly unambiguous, uncoupled paralanguage over coffee--that it made "absolutely no sense, whatsoever" to him without "getting it all," up front. Q.E.D. (quod erat (non) demonstrandum), so to speak!

Hehehe! (De-stressing with high front tense vowels!)

Clip art: Clker

Clip art: Clker

This should be enough to make you smile--or at least make your students smile . . . 2009 research by Kraft and Pressman, reported by Psychological Science, appears to demonstrate that even faking a smile can be de-stressing. In pronunciation work we often tell students to smile as they pronounce high, front tense vowels, like in "Hehehe!" or when attempting to make the distinction between 's' (as in see) and 'sh' (as in she), slightly rounding the lips on the latter. I'm not saying that that is the best way to fix and s/sh problem necessarily, but sometimes it is helpful. I have also used something similar when working on word-final velar nasals, as in 'ing'. In effect just activating the muscles of the face to look and feel like a smile "orders" the brain to feel better, less stressed, physiologically. For the native speaker, we generally think of facial expression as being driven by attitude or "feelings." For nonnative speakers--and actors--it often happens in the opposite direction, taking on the paralanguage of the other results (if only temporarily) in changes in attitude and personna. Even if you don't believe that is the case, at least smile when you say it. 

Moving accents (a kinaesthetic, Hollywood approach)

Photo credit: DrLillianglass.com

Over a decade ago I had explored the concept of beginning by attempting to change the overall body posture and typical gestural patterns of learners to be more "English-like" (whatever that was going to mean!) In a few specific instances, where the ethnic nonverbal "accent" was markedly different from, for example, typical North American business presentation style, I had some success, but as I became more involved with haptic work, I gave up on trying to identify a more general body-language-based "target" for  learners.

I have read numerous accounts of how Hollywood approaches the problem, seeing accent or dialect and body representation as being almost inseparable. Dr Glass (and her blog) provide a hint (only) into the way it is done. (Note the list of her former clients!) At some point in the future, using virtual reality technology--with haptic interface, of course--we will be able to truly "train the body first!" For the time being, however, we must be satisfied with just a glimpse into the "Looking (at) Glass" . . .