Cued Speech

• In 1966, R. Orin Cornett invented the Cued Speech

• Cued Speech is a communication system developed for deaf people to complement speechreading at the phonetic level with hands

In combination with the mouth movements of speech, it makes the phonemes of spoken language look different from each other.

• Used to acquire skills in reading, in lip reading and oral communication

• Increasingly popular: has been adapted for more than 65 languages

A visual communication mode

• Using handshapes in different placements near the face, ‘CV’ syllables are coded:

  • handshapes to represent a consonant ‘C’
  • hand position on the face to represent a vowel ‘V’

• Each time a speaker pronounces a ‘CV’ or ‘-V’ syllable, a cue is produced

• Other syllabic structures are produced with several cues

  • Example: ‘CCV’ is made of the 2 cues ‘C-’ + ‘CV’

French Cued Speech

Naming convention - Consonants

• we added (0) for the neutral shape

French Cued Speech

Naming convention - Vowels

• we added (n) for the neutral position

French Cued Speech

Representation

Screenshots

Description

• an HD video quality of the whole speaker;

• 23 different participants, some are CS certified and some are not;

• 10 different topics, each one read by 2 or 3 participants;

• 4 different sessions in each topic:

  • 32 isolated syllables,
  • 32 isolated words or phrases,
  • 7 up to 10 isolated sentences,
  • a text.

• CC-BY-NC-4.0 license

  • https://www.ortolang.fr/market/corpora/clelfpc/

Annotations are under construction. Already available: the 4 sessions of 5 topics time-aligned at the phonetic level and produced keys

Context

• No existing automatic CS prediction system

• Long-term project: creating an automatic Text-to-Cued system

  • Step 1: what? Fix the key from phonemes
  • Step 2: when? Fix the moments to start/stop producing the key
  • Step 3: where? Fix the hand trajectory

This study copes with time-aligned phonemes as input and produces an output with the cue names and their corresponding segmentation

The “what” question: algorithm

1. Time groups (TG) segmentation from the time-aligned phonemes
   • TG = an inter-break group where a break is a pause or any sound but a phoneme (laugh, noise, breath, etc)
2. Each phoneme of TG is then turned into its class: either labelled with C or V.
3. Analyse the sequence from left to right to made the keys:
   • Special rules are implemented for atypical class combinations such as VC, C, CC and CVC, instead of the regular ’CV’ that makes a key
4. When the sequence of class labels of a TG is segmented, we turn back the sequence of classes into phonemes. Each phoneme label is then mapped to its key code

Grammar of a CS key

We developed a grammar corresponding to these rules and implemented this grammar in software a deterministic finite automata (DFA).

Example

• Phonemes: 9~ d @ m i p o # d H i l d @ k o k o
• Time groups:
  • TG1: 9~ d @ m i p o
  • TG2: d H i l d @ k o k o
• Classes:
  • TG1: V C V C V C V
  • TG2: C C V C C V C V C V
• Segmentation:
  • TG1: -V.CV.CV.CV
  • TG2: C-.CV.C-.CV.CV.CV
• Phonemes:
  • TG1: 9~.d@.mi.po
  • TG2: d.Hi.l.d@.ko.ko
• Key codes:
  • TG1: 5t.1s.5m.1s
  • TG2: 1s.4m.6s.1s.2s.2s

Result: annotation layers

Number of keys

Quantitative analysis

• insertion The cuer added 8 keys compared to the predicted ones;
• deletion The cuer did not code 47 keys compared to the predicted ones;
• substitution The cuer and the prediction system coded 183 keys differently.

The number of inserted and deleted keys is very small relatively to the number of substitutions, and almost anecdotal relatively to the corpus size.

Quantitative analysis by speaker

Qualitative analysis of the 8 insertions

• 3 errors of the speaker

• 5 differences due to the “liaison” phenomena. Example:

  • “pour un” is phonetized /puR9~/
  • the automatic system is segmenting in 2 keys for: /pu/ + /R9~/
  • the speaker is segmenting in 3 keys for: /pu/ + /R/ + /9~/

Qualitative analysis of the 47 deletions

• 41 are ’C’ and 6 are ’CV’

• two speakers are significantly un-coding the consonants: 18 deletion for LM and 16 for CH

• The most frequently un-coded consonants are:

  • /t/ (9 times)
  • /R/ (8 times)
  • /p/ (5 times)
  • /l/ (4 times)

Qualitative analysis of the 183 substitutions

• 16 are ’C’ (8.7 %)

• 22 are ’V’ (12 %)

• 145 are ’CV’ (79.2 %)

  • 1 substitutes both the shape and the position;
  • 6 substitute the shape only;
  • 138 substitute the position only.

Substitutions mostly (87.4 %) concern the position (the vowel) rather than the handshape (the consonant).

Qualitative analysis

Vowel substitution: /@/

• /@/ is 88 times (48 %) at position (b) instead of (s)

• The (b) position is the one of the vowel /2/

  • but /2/ is never coded at (s) position like /@/

• Two cuers are significantly coding /@/ at (b) position: VT 45 times and LM 40 times

• A few words are concerned: “de”, “le”, “ne” and “que”

Qualitative analysis

Vowel substitution: /e/

• /e/ is 32 times at position (c) instead of (t)
• The (c) position is the one of the vowel /E/
• Two speakers are mostly coding this way: 18 times VT and 9 times CH
• It affects some words more that others but not systematically

Qualitative analysis

Vowel substitution: /9~/

• /9~/ is 17 times coded at position (b) instead of (t)
• The (b) position is the one of the vowel /e∼/
• This difference is mainly observed in the word un of CH speaker (12 times)

Conclusion

• an automatic system to predict Cued Speech keys from phonemes.
• an analysis of the differences between the predicted keys and the produced ones

Next work will focus on the analysis of duration and timing of the sequences of cues in the time-groups and on the temporal and spatial organization of the code in its speech co-production.

Future work: Proof of concept