Language Technologies
in e-Commerce

Lecture #2
(March 16, 2001)

Language Representation
and Cultural Issues

Displaying Text

How do you get "", Pounds Sterling ("£") or the Euro symbol "" to display on the user's screen?

•	insert a bitmap (GIF, JPEG, PNG, etc.)
•	approximate with ASCII characters:  O(+> 

That's OK for an isolated symbol or two,
but what about an entire page in a different alphabet:
Запрошенный вами документ не найден.

•	find and use a character set which includes the desired character

Meaning vs. Appearance

There is a distinction between the abstract
characters and their graphical representations (glyphs):
a a a a a a a a a a a a a a a a a a
a a a a a a a a a a a a a a a a a a
a a a a a a a a a a a a a a a a a

But due to the historical development of alphabets, sometimes the same glyph is actually two different characters:

Latin  'M'   --> lowercase is 'm'
Greek 'M'   --> lowercase is 'µ'

Alphabets also have directionality -- is it written from left to right or right to left?

So, Which Character Set?

"The nicest thing about standards is that there are so many to choose from."
		-- Ken Olsen

Baudot
ANSI X3.4-1968 (aka ASCII)
EBCDIC (at least 16 variants!)
IBM PC codepage 437, codepages 850-857
Microsoft Windows codepage 1250
ISO 646-x (25 variants!)
DIN 66003 (aka ISO 646-DE)
CSA Z243.4-1985-x (1/2/gr)
Cyrillic: KOI-8, ISO 5427
Japanese: JIS, Shift-JIS, ISO 2022-JP, EUC-JP
Chinese: GB-2312, Big5, ISO 2022-CN, EUC-CN
Korean: EUC-KR, ISO 2022-KR
Vietnamese: TCVN 5773:1993
ISO 8859-x (1/2/3/4/5/6/7/8/9/10/13/14/15)
Unicode
ISO 10646

Unicode -- All-Inclusive?

Of course not!  There are far more than the 49,000+ characters defined by Unicode v3.0 in the various languages of the world.

Dozens of alphabets are covered, but dozens more have been proposed for inclusion in future versions.

Covered: Latin alphabets, Greek, Cyrillic, IPA, Armenian, Hebrew, Arabic, Syriac, Thaana (Maldives), Devanagari (Hindi/Sanskrit), Bengali, Gurmukhi (Punjab), Gujarati, Oriya, Tamil, Telugu, Kannada, Malayalam, Sinhala, Thai, Lao, Tibetan, Myanmar (Burmese), Georgian, Hangul Jamo, Ethiopic, Cherokee, Canadian Aboriginal, Ogham (archaic Irish), Runic, Khmer, Mongolian, unified CJK radicals and ideographs, Kangxi radicals, Hiragana, Katakana, Bopomofo, Kanbun, Yi syllables, Currency Symbols, Mathematical Operators, OCR, Dingbats, Braille.

Proposed: Avestan (ancient Iranian), Egyptian Hieroglyphics, Brahmi, Buginese (Indonesia), Byzantine Musical Symbols, Cham (Vietnam), Cirth (JRR Tolkien), Coptic, Cypriot Syllabary, Etruscan, Glagolitic, Gothic, Javanese, Linear B, Meroitic (ancient Sudan), Old Hungarian Runic, Old Permic, Old Persian Cuneiform, Phillipine Scripts, Phoenician, Pollard (southeast Asia), Rong (Tibetan), Shavian, Sinaitic (ancient Semitic), South Arabian, Soyombo, Tai scripts, Tengwar (JRR Tolkien), Tifinagh (Berber), Klingon script, Ugaritic Cuneiform, Western Musical Symbols.

OK, so we'll soon have more than 65,000 characters defined.  Fortunately, Unicode already defines an extension mechanism to allow an additional 1,048,576 characters (and ISO 10646 uses 31-bit characters -- over 2 billion).

Transfer Encodings

It is often necessary to use an encoding different from the actual character set when transferring data.

Examples:
•	MIME "Base64"
•	Unicode UTF-7, UTF-8
•	ISO 2022

Fonts

Even if we have selected an appropriate character set and transfer encoding to get the text to the user, the user might not be able to actually see it!

Display fonts don't always contain glyphs for every character defined in the character set they purport to represent.  This is particularly the case for Unicode with its extremely large repertoire.

Entering Text

Chinese has thousands of characters -- it's not practical to build a keyboard with one key for every character!  (Although someone did once build a typewriter with the thousand or so most common characters)

So, syllabic and ideographic languages use input methods that compose the characters from smaller units (radicals or letters). 

Korean characters represent syllables of one through four letters (Jamo), with rigid constraints on ordering and positioning.  So, one can enter Korean by typing the Jamo and have software compose them appropriately.

Japanese Input

Phonetic Alphabets: Kana (Katakana and Hiragana)

Ideographic: Kanji and Kokuji
•	borrowed from Chinese in three waves between 222 and 1279 AD

Kana can be entered either by mapping the keyboard to kana characters or by entering the transliteration in Latin letters from a standard keyboard.

Kanji input is more complicated....  A conversion dictionary is required, which tells the software how each kanji character can be decomposed into kana -- but this is not always unique!

Chinese Input

Phonetic Alphabet: Zhuyin (aka bopomofo) or Pinyin (Latin letters)

Ideographs: Hanzi
•	85,000 in recent reference works

Korean Input

Phonetic syllables: Hangul (composed of up to four Jamo).

Some Chinese-derived ideographs: Hanja
•	rarely used in modern Korean, but the Korean Ministry of Education has required a basic set of 1800 hanja for students to learn

Some Korean-made Chinese-like characters: Gugja

Input Methods

The software that permits keyboard entry is called an Input Method Editor or FEP (front-end processor).

Usually a separate process that pastes its result into the user's application.

Ways to enter a character:
•	phonetically (by reading) -- can be highly ambiguous
•	by radical
•	by number of strokes
•	by stroke shapes
•	by corners
•	by overall shape
•	by multiple criteria

Input by Radical

Chinese characters can be decomposed into about 200 radicals (214 for traditional, 186/188 for simplified Chinese)
•	originated in a 1716 dictionary, and still used to index characters in dictionaries

Various input methods use the name, number of strokes, or index of the radical to specify which radical is desired.

But the radical used for indexing is not always obvious.

Input by Stroke Count

User enters number of strokes in the character, and the software displays a menu of possible
matches.

But for some characters, the number of strokes is ambiguous or may depend on the font!

Input by Corner

Works by categorizing the shapes present at each corner of the character.
•	Four Corner code commonly used to locate characters in Chinese dictionaries
•	a Three Corner code devised specifically for text entry

Input by Structure
and Reading

Goal: Reduce number of candidates by using both visual structure and sound of character.
•	Tze-loi Method: three keystrokes, specifying the upper-left and lower-right corner shapes and the first sound in the character's reading
•	Renzhi codes: typically consist of first letter of Pinyin reading and first and last strokes, but can also specify other elements with either two or four keystrokes.

Input by Postal Code

(For Japanese) User enters Japanese postal code and software supplies a list of the place names associated with the postal code

Input by Association

(aka two-stroke input method)

An older Japanese input method which associates two characters (normally kana) to a single kanji, usually by reading or meaning.

Multilingual Web Pages

Easiest way to produce a web page containing text in multiple languages: use a character set which supports all the characters used by those languages.

Other alternative: provide a means for instructing the receiver's software to switch character sets or fonts on the fly (ISO 2022,
HTML "font" tag, etc.).

Internationalized Pages

Rather than having to search for a link from an English home page to a page in the user's language, why not automatically present the page in the user's language?

Browsers can be set to indicate a preferred language when requesting a web page; a properly configured server will then provide the correct language for each user.

But, few web sites make use of this capability, and even fewer get it entirely right.

Cultural Issues:
Language Differences

If an online store has a "football" in its catalog, what exactly is it selling?

                                                      

Great Britain and the United States are nations separated by a common language. 
		-- George Bernard Shaw

Marketing Fiascoes

The difference between the right word and almost the right word is the difference between lightning and the lightning bug. 
		-- Mark Twain

A Parker Pen's advertising slogan was translated into Spanish as "use this pen if you want to avoid having babies.".... 
http://www.snopes.com/business/misxlate/parker.htm 

In China, foreign names are written using words having similar sounds.  Coca-Cola became "bite the wax tadpole" in (unofficial) transliterations by Chinese shopkeepers....
http://www.snopes2.com/cokelore/tadpole.htm

Misleading Connotations

I love Americans, but not when they try to talk French. What a blessing it is that they never try to talk English. 
		-- Saki (H. H. Munro)

Would you want to drink "Pocari Sweat"?

(It's a Japanese sports drink akin to Gatorade)

More generally, since first impressions count especially highly on the web, you want to use proper and acceptable language and terminology.

Formality

Many languages have different levels of formality, and the words one uses change with the level of formality ("register").

In German, "you" (singular) is either "Sie" or "Du"; in Spanish, either "usted" or "tú". But the pronoun isn't the only word that changes between registers.

English also has formal and informal registers, but nobody has used the formal register in everyday speech for centuries:
		Thou shalt not ....

Politeness

Japanese: different forms of address depending on whether the speaker is of higher, lower, or equal social status.

In many languages, the choice of words depends on who the speaker is, e.g. “male speech” vs. “female speech”, speaker’s status or role, etc.

Avoiding Offense

References

Unicode: A Primer
Tony Graham
M&T Books (2000), ISBN 0-7645-4625-2

CJKV Information Processing
Ken Lunde
O'Reilly (1999), ISBN 1-56592-224-7