Determine language of body of text?

[Mark B]

Does anyone have a method to determine the language (e.g. en-US, fr-FR,
zh-TW etc) of a body of text?

(In my case the body of text is an email received).

Naturally the text, if not English, may have a little bit of English. I'm
just trying to get the main language used.

[Mark B]

Re: Determine language of body of text?

http://code.google.com/apis/ajaxlanguage/documentation/#Detect

"Mark B" <none123@none.c omwrote in message
news:u$VCFVONJH A.456@TK2MSFTNG P06.phx.gbl...

Does anyone have a method to determine the language (e.g. en-US, fr-FR,
zh-TW etc) of a body of text?
>
(In my case the body of text is an email received).
>
Naturally the text, if not English, may have a little bit of English. I'm
just trying to get the main language used.

[MC]

Re: Determine language of body of text?

Each language has a few words that are extremely common, such as "the", "a", "an", "of", "for" in English. You could look for the 5 or 10 most common words in each language, and see which language wins.

To find the most common words, use a word-frequency-table program to analyze some text samples.

[rossum]

Re: Determine language of body of text?

On Thu, 23 Oct 2008 20:59:19 +1300, "Mark B" <none123@none.c omwrote:

>Does anyone have a method to determine the language (e.g. en-US, fr-FR,
>zh-TW etc) of a body of text?
>
>(In my case the body of text is an email received).
>
>Naturally the text, if not English, may have a little bit of English. I'm
>just trying to get the main language used.

As well as the other suggestions have a look at the characters used:
umlauts or the ss/beta character appear in German and acute or grave
accents in French.

Digram and trigram frequencies can also be a good indicator.

Essentially you are going to have to use some sort of statistical
method.

rossum

[JP]

Re: Determine language of body of text?

There's some code here that writes the internet headers of an email to
a text file, which you could then parse for the language. For example,
some email headers include a "Content-Type" line which indicates the
character set used.

i.e.: Content-Type: text/plain; charset=US-ASCII

http://blogs.technet.com/kclemson/archive/2006/10/27/quickly-view-the-internet-headers-of-a-message-in-outlook.aspx

HTH,
JP

On Oct 23, 3:59 am, "Mark B" <none...@none.c omwrote:

Does anyone have a method to determine the language (e.g. en-US, fr-FR,
zh-TW etc) of a body of text?
>
(In my case the body of text is an email received).
>
Naturally the text, if not English, may have a little bit of English. I'm
just trying to get the main language used.

[Dmitry Streblechenko]

Re: Determine language of body of text?

Look at the MailItem.Intern etCodepage property (corresponds to the
PR_INTERNET_CPI D MAPI propety).

--
Dmitry Streblechenko (MVP)

http://www.dimastr.com/

OutlookSpy - Outlook, CDO
and MAPI Developer Tool
-
"Mark B" <none123@none.c omwrote in message
news:u$VCFVONJH A.456@TK2MSFTNG P06.phx.gbl...

Does anyone have a method to determine the language (e.g. en-US, fr-FR,
zh-TW etc) of a body of text?
>
(In my case the body of text is an email received).
>
Naturally the text, if not English, may have a little bit of English. I'm
just trying to get the main language used.

[Mark B]

Re: Determine language of body of text?

Do all incoming emails have this? Even if they are not originating from an
Outlook client?

(It's for an Outlook 2007 Add-in C# BTW).


"Dmitry Streblechenko" <dmitry@dimastr .comwrote in message
news:%23$zmqwTN JHA.4172@TK2MSF TNGP06.phx.gbl. ..

Look at the MailItem.Intern etCodepage property (corresponds to the
PR_INTERNET_CPI D MAPI propety).
>
--
Dmitry Streblechenko (MVP)

http://www.dimastr.com/

OutlookSpy - Outlook, CDO
and MAPI Developer Tool
-
"Mark B" <none123@none.c omwrote in message
news:u$VCFVONJH A.456@TK2MSFTNG P06.phx.gbl...

>Does anyone have a method to determine the language (e.g. en-US, fr-FR,
>zh-TW etc) of a body of text?
>>
>(In my case the body of text is an email received).
>>
>Naturally the text, if not English, may have a little bit of English. I'm
>just trying to get the main language used.

>
>

[Dmitry Streblechenko]

Re: Determine language of body of text?

Most of them. But that really tells you more about the defaut code page of
the sender.

--
Dmitry Streblechenko (MVP)

http://www.dimastr.com/

OutlookSpy - Outlook, CDO
and MAPI Developer Tool
-
"Mark B" <none123@none.c omwrote in message
news:%23mASG5UN JHA.276@TK2MSFT NGP02.phx.gbl.. .

Do all incoming emails have this? Even if they are not originating from an
Outlook client?
>
(It's for an Outlook 2007 Add-in C# BTW).
>
>
"Dmitry Streblechenko" <dmitry@dimastr .comwrote in message
news:%23$zmqwTN JHA.4172@TK2MSF TNGP06.phx.gbl. ..

>Look at the MailItem.Intern etCodepage property (corresponds to the
>PR_INTERNET_CP ID MAPI propety).
>>
>--
>Dmitry Streblechenko (MVP)
>http://www.dimastr.com/
>OutlookSpy - Outlook, CDO
>and MAPI Developer Tool
>-
>"Mark B" <none123@none.c omwrote in message
>news:u$VCFVONJ HA.456@TK2MSFTN GP06.phx.gbl...

>>Does anyone have a method to determine the language (e.g. en-US, fr-FR,
>>zh-TW etc) of a body of text?
>>>
>>(In my case the body of text is an email received).
>>>
>>Naturally the text, if not English, may have a little bit of English.
>>I'm just trying to get the main language used.

>>
>>

>

[=?ISO-8859-1?Q?Arne_Vajh=F8j?=]

Re: Determine language of body of text?

Mark B wrote:

Does anyone have a method to determine the language (e.g. en-US, fr-FR,
zh-TW etc) of a body of text?
>
(In my case the body of text is an email received).
>
Naturally the text, if not English, may have a little bit of English.
I'm just trying to get the main language used.

If you are willing to write some code, then you can detect
the language (but probably not the regional dialect).

* dictionary with common words
* special letters (forward and backward accents, umlauts etc.)
* distribution of letters
* distribution of pairs of letters

Arne