Beefy Boxes and Bandwidth Generously Provided by pair Networks
Come for the quick hacks, stay for the epiphanies.
 
PerlMonks  

Re^9: Problem with regex wildcard operator (.)...?

by afoken (Chancellor)
on Sep 06, 2021 at 17:47 UTC ( [id://11136509]=note: print w/replies, xml ) Need Help??


in reply to Re^8: Problem with regex wildcard operator (.)...?
in thread Problem with regex wildcard operator (.)...?

lc() is rocket fast because this only involves fiddling with one bit within a byte.

That's only true for ASCII. For Unicode, things are quite different and will probably need some kind of table lookup. Just an example for the german language:

Upper Case CharacterCharacter CodeLower Case CharacterCharacter Code
ÄU+00C4äU+00E4
ÖU+00D6öU+00F6
ÜU+00DCüU+00FC
U+1E9EßU+00DF

And unfortunately, that's not all. The LATIN CAPITAL LETTER SHARP S, as ẞ is called in Unicode, is very new. First proposals are about 140 years old, but it became part of the standard german orthography in 2017, after becoming part of Unicode in v5.1.0 (April 2008) and of ISO/IEC 10646 (June 2008).

Before that (i.e. up to 2017), converting LATIN SMALL LETTER SHARP S to upper case resulted in converting it to SS or SZ. And to make things worse, the conversion depends on context. SZ is always ok, but considered harder to read. In general, SS was (and still is) used, except where it may cause misunderstandings. A common example for that problem is "Maße" (dimensions) vs. "Masse" (mass), both can be upper-cased to "MASSE". The meaning can often be derived from the context, to avoid misunderstandings, "Maße" would be upper-cased to "MASZE". Reversing that (i.e. implementing lc()) opens another can of worms, as other words may contain "ss" or "sz" that must not be converted to "ß". The LATIN CAPITAL LETTER SHARP S solves that problem nicely, lc("ẞ") is always "ß", no funny context rules needed.

See also https://en.wikipedia.org/wiki/%C3%9F#Development_of_a_capital_form.

BTW, swiss standard german solved that problem differntly: ß was completely replaced by "ss" in the 20th century (mainly between the 1940s and 1970s). "Maße" vs. "Masse", both written as "Masse", has to be resolved by context.

Alexander

--
Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)
  • Comment on Re^9: Problem with regex wildcard operator (.)...?

Replies are listed 'Best First'.
Re^10: Problem with regex wildcard operator (.)...?
by Marshall (Canon) on Sep 08, 2021 at 06:50 UTC
    Also, to the best of my knowledge, not having a capital ẞ creates another issue because the vowel before a double consonant is pronounced differently than a vowel before a single consonant (short vs long). I don't think that there is any idea of a "spelling bee" in German. For the most part, if you can say a word correctly, you can spell it.
      I don't think that there is any idea of a "spelling bee" in German. For the most part, if you can say a word correctly, you can spell it.

      For most words, yes. Exceptions are of course words from foreign languages. You may get little details wrong, like ä vs. e, s vs. ss. But in general, almost letters are spoken in Standard High German, and the mapping from spoken sounds to letters / letter combinations and vice versa is quite regular. https://en.wikipedia.org/wiki/German_orthography has more information. Stemming is important, see below.

      Some dialects are quite far away from Standard High German, changing both grammar and how letters are spoken. A common joke for Saxonian dialects is "(some word) with a hard b", because "b" and "p" are both spoken as "b".

      In 1996, ortography was reformed (see https://en.wikipedia.org/wiki/German_orthography_reform_of_1996), simplifying some cases, use of ß was reduced ("daß" was changed to "dass", "muß" to "muss"), and foreign words were changed to be written according to german rules (e.g. "Delfin" instead of "Delphin").

      The reform caused a lot of public debate, many people thought (and still think) it oversimplified, and changes what previously was an error the official rule, while at the same time deprecating the old rules. One of my favorite examples is "aufwendig" (elaborate, costly, complex, opulent), spelled "aufwändig" according to the new rules. "ä" and "e" sound exactly the same here. And here is where stemming comes into play:

      (1) "aufwendig" (spelled according to old rules) is the adjective derived from the verb "aufwenden" (to spend, to expend). From the same verb, you get the nouns "Aufwand" and "Aufwendung" (efford, cost, sometimes also overhead). Note that the noun "Aufwand" has nothing to do with "auf Wand" (on/onto (a) wall), it just uses the same letters in the same order, but in a single word. <Update>"Aufwand" is based on the past tense form of the verb, hence the "a" instead of "e".</Update>

      (2) A very similar adjective is "wendig" (agile, maneuverable), derived from "wenden" (to turn). The noun derived from "wenden" is "Wende" (turn, turnaround). It may seem like that this is the stem of "aufwenden", as "auf-" is a common prefix meaning "on" or "onto", but to my understanding of my native language, this is wrong, just a coincidence. "on" or "onto" makes no sense in "aufwenden". Also, there is no turning involved in "aufwenden".

      (3) A third adjective is "ebenerdig", composed of "eben" (flat) "Erde" (ground, the "e" is removed due to the suffix) and the suffix "-ig" (roughly meaning "like") to make a word an adjective. It literally means that something is on flat ground, especially absense of stairs and ramps.

      Look at the old spelling: "aufwendig" - strip "-ig" and you get the stem of the verb "aufwenden". Note that the "e" from the verb stays. Compare with the new spelling: "aufwändig" - strip "-ig" and you get a false stem "aufwand" ("ä" usually stems from "a"). There is no verb "aufwand" in German, so that must still be a composed word, so you need to split into the commonly used prefix "auf-" and the stem "Wand". And now apply what you have learned in the previous three paragraphs: "auf-wänd-ig" must mean something onto-wall-like, something like something else placed onto a wall (think of wallpapers). And that is plain nonsense. But since 1996, that nonsense is official ortography taught in schools and used in all official texts.

      Alexander

      --
      Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)
        In 1996, ortography was reformed (see https://en.wikipedia.org/wiki/German_orthography_reform_of_1996), simplifying some cases, use of ß was reduced ("daß" was changed to "dass", "muß" to "muss"), and foreign words were changed to be written according to german rules (e.g. "Delfin" instead of "Delphin").

        ORLY? 1996? boy, time is passing, I'm getting old - 27 years ago! and I'm still stubbornly sticking to ye olde rules.

        One of my favorite examples is "aufwendig"

        Hell yes, reason for "Bier in Massen trinken" (in Maßen = in moderation, in Massen = galore) :P

        perl -le'print map{pack c,($-++?1:13)+ord}split//,ESEL'
        Interesting! I lived in Munich from 1990-1995. So these changes happened after I left. I took a lot of German classes at the Volkshochschule. Surprisingly, I learned a lot about English grammar. The various parts of speech are a lot easier to identify in German and that knowledge helped me to understand English better.

        I basically "flunked" in the speech laboratory. In order to say a German vowel correctly, the tongue moves to a position within the mouth and never moves. To say an English vowel correctly, the tongue must always move. This difference between "never" and "always" was too great for me to overcome.

        I did have one small victory. The German word for squirrel is Eichhörnchen. I am not sure of the spelling, but the Bavarian word is Ohrkatzelschworf. That contains phonemes that don't exist in High German. I can say this word, and a guy from Hamburg has no chance. My English trained agile tongue can come very close to what a native Bavarian would say.

        I guess we are off topic right now. But the discussion about how to map the Sharp S to the English keyboard is relevant. Thanks!

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11136509]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others studying the Monastery: (7)
As of 2024-04-18 08:22 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found