这是indexloc提供的服务，不要输入任何密码
Skip to content

Remove legacy transliterator #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: 6.9
Choose a base branch
from
Open

Remove legacy transliterator #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
2f46b4b
Fix namespaces in the Pimcore 6 upgrade docs
jdreesen Jan 31, 2022
86a86b1
Apply php-cs-fixer changes
brusch Feb 1, 2022
8c24ce5
fix: Remove legacy transliterator
mugge6 Jun 14, 2022
5a3ec1d
fix: Fix umlaut transliteration to ICU 60.x
mugge6 Jun 15, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions ...evelopment_Documentation/23_Installation_and_Upgrade/09_Upgrade_Notes/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ If you have custom editables or wysiwyg global config then please change namespa
- [Ecommerce] Deprecated FactFinder integration and will be removed in Pimcore 10.
- Saving unpublished data objects via API will not throw Validation exceptions anymore (just like Admin UI). Please set `omitMandatoryCheck` explicitly to `false` to force mandatory checks.
- `\Pimcore\DataObject\GridColumnConfig\Operator\ObjectBrickGetter` operator is deprecated and will be removed in Pimcore 10
- Calling `Pimcore\Model\DataObject\ClassDefinition\Data::isEqual()` is deprecated since version 6.7.0 and will be removed in version 7 . Implement `\Pimcore\Model\DataObject\ClassDefinition\Data\EqualComparisonInterface` instead.
- Calling `Pimcore\Model\DataObject\ClassDefinition\Data::isEqual()` is deprecated since version 6.7.0 and will be removed in version 10 . Implement `\Pimcore\Model\DataObject\ClassDefinition\Data\EqualComparisonInterface` instead.
- Following properties and methods are deprecated to unify document editables and will be removed in Pimcore 10. [#6900](https://github.com/pimcore/pimcore/pull/6900):
- `PageSnippet::$elements`. Use `PageSnippet::$editables` instead.
- `PageSnippet::$inheritedElements`. Use `PageSnippet::$inheritedEditables` instead.
Expand Down Expand Up @@ -230,10 +230,10 @@ pimcore:

## 6.4.0
- Deprecated the REST Webservice API. The API will be removed in Pimcore 10, use the [Pimcore Datahub](https://github.com/pimcore/data-hub) instead.
- Removed `Pimcore\Bundle\EcommerceFrameworkBundle\PricingManagerPricingManagerInterface::getRule()` and `Pimcore\Bundle\EcommerceFrameworkBundle\PricingManager::getRule()`
- Removed `Pimcore\Bundle\EcommerceFrameworkBundle\PricingManager\PricingManagerInterface::getRule()` and `Pimcore\Bundle\EcommerceFrameworkBundle\PricingManager\PricingManager::getRule()`
- The `DocumentRenderer::setEventDispatcher()` method has been removed. Pass event dispatcher to the constructor instead.
- `RedirectHandler::setRequestHelper()` and `RedirectHandler::setSiteResolver()` methods have been removed. Pass instance of `Pimcore\Http\RequestHelper` & `Pimcore\Http\Request\Resolver\SiteResolver` to the constructor instead.
- The `ContainerService::setEventDispatcher()` method has been removed and DocumentRenderer event listeners moved to`Pimcore\Bundle\CoreBundle\EventListener\FrontendDocumentRendererListener`
- The `ContainerService::setEventDispatcher()` method has been removed and `DocumentRenderer` event listeners moved to`Pimcore\Bundle\CoreBundle\EventListener\FrontendDocumentRendererListener`
- Ecommerce: max length of `cartId` is now `190` characters instead of `255`
- MaxMind GeoIP database is **not** updated automatically anymore, please read the [instructions](../../18_Tools_and_Features/37_Targeting_and_Personalization/README.md) for setting up geo support for targeting.

Expand All @@ -249,7 +249,7 @@ pimcore:
full_page_cache:
...
```
in system.yml to avoid conflicts between output and data cache [#5369](https://github.com/pimcore/pimcore/issues/5369). If you are using custom config files then you have to migrate them manually. Also new config `pimcore:full_page_cache` is disabled by default, so you have to enable fullpage cache again in system settings.
in `system.yml` to avoid conflicts between output and data cache [#5369](https://github.com/pimcore/pimcore/issues/5369). If you are using custom config files then you have to migrate them manually. Also new config `pimcore:full_page_cache` is disabled by default, so you have to enable fullpage cache again in system settings.
- Properties `$children`, `$hasChildren`, `$siblings`, `$hasSiblings` in `Pimcore\Model\Document` & `$o_children`, `$o_hasChildren`, `$o_siblings`, `$o_hasSiblings` in `Pimcore\Model\AbstractObject` uses array to cache result.

## 6.3.0
Expand Down
4 changes: 2 additions & 2 deletions lib/Mail.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -419,7 +419,7 @@ public function getParams()
/**
* Returns a parameter which was set with "setParams" or "setParam"
*
* @param string | integer $key
* @param string|int $key
*
* @return mixed
*/
Expand Down Expand Up @@ -457,7 +457,7 @@ public function unsetParams(array $params)
/**
* Deletes a single parameter which was set with "setParams" or "setParam"
*
* @param string | integer $key
* @param string|int $key
*
* @return \Pimcore\Mail Provides fluent interface
*/
Expand Down
203 changes: 1 addition & 202 deletions lib/Tool/Transliteration.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,210 +27,9 @@ class Transliteration
*/
public static function toASCII($value, $language = null)
{
// the transliteration is based on the locale
// äüö is in EN auo in DE aeueoe
if (!$language) {
$locale = \Pimcore::getContainer()->get('pimcore.locale')->findLocale();
if ($locale) {
$language = \Locale::getPrimaryLanguage($locale);
}

if (!$language) {
$language = 'en'; // default is "en"
}
}

$value = self::_transliterationProcess($value, '~', $language);
$value = transliterator_transliterate('Any-Latin; Latin-ASCII; [^\u001F-\u007f] remove', $value);
$value = transliterator_transliterate('Any-Latin; de-ASCII; [^\u001F-\u007f] remove', $value);
$value = trim($value);

return $value;
}

/**
* The following functions are based on:
* http://drupal.org/project/transliteration
* http://sourceforge.net/projects/phputf8/
*
* The data in Transliteration/Data is a port of Text::Unidecode
* Data was transformed to PHP by the libs above
* http://search.cpan.org/~sburke/Text-Unidecode-0.04/lib/Text/Unidecode.pm
*/

/**
* @static
*
* @param string $string
* @param string $unknown
* @param string|null $source_langcode
*
* @return string
*/
private static function _transliterationProcess($string, $unknown = '?', $source_langcode = null)
{
// ASCII is always valid NFC! If we're only ever given plain ASCII, we can
// avoid the overhead of initializing the decomposition tables by skipping
// out early.
if (!preg_match('/[\x80-\xff]/', $string)) {
return $string;
}

static $tail_bytes;

if (!isset($tail_bytes)) {
// Each UTF-8 head byte is followed by a certain number of tail bytes.
$tail_bytes = [];
for ($n = 0; $n < 256; $n++) {
if ($n < 0xc0) {
$remaining = 0;
} elseif ($n < 0xe0) {
$remaining = 1;
} elseif ($n < 0xf0) {
$remaining = 2;
} elseif ($n < 0xf8) {
$remaining = 3;
} elseif ($n < 0xfc) {
$remaining = 4;
} elseif ($n < 0xfe) {
$remaining = 5;
} else {
$remaining = 0;
}
$tail_bytes[chr($n)] = $remaining;
}
}

// Chop the text into pure-ASCII and non-ASCII areas; large ASCII parts can
// be handled much more quickly. Don't chop up Unicode areas for punctuation,
// though, that wastes energy.
preg_match_all('/[\x00-\x7f]+|[\x80-\xff][\x00-\x40\x5b-\x5f\x7b-\xff]*/', $string, $matches);

$result = '';
foreach ($matches[0] as $str) {
if ($str[0] < "\x80") {
// ASCII chunk: guaranteed to be valid UTF-8 and in normal form C, so
// skip over it.
$result .= $str;

continue;
}

// We'll have to examine the chunk byte by byte to ensure that it consists
// of valid UTF-8 sequences, and to see if any of them might not be
// normalized.
//
// Since PHP is not the fastest language on earth, some of this code is a
// little ugly with inner loop optimizations.

$head = '';
$chunk = strlen($str);
// Counting down is faster. I'm *so* sorry.
$len = $chunk + 1;

for ($i = -1; --$len;) {
$c = $str[++$i];
if ($remaining = $tail_bytes[$c]) {
// UTF-8 head byte!
$sequence = $head = $c;
do {
// Look for the defined number of tail bytes...
if (--$len && ($c = $str[++$i]) >= "\x80" && $c < "\xc0") {
// Legal tail bytes are nice.
$sequence .= $c;
} else {
if ($len == 0) {
// Premature end of string! Drop a replacement character into
// output to represent the invalid UTF-8 sequence.
$result .= $unknown;

break 2;
} else {
// Illegal tail byte; abandon the sequence.
$result .= $unknown;
// Back up and reprocess this byte; it may itself be a legal
// ASCII or UTF-8 sequence head.
--$i;
++$len;

continue 2;
}
}
} while (--$remaining);

$n = ord($head);
$ord = null;
if ($n <= 0xdf) {
$ord = ($n - 192) * 64 + (ord($sequence[1]) - 128);
} elseif ($n <= 0xef) {
$ord = ($n - 224) * 4096 + (ord($sequence[1]) - 128) * 64 + (ord($sequence[2]) - 128);
} elseif ($n <= 0xf7) {
$ord = ($n - 240) * 262144 + (ord($sequence[1]) - 128) * 4096 + (ord($sequence[2]) - 128) * 64 + (ord($sequence[3]) - 128);
} elseif ($n <= 0xfb) {
$ord = ($n - 248) * 16777216 + (ord($sequence[1]) - 128) * 262144 + (ord($sequence[2]) - 128) * 4096 + (ord($sequence[3]) - 128) * 64 + (ord($sequence[4]) - 128);
} elseif ($n <= 0xfd) {
$ord = ($n - 252) * 1073741824 + (ord($sequence[1]) - 128) * 16777216 + (ord($sequence[2]) - 128) * 262144 + (ord($sequence[3]) - 128) * 4096 + (ord($sequence[4]) - 128) * 64 + (ord($sequence[5]) - 128);
}
$result .= self::_transliterationReplace($ord, $unknown, $source_langcode);
$head = '';
} elseif ($c < "\x80") {
// ASCII byte.
$result .= $c;
$head = '';
} elseif ($c < "\xc0") {
// Illegal tail bytes.
if ($head == '') {
$result .= $unknown;
}
} else {
// Miscellaneous freaks.
$result .= $unknown;
$head = '';
}
}
}

return $result;
}

/**
* @static
*
* @param int $ord
* @param string $unknown
* @param string|null $langcode
*
* @return string
*/
private static function _transliterationReplace($ord, $unknown = '?', $langcode = null)
{
$map = [];

if (!$langcode) {
$langcode = 'en';
}

$bank = $ord >> 8;

if (!isset($map[$bank][$langcode])) {
$file = __DIR__ . '/Transliteration/Data/' . sprintf('x%02x', $bank) . '.php';
if (file_exists($file)) {
$base = [];
$variant = [];
// contains the $base variable
include($file);
if ($langcode !== 'en' && isset($variant[$langcode])) {
// Merge in language specific mappings.
$map[$bank][$langcode] = $variant[$langcode] + $base;
} else {
$map[$bank][$langcode] = $base;
}
} else {
$map[$bank][$langcode] = [];
}
}

$ord = $ord & 255;

return isset($map[$bank][$langcode][$ord]) ? $map[$bank][$langcode][$ord] : $unknown;
}
}
55 changes: 0 additions & 55 deletions lib/Tool/Transliteration/Data/x00.php
View file
Open in desktop

This file was deleted.

49 changes: 0 additions & 49 deletions lib/Tool/Transliteration/Data/x01.php
View file
Open in desktop

This file was deleted.

Loading