why are national letters (ěščř) converted to html special chars?

[MayheM]

Can someone explain why are common letters as "ě š č ř..." converted to html special chars?

Code:

Š á í ...

GS uses UTF-8 so there is no need to convert or am I wrong?

I have no aditional plugins and 3.0 version and it appears in title, meta tags and page's content, but not in menu items.

Is there a way to change this behavior?

Thanks for any response.

[Connie]

First: welcome here!

Second: your question:

it is answered in the wiki: this is an option of the editor:

http://get-simple.info/wiki/how_to:edito...characters

so, add "entities : false" to the editor-options in gsconfig.php and you are done

[ccagle8]

First: welcome here!

Second: your question:

it is answered in the wiki: this is an option of the editor:

http://get-simple.info/wiki/how_to:edito...characters

so, add "entities : false" to the editor-options in gsconfig.php and you are done

Connie - i know we discussed this earlier, but why can't this option be added within the core? Will it cause any problems with back-wards compatibility?

[Connie]

Connie - i know we discussed this earlier, but why can't this option be added within the core? Will it cause any problems with back-wards compatibility?

Chris,

in my opinion it can be added to the core
old posts with converted / escaped characters will work in all browsers
new posts will keep unconverted characters, what will work in UTF-8 with no problems

users who want special rules for special characters, for example greek characters, can add additional options, like here: http://docs.cksource.com/ckeditor_api/sy...ties_greek

so I think it will be the best to include it in the new version

[MayheM]

Sorry I must bother you again, but adding mentioned option to gsconfig.php didn't helped.

Code:

# WYSIWYG Editor Options

define('GSEDITOROPTIONS', 'entities : false');

I corrected it in xml files manually, but when I edit for example title in GS backend again, it converts string to special chars again. I think it has nothing to do with WYSIWYG, becouse it converts site name too (form field without WYSIWYG)
Any other ideas?

EDIT: The problem is in htmlentities() function used in function safe_slash_html() and in settings.php for $SITENAME, so I removed it. Now it works as I expect What do you think? Can I let it as I've got now without any secure vulnerabilities?

[Connie]

I don't think you will have vulnerable risks by that

[Connie]

Connie - i know we discussed this earlier, but why can't this option be added within the core? Will it cause any problems with back-wards compatibility?

Chris,

more and more I think this HTMLentitiy-treatment should be taken off from the code. In times of UTF-8 it is not necessary anymore and it is generating more problems than profit ;=)

[Ampersand]

this HTMLentitiy-treatment should be taken off from the code.

Keeping my all fingers and toes crossed for this.

Sad background

[ccagle8]

i am so confused with this all... At this point, Im not sure what i should be doing in the core to fix these encoding issues.

[Connie]

Chris,

The problem is in htmlentities() function used in function safe_slash_html() and in settings.php for $SITENAME, so I removed it.

http://de3.php.net/manual/en/function.htmlentities.php

this function is not needed anymore. It was needed in times when pages were not coded in UTF-8 with limited charactersets and codepages, when browsers were configured to show pages in american codepage f.e., and it was necessary to have special characters presented correctly (remember: english has no umlauts for example, german or turkish language has many...)

so, nowadays, when GS is completely in UTF-8, templates are in UTF-8, plugins are coded in UTF-8, this special treatment is not needed anymore, but does some harm at some places

so take it off in the above named functions, I would even recommend to never use it again ;=)

[Ampersand]

what i should be doing in the core to fix these encoding issues.

Someone also mentioned it was html_entity_decode

[ccagle8]

can someone who is having these issues with encoding make the changes in basic.php to the functions above to see if it does what everyone wants it to do?

So the checklist of changes is below. Is this it?

- remove htmlentities() from safe_slash_html()
- and i guess remove html_entity_decode() from the functions that display the content... right? But if I remove this, wouldnt pages then not show correctly, if it's on a site that was just recently upgraded, and the content is still encoded from before?

[Ampersand]

not show correctly, if it's on a site that was just recently upgraded, and the content is still encoded from before?

Would it be possible to include in the upgrade a cunning "run-once-recode-all-pages-automatically-in-correct-style-can't-stop-me-invisible" plugin to fix it?

[ccagle8]

yeah, probably. We already have 'upgrade.php' that runs each time it detects a system that is being upgraded. We could probably add it there.... but it would have to be bulletproof - which may be difficult. Is removing the html_entity_decode() function essential to solving this problem?

[grs84pl]

I've got this problem with entities but my solution only decode what was coded in first place... so it isn't good a solution.

I also keep my fingers crossed for taking off HTMLentity from the code. life would be so easier...

[nomik05]

Hi all!
1) Commenting out some rows as bellow is necessary, safe and does the job!
(file: \admin\inc\basic.php)

Code:

function safe_slash_html($text)
    {
        if (get_magic_quotes_gpc()==0)
        {
            //$text = addslashes(htmlentities($text, ENT_QUOTES, 'UTF-8'));
        }
        else
        {
            //$text = htmlentities($text, ENT_QUOTES, 'UTF-8');
        }
        return $text;
    }

2) Changing some options in gsconfig.php is also necessary!
(weird) this doesn't seems to work for me: define('GSEDITOROPTIONS',"entities : false");
BUT this works for both options:

Code:

define('GSEDITOROPTIONS',"colorButton_enableMore : true, entities : false");

-> I believe html_entity_decode() function IS NOT essential to solving this problem?
HTML is saved and returned properly
html_entity_decode() is JUST USELESS and maybe time consuming
BUT DO NOT REMOVE html_entity_decode() IF YOU DONT DO THE ABOVE 2 STEPS!

[shawn_a]

My understanding.

So at some point we started saving UTF-8 without encoding first, and now we have a mismatch with decoding ? Or rather we would if we removed this.

I assume we not adding versioning strings to datafiles either?

Not sure i understand since htmlentities is supposed to be UTF-8 safe when the argument is present.
Which makes it sound like the issue in in the encoding to xml ?

Anyone have a flow chart for this ? lol

Shouldn't we be using htmlspecialchars instead?

[shawn_a]

https://github.com/GetSimpleCMS/GetSimpleCMS/issues/618