Summernote strange characters when copying / pasting

BWCWeb · December 13, 2021, 4:36pm

idk…its def something wierd. I just removed the Summernote and installed the Medium Editor.

then tried to update the database with the same text…and the same characters appear.
On the medium editor, i had the html show below and it displays fine but when the update occurs, it adds those funky characters.

here is the text i’m copying and pasting

Frank said, “The best Christmas I ever had was when I didn’t even get a present.”

The others were surprised. They had to hear the story.

Frank began to talk… “I grew up in New York. It was the great depression and we were poor. My Mother had died when I was just eight years old. My Dad had a job but he only worked two or three days a week and that was considered good. We lived in a walk up and we just barely had enough food and clothes. I was a kid and didn’t really notice.”

here is the screenshot of the html from that.

Then after the update to the database…

bpj · December 13, 2021, 4:36pm

Open Chrome and go to the page. Right click on the page > inspect
In the developer tools, you should see Network as a tab:

When you click submit on the form, you'll see the request being sent from the browser.

Click on 'payload' to see the data being sent from the form

BWCWeb · December 13, 2021, 4:40pm

no, its not in the payload section

bpj · December 13, 2021, 4:42pm

OK so that means it is the DB character set. The correct characters are being sent from the form but your DB is saving them with a different character set.

BWCWeb · December 13, 2021, 4:43pm

any suggestions on what the character set should be to stop this? I have tried all the suggestions above and none of them work.

bpj · December 13, 2021, 4:53pm

Run this in MySQL:
SHOW VARIABLES LIKE 'character_set%'

BWCWeb · December 13, 2021, 4:53pm

bpj · December 13, 2021, 4:56pm

OK - all UTF8

try
SHOW CREATE TABLE monday_morning_coffee_master

BWCWeb · December 13, 2021, 4:57pm

BWCWeb · December 13, 2021, 4:57pm

also, just for reference…when i ran this script it removed all the odd characters.

UPDATE monday_morning_coffee_master SET newsletter = CONVERT(CAST(CONVERT(newsletter USING latin1) AS BINARY) USING utf8)

bpj · December 13, 2021, 5:03pm

It’s the truncated value that is useful to see - you’re looking to see if the table/columns have any collation/character set values.

Something is either sending the value to the DB from Server Connect in Latin 1 or your table/field is set to it.

You could just check in Wappler:
Workflows > Globals > Database Connections > Connection Options that the character set for the connection is utf8

patrick · December 13, 2021, 5:12pm

If your page is correctly submitting the data in utf8 to the server and your database is using utf8_general_ci then the problem must be in IIS/ASP.

Edit dmxConnectLib/dmxConnect.asp

Add directly after <script runat="server" language="jscript">:

Session.CodePage=65001

CodePage 65001 is for utf8

BWCWeb · December 13, 2021, 6:09pm

i tried to add this and then inserted the same text into the database…still came out the same.

so frustrating.

BWCWeb · December 13, 2021, 6:09pm

Yes, i confirmed that utf8 is setup in my project settings for the database.

BWCWeb · December 13, 2021, 9:51pm

okay, if i copy and paste the text into the summernote text area, then click the Reveal Code option…

i then copy and paste that directly into the phpmyadmin mysql database table, it does not have any formatting issues.

But when i click the save button and it updates through the Wappler system, it creates these odd characters. Its gotta be something on the update that is causing this.

Teodor · December 13, 2021, 9:53pm

Well of course, this way the data is not sent through an ASP page on an IIS server...

Did you test Patrick's last suggestion?

BWCWeb · December 13, 2021, 9:55pm

yes, i added that to the script, but its still the same.

patrick · December 14, 2021, 1:54pm

It is clearly a codepage/encoding problem. You have somewhere configured the wrong encoding.

Check in devtools the content-type header of the page request, it should be text/html; charset=utf-8. Does your page include <meta charset="UTF-8">.

Add <%@ CodePage = 65001 %> to the top of your ASP, this sets the codepage for asp pages. Other ways of setting the codepage in asp is using <% Session.CodePage = 65001 %> and <% Response.CodePage = 65001 %> . The codepage 65001 is for UTF-8. Default the codepage depends on your IIS settings, in most cases the default is windows-1252.

The ASP codepage is important since the ASP code is parsing your form submit and then does the database insert query.

The last step is the database, this also be set to UTF-8. You do this by setting the collation to utf8_general_ci. Which you’ve already done, seeing the screenshots above.

BWCWeb · December 14, 2021, 3:35pm

okay, i think we are getting closer…but still can’t figure out where its pulling the coding from.

So here is what i did. I pulled out the Server Action on the form submission and just had it send it to a -push- page where i can view the details of the form submission. here is the screenshots from that