Re^3: Escaping double quotes in complete document

Hi again haukex and huck

I would agree in the suspicion that something else than Perl has changed, but I already dived fairly deep into this.

Java (that runs the JBoss server from which I get my input) hasn't been updated, because it is not a package available in my package manager, neither are the JBoss itself or the database host.
The Apache has been updated, but as I understand it, it would have interpreted double quotes wrongly regardless of its version, right?
That leaves Perl and my changes in the scripts as possible culprits. As I said, the update messed up the page encoding, because none of the scripts were explicitly using UTF-8 and most of them started the HTML page encoded as ISO-8859-1.

I can confirm with 100% certainty that saving and loading UTF-8 characters and double quotes worked before the update.
However, I do not know, if it didn't work at all after the update or if it started to malfunction this way after I enabled the scripts to use UTF-8 (though I think these errors are not necessarily connected).

I appreciate your answers a lot and I will definitely propose updating our infrastructure according to your advice.

Comment on Re^3: Escaping double quotes in complete document

Replies are listed 'Best First'.
Re^4: Escaping double quotes in complete document by haukex (Archbishop) on Jun 27, 2017 at 13:42 UTC
I can confirm with 100% certainty that saving and loading UTF-8 characters and double quotes worked before the update. I am unclear on this for several reasons. First, does this mean that you have both the new and the old software environment available in parallel? If so, one obvious approach to debugging it would be to pepper the script with plenty of debugging output until the differences and the origin of those differences becomes clear. Second, can you be more specific on the double quotes issue, in what way did it previously "work" compared to now? Again, showing a short but representative code snippet (SSCCE) along with sample input, the expected output and the actual output including exact error messages is most helpful here - if we can reproduce the issue locally we will be able to help much better. While I can't say much about how Java, JBoss, or Apache plays a role in all this, I can say with a high degree of certainty that there have been no changes in Perl itself between v5.10.1 and v5.16.3 that would explain different handling of double quotes in strings interpolated into the HTML (while there has been a lot of work done on Unicode handling over the years). I would agree that based on your descriptions so far it is unlikely that the issue of Unicode/UTF-8 handling is connected to the issue of double quotes. Code like `print qq{<input type="text" name="mytext" value="$SOAPResult"/>};` would always have been a problem if `$SOAPResult` contained double quotes. Since you say the HTML is output with print and not a module, and it sounds like there was previously no module being used to do escaping (like HTML::Entities or CGI.pm's `escapeHTML`), the only other possible culprit might be whatever Perl modules you are using to fetch and store the data (SOAP::Lite, or perhaps DBI), or the source of the data itself. For example, as huck said, perhaps there was previously a validation of data going into the database that kept double quotes out of the database entries, or there was some layer somewhere doing escaping of the values coming out of the database. Or it's something simpler like there never was any validation, and over time you've simply accumulated more and more records in the DB that bring to light this issue which was always present. Without more information (= code to reproduce), this is a bit of a stab in the dark, but my best guess at the moment is that you could start with looking at what SOAP::Lite is doing, i.e. what strings it was giving your code before and after the upgrade.	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re^4: Escaping double quotes in complete document
by haukex (Archbishop) on Jun 27, 2017 at 13:42 UTC

I can confirm with 100% certainty that saving and loading UTF-8 characters and double quotes worked before the update.

I am unclear on this for several reasons. First, does this mean that you have both the new and the old software environment available in parallel? If so, one obvious approach to debugging it would be to pepper the script with plenty of debugging output until the differences and the origin of those differences becomes clear. Second, can you be more specific on the double quotes issue, in what way did it previously "work" compared to now? Again, showing a short but representative code snippet (SSCCE) along with sample input, the expected output and the actual output including exact error messages is most helpful here - if we can reproduce the issue locally we will be able to help much better.

While I can't say much about how Java, JBoss, or Apache plays a role in all this, I can say with a high degree of certainty that there have been no changes in Perl itself between v5.10.1 and v5.16.3 that would explain different handling of double quotes in strings interpolated into the HTML (while there has been a lot of work done on Unicode handling over the years). I would agree that based on your descriptions so far it is unlikely that the issue of Unicode/UTF-8 handling is connected to the issue of double quotes.

Code like print qq{<input type="text" name="mytext" value="$SOAPResult"/>}; would always have been a problem if $SOAPResult contained double quotes.

Since you say the HTML is output with print and not a module, and it sounds like there was previously no module being used to do escaping (like HTML::Entities or CGI.pm's escapeHTML), the only other possible culprit might be whatever Perl modules you are using to fetch and store the data (SOAP::Lite, or perhaps DBI), or the source of the data itself. For example, as huck said, perhaps there was previously a validation of data going into the database that kept double quotes out of the database entries, or there was some layer somewhere doing escaping of the values coming out of the database. Or it's something simpler like there never was any validation, and over time you've simply accumulated more and more records in the DB that bring to light this issue which was always present.

Without more information (= code to reproduce), this is a bit of a stab in the dark, but my best guess at the moment is that you could start with looking at what SOAP::Lite is doing, i.e. what strings it was giving your code before and after the upgrade.

[reply]
[d/l]
[select]