Purpose of not validating & (ampersand) in xthml strict?

[vineld]

What is the actual purpose of not letting ampersands (&) pass through in xhtml strict? Since I usually use this format in my work it makes my life so much more painful. I don't use & very often though since it does not regularly appear in well written text (with certain exceptions that is) and my urls are mostly user friendly except for admin pages and then I don't really care if it validates or not.

 

However, when printing database values I always need to replace & with & which sucks big time as it's a completely unnecessary operation and trying to build as efficient applications as possible that is not on my wishlist. I really don't want to save & as & in the database.

 

Is there an easier way for me to get rid of this annoying issue?

[Daniel0]

Run it through htmlentities before outputting.

[vineld]

Do I really need to run htmlentities()? It replaces a lot of other characters as well that do validate nonetheless.

[Daniel0]

Yes, and you should also do it to protect against things like XSS (see this tutorial).

[vineld]

How could displaying database values open up for XSS vulnerability?

[Daniel0]

Because whoever entered it could just as well enter Javascript code.

[vineld]

Agreed but wouldn't it be much better to handle that prior to inserting the data into the database instead?

[Daniel0]

No, you should store it raw in the database so you can filter it for a variety of contexts after retrieval from the database.

[vineld]

Are you seriously saying that I should always store raw data and validate it each time it is displayed?

[Daniel0]

No. Validation isn't the same as filtering.

 

Validation could be checking that an entered date is valid. Filtering could be making sure it fits for usage in SQL, HTML, JSON, XML, etc.

[vineld]

Twisting words  For me making sure that I get what I want is validation, no matter what I look for.

 

Perhaps this is just a mistunderstanding. Could you give me some example of those scenarios where it will be necessary to filter database values where it can not easily be done prior to insertion?

[Daniel0]

I don't understand your question, but say that $post is an array holding information about a forum post, like the one I'm writing now.

 

That is just raw information, and it can be represented or output in a variety of manners. In HTML I might display the post like this:

<?php echo nl2br(htmlentities($post['body'])) ?>

 

Another scenario might be that I was writing some AJAX and wanted to get the post via a backend API, so I might do something like this instead:

echo json_encode($post);

 

So depending on how I want to use it, the way I'll filter and manipulate the data is different.

[vineld]

Alright, when put that way I will have to agree with you  There are situations where the border between filtering and validation is very thin though. Most data stored in databases are plain text or numbers though and then I will not want any unnecessary operations on display. Of course a forum post or WYSIWYG content is a bit more complex to handle.

 

I think we are on the same page after all  In my defense, I haven't slept much lately 

[Daniel0]

Well, there are certain data types that don't necessarily need to be filtered on HTML output. The numeric and date fields for instance.