How to escape the "&" in a JSON.stringfy() ajax call?

[Claus Mygind]

I am having trouble escaping the & in a JSON.stringfy() ajax call. I don't even know if I am stating the problem correctly here.

In my app I have linked json2.js from http://www.JSON.org/json2.js 2010-08-25(see attached file) to my app.

Then I have created an ajax call in the app to my server_side app in which I invoke the JSON.stringify( dataObj). My "dataObj" is an Associative Array. If one of the array element values contains the "&" the queryString variable is truncated. I was under the impression that JSON.stringify( ) would escape any control characters. but I am wrong.

What baffles me more is the fact I have even commented out the link to json2.js in my page header and the app does not crash when it get to the JSON.stringify( ) command.

<!-- <script type="text/javascript" language="JavaScript"    src="json2.js"></script>
 -->

I don't know if it is because I am using FireFox 3.6.13. I thought that I had read that they have now included JSON right in FireFox.

The question remains "How do I escape the & or any other control character, when it occurs in a value"? I would hope this would be part of the json process.

Here is the offending name/value pair:
"TITLE":"Earthw ork & Excavating Contracts Manager"

This is the string as transmitted
"TITLE":"Earthw ork

Here is my ajax code.

//a public array used to store the ajax call response
var	aCallBack = new Array();

function myAjax (callApp, dataObj, getOrSave) {

	this.makeCall = function (callApp, dataObj, getOrSave) {
		var queryString  = "&appData="+JSON.stringify(dataObj);

					//create ajax object
		var request = window.ActiveXObject ?
			new ActiveXObject('Microsoft.XMLHTTP') :
		    new XMLHttpRequest;

			//assemble data
			var url = callApp+'?timeStamp=' + new Date().getTime();

			//actually make call and get response in chkStateChange function
			request.open("POST", url, true);
			request.onreadystatechange = function chkStateChange(){
											if (request.readyState == 4) {
												if (request.status == 200) {
													processResponse(request.responseText);
												}
											}
										 };
			request.setRequestHeader("Content-Type", "application/x-www-form-urlencoded");
			request.send( queryString );
	};

	var processResponse = function (response) {

		aCallBack = response.split(';');
	
		if ( aCallBack[0] == "An error occured") { 
			eAlert  = aCallBack[0]+"\n";
			for (var i = 1; i < aCallBack.length; i++ )
			{
				eAlert += aCallBack[i]+"\n";
			}
			alert( eAlert );
		}else if (aCallBack[0] == "Record Saved")	{
			alert("temp message Record Saved. /n look in _Navigation.js");		
		}else{
			updatePage();
		}
	};
}

//ajax object created
var ajax = new myAjax();

[Oralloy]

Claus,

Sorry for being quick, but I didn't take the time to completely analyze what you wrote. No insult intended, I'm searching for an answer and found your post.

Do you need to JSON escape the "&", or HTML escape it? Remember that there are a number of magic characters, that must be escaped in HTTP "GET" queries. Not just the "&" character, but the "+" and " " characters, as well. So what you probably need to do is to HTML encode your arguments. There are standard ways to do this, but I'm not an AJAX Guru, so I'll leave that to smarter folk than I.

Also, JSON is delivered as a standard part of recent browsers. You can test this out by building a trivial page like so:

<html>
 <head>
  <title>JSON Test</title>
 </head>
 <body>
  JSON Test<br>
  JSON.stringify = "<span id='myJSON' />"
 </body>
</html>
<script type="text/javascript" language="JavaScript">
  alert(JSON.stringify);
  document.getElementById('myJSON').innerHTML = String(JSON.stringify);
</script>

Cheers!
Oralloy

[Claus Mygind]

Thanks for the reply. Not sure what I was suppose to get from that. Cut and pasted the code in my FireFox 3.6.13 and got this result. Is this the intended and could you explain the outcome in a little more depth, so I have a better understanding?

This was the output I got:
JSON Test
JSON.stringify = "function stringify() { [native code] }

[Oralloy]

What that tells you is that JSON is already incorporated in your browser, so you don't have to include the json2 script library as part of your web page. Basically JSON is so useful, that it's part of the default scripting environment now days.

Also, as a test, try substituting '&ampl;' for '&' in your server response and see if that helps any. In general you should HTML encode all responses to queries, as '&' is not the only magic character you'll encounter.

Luck!
Oralloy

[Claus Mygind]

Thanks again for the quick explanation.

Just a couple of items here.

1) I am doing a POST not a GET
2) the array I am passing is being assembled on the client side not the server side.

& alone is the offending character. So I don't think I can substitute & with &ampl;

I also see now that & is not one of the offending characters to JSON. So that kind of leads back to your original thought about doing an HTML escape.

As explained here:

http://bytes.com/topic/javascript/answers/478176-escaping-special-characters-json

I am pretty confident that my JSON.stingify() works because I tested it with the " double quote.

So I wonder if I will have to run the standard javaScript encode and decode when I create the array?

Also I would like to know if the code snippet I posted is correctly written? Meaning when it gets called, it indeed does stringify the array?

[Oralloy]

Claus,

Your basic problem is HTTP parameter parsing.

What happens is that the server recieves the parameter string queryString, which looks like "&appData="+JSO N.stringify(dat aObj).

Then the server parses the string based on HTTP's rules. To whit, '&' separates separates parameters, '=' separates parameter names from values, etc... Basically the rules are exactly the same as for a "GET" query; the only difference being that the parameters are sent in the body of the query, and not the URL part.

Nothing in HTTP's parameter specifications gives special meaning to quote (') characters in the parameter string. So, even though your '&' character is embedded in a well-formed JSON string, and is fully quoted, HTTP doesn't care. Similarly you will find that addition symbols, '+', are mapped to spaces.

The way to deal with this is to URL encode the parameter string by mapping special characters ('&', '+', ' ', '=', etc...) which occur in the parameter names and values into their appropriate transmission encodings.

Here are a couple links that might help:
Query_string#UR L_encoding
Percent-encoding

Hopefully this helps some.

What surprises me is that the request object doesn't have a method which accepts "key/value" pair and encodes them automagically. That would save a lot of grief in the long run.

Oh well.

Good Luck! Let me know how it turns out.
Oralloy

[Claus Mygind]

Thank you for a very thorough reply. I get it now and I will just encode the data when I form the array.

Likewise I will decode the data on the server side.

I just did not know that when you do ajax POST that it would be different than a standard POST when you submit a form.

[Claus Mygind]

Well it is nice to know someone has already invented the wheel.

http://www.openjs.com/scripts/data/ued_url_encoded_data/

Looks like it should be a simple copy and paste operation for me.

Thanks again Oralloy for your outstanding help.

[Oralloy]

Claus,

I'm glad that you found a complete solution. It looks like the key to his code is the "encodeURICompo nent" call at the bottom of his script. I didn't know about that method, so I'm glad you found it.

Cheers!

[Claus Mygind]

It turns out a better option is to simply encode the & when sending the JSON string to the server using POST.

The array I created is encoded by the:
var queryString = "appData="+JSON .stringify(data Obj);
(see code below)

the stringify function of JSON will encode all the special characters execpt the &. if the actual data contains an & the ajax call fails because the & will split the data at that point thinking it is a 2nd name/value pair.

What I can't understand is by comparison I am a novice at this why can I not find any reference on the net about how to handle the & in the data transmitted. The link I posted earlier uses a different format than JSON when transmitting data from the client to the server. So you are not taking advantage of JSON when the data arrives at the server. If there is more links that explain how to handle the &, I sure would like to read about it.

I made the following change to my code and got it to work

I build my array in the following manner:

/*
-----------------------------------------
I call this function to read all the data 
nodes/elements on the form I want to save
------------------------------------------
*/
function saveData(form)
{
	var dataObj = new Object();

	for (var i = 0; i < form.elements.length; i++)
	{
		var element = form.elements[i];
		var type = element.type;
		if (type == "checkbox")
		{
			if (element.checked)
			{
				dataObj[element.id] ="T";
			}else{
				dataObj[element.id] ="F";
			}
		}
		else if (type == "radio")
		{
			if (element.checked)
			{
				dataObj[element.id] = element.value;
			}
		}
		else if (type == "hidden" || type == "password" || type == "text" ||
			     type == "textarea")
		{
//text nodes which could contain the & are checked and encoded if necessary
				dataObj[element.id] = encode(element.value);
		}
		else if (type == "select-one" || type == "select-multiple")
		{

			xText += form.element.selectedIndex.value;
		        for (var j = 0; j < element.options.length; j++)
			    {
				    if (element.options[j].selected )
	                {
//text nodes which could contain the & are checked and encoded if necessary
dataObj[element.id] = encode(element.options[j].value);
					}
				}
		}
	}

/*
-------------------------------
Here I make an ajax call to the 
server via the function/method 
I previously bound to the 
variable ajax, see code below.
-------------------------------
*/
   ajax.makeCall("myServerSideApp.php", dataObj);

}

function encode(strVal)
{
/*
-------------------------------------------
using a regular expression I search 
for the & and encode it with %26.  

This will automatically be decoded 
in my php server side app when I call "json_decode($_REQUEST['appData'], true);",
where appData is the JSON object 
I transmitted.
--------------------------------------------
*/
	if (strVal.indexOf('&') > -1)
	{
		var searchStr = "&";
		var replaceStr = "%26";
		var re = new RegExp(searchStr, "g");
		var result = strVal.replace(re, replaceStr);
	}else{
		var result = strVal;
	}
			
	return result;
}


/*
----------------------------------------------
Here is the actual ajax function called above
-----------------------------------------------
*/

//a public array used to store the ajax call response
var	aCallBack = new Array();

function myAjax (callApp, dataObj) {

	this.makeCall = function (callApp, dataObj) {
		var queryString  = "appData="+JSON.stringify(dataObj);

		//create ajax object
		var request = window.ActiveXObject ?
			new ActiveXObject('Microsoft.XMLHTTP') :
		    new XMLHttpRequest;

			//create url
			var url = callApp+'?timeStamp=' + new Date().getTime();

			//actually make call and get response in chkStateChange function
			request.open("POST", url, true);
			request.onreadystatechange = function chkStateChange(){
											if (request.readyState == 4) {
												if (request.status == 200) {
													processResponse(request.responseText);
												}
											}
										 };
			request.setRequestHeader("Content-Type", "application/x-www-form-urlencoded");
			request.send( queryString );
	};

	var processResponse = function (response) {

		aCallBack = response.split(';');
	
		if ( aCallBack[0] == "An error occured") { 
			eAlert  = aCallBack[0]+"\n";
			for (var i = 1; i < aCallBack.length; i++ )
			{
				eAlert += aCallBack[i]+"\n";
			}
			alert( eAlert );
		}else if (aCallBack[0] == "Record Saved")	{
			alert("message Record Saved.");		
		}else{
			updatePage();
		}
	};
}

/*
-----------------------------
ajax object created and bound 
to my myAjax function/method
-----------------------------
*/
var ajax = new myAjax();