I have a Windows service that publishes data to a .NET client and a web client over SignalR. I've recently had some funny issues, but can't quite get a consistent behavior.
The problem lies in serializing the degrees sign, i.e. "°C". Most of the time it is serialized correctly, but I've had a few times where I see the following in my debugger:
See how the first time the "°" is serialized correctly, but the second time we see the question marks in the diamonds?
I've read this means it is an invalid UTF-8 character. But then why do all the other properties serialize correctly? This is a screenshot where you see one correct and one incorrect, but the entire JSON contains hundreds of these "°C" strings that look correct.
So why this one exception? It's not always the same position/property, and it doesn't always happen. This makes me think it must be a combination with preceding/succeeding characters, no?
Any ideas how to fix this or at least how to investigate this further?
Update
This is how I do serialization. I set it up on startup:
var serializerSettings = new JsonSerializerSettings();
serializerSettings.Converters.Add(new StructuredAmountJsonConverter());
var serializer = JsonSerializer.Create(serializerSettings);
GlobalHost.DependencyResolver = new AutofacDependencyResolver(_lifetimeScope);
GlobalHost.DependencyResolver.Register(typeof(JsonSerializer), () => serializer);
What's happening here is I'm telling SignalR to use Autofac to resolve dependencies. Then I register my JSON.NET serializer. The JSON.NET serializer has one custom converter, which converts my Amount class to the structure you see above (with a value and a unit property).
So you could think the problem lies in the converter, but then why is it working 95% of the time? Or should I specify the encoding in my converter?
Update 2
I've been using Fiddler to capture my network traffic and I can't see the wrong characters there. So I'm guessing the encoding problem is at the client side. I will investigate further.
Update 3
I've managed to capture the traffic in Fiddler and while it looks good in the Text view, when I select the HexView I do see something weird:
Notice how it says "°C" instead of "°C". So maybe it is sending it from the server in the wrong way?
Also, keep in mind my client is a .NET (WPF) client. This is my code to connect on the client side (simplified):
var url = "myUrl...";
_hubConnection = new HubConnection(url);
var hubProxy = _hubConnection.CreateHubProxy("MyHub");
hubProxy.On<object>("receive", OnDataReceived);
await _hubConnection.Start();
And when receiving data:
var message = JsonConvert.DeserializeObject<MyDataContract>(obj.ToString(), new StructuredAmountJsonConverter());
Update 4
This post makes me think this is happening:
The server/SignalR is sending my data as UTF-8, but the client is expecting latin-1 or Windows-1252 (probably the latter). So now I need to find out how I can make it use UTF-8.
May I know how you are serializing the objects? I think there is a way to specify the encoding types for serializing special characters like this. Here is a link I found- Special characters in object making JSON invalid (DataContractJsonSerializer)
Related
This question already has answers here:
What is JavaScript's highest integer value that a number can go to without losing precision?
(21 answers)
Closed 1 year ago.
I'm building an application with a react.js front-end (although I'm fairly sure that's not relevant to the issue) and an asp.net back-end (targeting net5.0, not sure if that's relevant). When I call to the back-end API, I get back an object that includes, in part, an ID it generated based on the data passed in, that is of type long in C# (a 64 bit int). The behavior that I'm seeing is that the variable in C# and the variable read from the response on the front end are different. They appear to drift after about 16-17 digits.
Is this expected? Is there any way to get around it?
Code to reproduce / Pictures of what I see:
C#
[HttpPost]
[Route("test")]
public object TestPassingLongInObject()
{
/* actual logic omitted for brevity */
var rv = new
{
DataReadIn = new { /* omitted */ },
ValidationResult = new ValidationResult(),
GeneratedID = long.Parse($"922337203{new Random().Next(int.MaxValue):0000000000}") // close to long.MaxValue
};
Console.WriteLine($"ID in C# controller method: {rv.GeneratedID}");
return rv;
}
Console output: ID in C# controller method: 9223372030653055062
Chrome Dev Tools:
When I try to access the ID on the front-end I get the incorrect ID ending in 000 instead of 062.
Edit 1: It's been suggested that this is because JavaScript's Number.MAX_SAFE_INTEGER is less than the value I'm passing. I don't believe that is the reason, but perhaps I'm wrong and someone can enlighten me. On the JS side, I'm using BigInt, precisely because the number I'm passing is too large for Number. The issue is before I'm parsing the result to a JS object though (unless Chrome is doing that automatically, leading to the picture referenced in the issue).
Edit 2: Based on the answers below, it looks like perhaps JS is parsing the value to a number before I'm parsing it to a BigInt, so I'm still losing precision. Can someone more familiar with the web confirm this?
JavaScript is interpreting that value as a number which is a 64-bit floating-point number. If you want to keep that value, you're better off passing it back as a string.
Example JS to demonstrate the problem:
var value = 9223372030653055062;
console.log(value);
console.log(typeof(value));
JavaScript engines will (apparently) fail to create numbers this big. Try to open your console and enter the following:
var myLong = 9223372030653055062;
console.log(myLong);
You can check 9223372030653055 > Number.MAX_SAFE_INTEGER as well, as a pointer that you're going out of bounds.
The maximum size of a Number in Javascript is 2^53 - 1 which is 9007199254740991. In C# a long is 2^64 - 1, which is bigger. So "9223372030653055062" works as a C# long but not as a JavaScript Number because it is too big. If this ID is being used in a database using a long, I'd suggest you just pass it as a string to JavaScript.
Although the reason is that the number is bigger than JS can accurately represent, you seem to have been distracted be Chrome's Dev Tools Preview. This is only a "helpful" preview and not necessarily the truth.
The dev tools preview shows a helpful view of the response. I presume the AJAX response is transferred with content type of application/json, so Chrome helpfully parses the JSON to a JS object and lets you preview the response. Check the Response tab, this is what will actually be received by your AJAX code.
The chances are that the JSON is being parsed before you have chance to use BigInt on that number, capping the precision. You haven't shown us your AJAX code so this is my best guess.
The only solution would be to serialize it as a string then add a manual step to convert the string to BigInt.
jsObject.generatedId = BigInt(jsObject.generatedId);
If you're using it as an ID then you might as well keep it as a string on the client-side as you're not likely to be using it to do any calculations.
I struggle with safely encoding html-like text in json. The text should be written into a <textarea>, transferred by ajax to the server (.net45 mvc) and stored in a database in a json-string.
When transferring to server, I get the famous "A potentially dangerous Request.Form value was detected" 500 server error. To avoid this message, I use the [AllowHtml] attribute on the model that are transferred. By doing so I open up for XSS-vulnerability, in case anyone paste in { "key1": "<script>alert(\"danger!\")</script>" }. As such, I would like to use something like
tableData.Json = AntiXssEncoder.HtmlEncode(json, true);
Problem is I cannot do this on the full json string, as it will render something like
{
"key1": ...}
which of course is not what I want. It should be more like
{ "key1": "<script>alert("danger!")</script>" }
With this result the user can write whatever code they want, but I can avoid it to be rendered as html, and just display it as ordinary text. Does anyone know how to traverse json with C# (Newtonsoft Json.NET) such that strings can be encoded with AntiXssEncoder.HtmlEncode(... , ....);? Or am I on a wrong track here?
Edit:
The data is non-uniform, so deserialization into uniform objects is not an option.
The data will probably be opened to the public, so storing the data encoded would ease my soul.
If you already have the data as a JSON string, you could parse it into proper objects with something like Json.NET using JsonConvert.DeserializeObject() (or anything else, there are actually quite a few options to choose from). Once it's plain objects, you can go through them and apply any encoding you want, then serialize them again into a JSON string. You can also have a look at this question and its answers.
Another approach that you may take is just leave it alone until actually inserting stuff into the page DOM. You can store unencoded data in the database, you can even send it to the client without HTML encoding as JSON data (of course it needs to be encoded for JSON, but any serializer does that). You need to be careful not to generate it this way directly into the page source though, but as long as it's an AJAX response with text/json content type, it's fine. Then on the client, when you decide to insert it into the actual textarea, you need to make sure you insert it as text, and not html. Technically this could mean using jQuery's .text() instead of .html(), or your template engine's or client-side data binding solution's relevant method (text: instead of html: in Knockout, #: instead of #= in say Kendo UI, etc.)
The advantage of this is latter approach is that when sending the data, the server (something like an API) does not need to know or care about where or how a client will use the data, it's just data. The client may need different encoding for an HTML or a Javascript context, the server cannot necessarily choose the right one.
If you know it's just that text area though where this data is needed, you can of course take the first (your original) approach, encode it on the server, that's equally good (some may argue that's even better in that scenario).
The problem with answering this question is that details count a lot. In theory, there are a myriad of ways you could do it right, but sometimes a good solution differs from a vulnerable one in one single character.
So this is the solution I went for. I added the [AllowHtml] attribute in the ViewModel, so that I could send raw html from the textarea (through ajax).
With this attribute I avoid the System.Web.HttpRequestValidationException that MVC gives to protect against XSS dangers.
Then I traverse the json-string by parsing it as a JToken and encode the strings:
public class JsonUtils
{
public static string HtmlEncodeJTokenStrings(string jsonString)
{
var reconstruct = JToken.Parse(jsonString);
var stack = new Stack<JToken>();
stack.Push(reconstruct);
while (stack.Count > 0)
{
var item = stack.Pop();
if (item.Type == JTokenType.String)
{
var valueItem = item as JValue;
if(valueItem == null)
continue;
var value = valueItem.Value<string>();
valueItem.Value = AntiXssEncoder.HtmlEncode(value, true);
}
foreach (var child in item.Children())
{
stack.Push(child);
}
}
return reconstruct.ToString();
}
}
The resulting json-string will still be valid and I store it in DB. Now, when printing it in a View, I can use the strings directly from json in JS.
When opening it again in another <textarea> for editing, I have to decode the html entities. For that I "stole" some js-code (decodeHtmlEntities) from string.js; of course adding the licence and credit note.
Hope this helps anyone.
I am working on iOS project and sending the Json string to backend through wcf webservice, Its working successfully for many users but for some users backend getting incomplete json string.
Code for generating Json string
NSData *data = [NSJSONSerialization dataWithJSONObject:EmployeeDetails options:0 error:nil];
NSString *JsonString = [[NSString alloc] initWithData:data
encoding:NSUTF8StringEncoding];
And Attaching the json string with request by below code
[request setValue:#"application/x-www-form-urlencoded" forHTTPHeaderField:#"Content-Type"];
[request setHTTPBody:[jsonString dataUsingEncoding:NSUTF8StringEncoding]];
EmployeeDetails is NSMutableDictionary with employee details.
And
Here is successful Json string deserialized from BackEnd
{"FirstName":"Donald","LastCBPaymentType":6,"AcceptEmail":true,"CellPhone":"321-300-6874","Position":"3","CarrierNum":"4","EmpNum":"96874"}
And Incomplete Json string
{"FirstName":"roset","LastCBPaymentType":6,"AcceptEmail":true,"CellPhone":"321-300-6874","Position":"3","CarrierNum":"4","EmpNum":"98799
The incomplete json string occuring in backend for some user.
Any one can help to solve this issue ?
Well, I'd say TomSwift is on the right track... It's apparent that the broken JSON is simply missing "} from the end of the string. That does indeed look like something being sanitised by a regex...
You really should find the source of the problem; But for a quick and dirty fix while you figure it out, you could just run your own regex on the JSON you receive and if it doesn't end with a "}, just add it... First do something like: '/"[0-9]+$/' (to check if there is an open numeric string at the end, if there is, you close it by adding " ) then check if '/}$/' (if } is the last char, if it isn't: add it).
Remember, the above would only be a temporary fix (as it is somewhat heavy and not really that pretty), but I could imagine myself being forced to do something like this, simply because of time constraints. If the issue is actually coming from within the API, you could tell the creators about it and use something like that fix above while they fixed it.
PS. If anyone does choose to use a fix like the above, please please please, for the love of all that you hold holy, leave a comment in the code explaining why you did it (as opposed to telling us what you did...).
I had a similar problem in PHP/CodeIgniter recently and the problem was that the JSON was being "sanitized" via some overly ambitious regular expressions before it arrived to my handler. Perhaps there's some similar sanitization that happens in WCF?
Things I would try:
Using Charles Proxy to confirm that the request (POST?) is actually sending the full body as you expect.
Try Content-Type application/json instead.
Say I have a sample Json format string as
string per1 = #"[{""Email"":""AAA"",""mj_campaign_id"":""22"",""mj_contact_id"":""PPP"",""customcampaign"":""AAA"",""blocked"":""22"",""hard_bounce"":""PPP"",""blocked"":""22"",""hard_bounce"":""PPP""},"
+ #"{""Email"":""BBB"",""mj_campaign_id"":""25"",""mj_contact_id"":""QQQ"",""customcampaign"":""AAA"",""blocked"":""22"",""hard_bounce"":""PPP"",""blocked"":""22""},"
+ #"{""Email"":""CCC"",""mj_campaign_id"":""38"",""mj_contact_id"":""RRR"",""customcampaign"":""AAA"",""blocked"":""22"",""hard_bounce"":""PPP""}]";
I am trying to deserialize it using
var result = JsonConvert.DeserializeObject(per1);
Its working fine as long as all the rows of the string has values for the following attributes Email, mj_campaign_id, mj_contact_id, customcampaign, blocked, hard_bounce, error_related_to, error. But when I skip some sttribute values in some rows its throwing an error saying
Can not add Newtonsoft.Json.Linq.JValue to Newtonsoft.Json.Linq.JObject.
Any help would be appreciated. Thanks
Your error is because you are not assigning a value to an object, which you need to do. If you remove the value, at least add an empty string.
THAT SAID!
Herein lies the danger of manually building JSON strings. You should always avoid it if you can. If you are reading from a web page, that web page should serialize the payload for you, and then you should deserialize it with whatever you are using to pull in the payload (controller, restful service, etc). The beauty of .NET is that it handles all of this plumbing for you and you really are going to run into painful issues if you try to reinvent the .NET wheel
I have a java backend that sends messages using protobuf, it sends delimited message objects in one big byte array blob over tib. I can deserialize them fine using the function parseDelimitedFrom(yourStreamHere) in java but on the C# side we are having some issues and I couldn't find any examples but I may just be missing something obvious here.
We are doing something in C# like this
using (MemoryStream mem = new MemoryStream())
{
mem.Write(byteArray, 0, byteArray.Length);
mem.Position = 0;
return Serializer.Deserialize<List<OrderState>>(mem);
}
Note: I saw an older post on this but it looked pretty dated and I think changes have occurred to protobuf-net since then but correct if I'm wrong there
The developer was using tag 0 and prefix style 128 at one point yesterday like so
IEnumerable<SomeObject> list = (Serializer.DeserializeItems<SomeObject>(memoryStream, PrefixStyle.Base128, 0));
but we were still getting an error. When we called getProto on the C# side today it appears it was converting our properties that were set to the double type to the fixed64 type, on the java side we had specified double so I think this mismatch was causing the errors we were seeing. We temporarily changed those fields to the string type and now the above snippet works. Of course ideally we don't want to send strings when we don't have to.