Develop Reference

Hashtable serialization without BinaryFormatter?

I used BinaryFormatter which is now obsolete, to (de)serialize Hashtable objects.
Hashtables are tricky as they can wrap anything, including other Hashtables:
[ "fruits": [ "apple":10, "peach":4, "raspberry":5 ], "vegetables": [ "lettuce":5 ]
"animals": [ "with4legs": [ "cat":15, "dog":2 ], "withwings": [ "bird":51 ]
]
BinaryFormatter could perfectly serialize these Hashtables (under dotNET 4.8):
Hashtable main = new Hashtable();
Hashtable fruits = new Hashtable()
{
{ "apple", 10 },
{ "peach", 4 },
{ "raspberry", 5 }
};
Hashtable vegetables = new Hashtable()
{
{ "lettuce", 5 }
};
Hashtable with4legs = new Hashtable()
{
{ "cat", 15 },
{ "dog", 2 }
};
Hashtable withwings = new Hashtable()
{
{ "bird", 51 }
};
Hashtable animals = new Hashtable()
{
{ "with4legs", with4legs },
{ "withwings", withwings }
};
main.Add("fruits", fruits);
main.Add("vegetable", vegetables);
main.Add("animals", animals);
BinaryFormatter binaryFormatter = new BinaryFormatter();
using (Stream stream = new FileStream("Test.file", FileMode.Create))
{
binaryFormatter.Serialize(stream, main);
}
The file is somewhat messy:
However, when reading it back with the BinaryFormatter:
Hashtable newMain = new Hashtable();
using (Stream stream = new FileStream("Test.file", FileMode.OpenOrCreate))
{
newMain = (Hashtable)binaryFormatter.Deserialize(stream);
}
It could perfectly reassemble the Hashtable:
Now, with dotNET, BinaryFormatter is obsolete, and XmlSerializer or JsonSerializer is recommended to be used instead:
using (Stream stream = new FileStream("Test.file", FileMode.Create))
{
JsonSerializer.Serialize(stream, main);
}
File is in JSON format now:
And unfortunately when deserializing it:
Hashtable newMain = new Hashtable();
using (Stream stream = new FileStream("Test.file", FileMode.OpenOrCreate))
{
newMain = (Hashtable)JsonSerializer.Deserialize<Hashtable>(stream);
}
Hashtable loses its structure:
I did also try with MessagePack: https://msgpack.org, but I can't make it to go below one level either:
Now I know there can be more efficient or robust solution for this than Hashtable, but still, is there a way to move from BinaryFormatter to any recommendation which can handle saving and reloading this structure?

JToken.WriteToAsync dosen't write to stream correctly

I want to write a JToken to a stream asynchronously. And I referred to JToken.WriteToAsync does not write to JsonWriter.
However, the stream output is ?[], while ToString() output is []. Why does the stream contain extra bytes at the beginning?
My code is below:
static async Task Main(string[] args)
{
JArray arr = new JArray();
//var c = JToken.FromObject("abc");
//arr.Add(c);
var stream = new MemoryStream();
await using (var requestWriter = new StreamWriter(stream, System.Text.Encoding.UTF8, leaveOpen: true))
{
var jsonWriter = new JsonTextWriter(requestWriter);
try
{
await arr.WriteToAsync(jsonWriter);
}
finally
{
await jsonWriter.CloseAsync();
}
Console.WriteLine(System.Text.Encoding.UTF8.GetString(stream.GetBuffer(), 0, checked((int)stream.Length)));
Console.WriteLine(arr.ToString());
}
}
Why stream output is not correct?
The Json.net's version is 13.0.1.

Summary
Your problem has nothing to do with asynchronous writing. Your problem is that Encoding.UTF8:
returns a UTF8Encoding object that provides a Unicode byte order mark (BOM).
The extra ? you are seeing is that BOM. To prevent the BOM from being written, use new UTF8Encoding(false) when writing. Or, you could just do new StreamWriter(stream, leaveOpen: true) as the StreamWriter constructors will use a UTF-8 encoding without a Byte-Order Mark (BOM) by default.
Details
Your problem can be reproduced more simply as follows:
JArray arr = new JArray();
var stream = new MemoryStream();
using (var requestWriter = new StreamWriter(stream, System.Text.Encoding.UTF8, leaveOpen: true))
using (var jsonWriter = new JsonTextWriter(requestWriter))
{
arr.WriteTo(jsonWriter);
}
var resultJson = Encoding.UTF8.GetString(stream.GetBuffer(), 0, checked((int)stream.Length));
Console.WriteLine(BitConverter.ToString(stream.GetBuffer(), 0, checked((int)stream.Length)));
Console.WriteLine(resultJson);
Console.WriteLine(arr.ToString());
Assert.AreEqual(arr.ToString(), resultJson);
The assertion fails with the following message:
NUnit.Framework.AssertionException: Expected string length 2 but was 3. Strings differ at index 0.
And with the following output from BitConverter.ToString():
EF-BB-BF-5B-5D
Demo fiddle here.
The 5B-5D are the brackets, but what are the three preamble characters EF-BB-BF? A quick search shows it to be the UTF-8 byte order mark. Since RFC 8259 specifies that Implementations MUST NOT add a byte order mark (U+FEFF) to the beginning of a networked-transmitted JSON text you should omit the BOM by using new UTF8Encoding(false). Thus your code should look like:
JArray arr = new JArray();
var stream = new MemoryStream();
await using (var requestWriter = new StreamWriter(stream, new UTF8Encoding(false), leaveOpen: true))
{
var jsonWriter = new JsonTextWriter(requestWriter);
try
{
await arr.WriteToAsync(jsonWriter);
}
finally
{
await jsonWriter.CloseAsync();
}
}
var resultJson = Encoding.UTF8.GetString(stream.GetBuffer(), 0, checked((int)stream.Length));
Console.WriteLine(BitConverter.ToString(stream.GetBuffer(), 0, checked((int)stream.Length)));
Console.WriteLine(resultJson);
Console.WriteLine(arr.ToString());
Assert.AreEqual(arr.ToString(), resultJson);
Demo fiddle #2 here.

Why is streamwriter writting in the half of the file instead of overwriting

Here's my original file
[
{
"Quantity": 34,
"ShopOrderId": "51e400ff-76b8-4be4-851a-86e2681db960",
"Id": "ae7664cb-135e-4c01-b353-5ecf09ac56af",
"Direction": 2
},
{
"Accepted": true,
"Id": "7bfc2163-2274-4a0e-83b9-203cb376a8f8",
"Direction": 1
}
]
Now I want to load its content, remove one item, and overwrite the whole file
// items loaded from file
var result = new List<QueueItem>();
using (var fs = new FileStream(Path, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.Read))
{
// reading from file
using (var streamReader = new StreamReader(fs))
{
var json = streamReader.ReadToEnd();
result = JsonConvert.DeserializeObject<List<QueueItem>>(json) ?? new List<QueueItem>();
}
}
// removing unwanted items
result = result.Where(x => !IdsToRemove.Contains(x.Id)).ToList();
// overwriting whole file
using (var fs = new FileStream(Path, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.None))
{
using (var streamWriter = new StreamWriter(fs))
{
var json = JsonConvert.SerializeObject(result, Formatting.Indented);
streamWriter.Write(json);
}
}
The value of json in streamWriter.Write(json) is
[
{
"Accepted": true,
"Id": "7bfc2163-2274-4a0e-83b9-203cb376a8f8",
"Direction": 1
}
]
so it's valid.
But for some reason after performing Write it's a mess
[
{
"Accepted": true,
"Id": "7bfc2163-2274-4a0e-83b9-203cb376a8f8",
"Direction": 1
}
]-135e-4c01-b353-5ecf09ac56af",
"Direction": 2
},
{
"Accepted": true,
"Id": "7bfc2163-2274-4a0e-83b9-203cb376a8f8",
"Direction": 1
}
]
how can I make it that my streamWriter actually overwrite and also why does it happen when I'm opening new FileStream? shouldn't it be not aware of previous operations with streamReader? I guess it's the reason why it doesn't start from index 0.
Or maybe there is easier method to read/write from file while having locks (preventing other programs from modifying?)

Sounds like you need to use Truncate for the file mode when you open it to overwrite.

Reading the content of the stream "Stream" using "ReadToEnd()" in C#

I have a function which has an input parameter of type "stream".
I want to read the content of this stream and compare to a content of a file.
The following has been done:
public bool func(Stream stream, string tempFilePath, string dstFilePath)
{
string[] File1 = File.ReadAllLines(tempFilePath);
stream.Seek(0, SeekOrigin.Begin);
int line = 0;
StreamReader reader = new StreamReader(stream);
string[] File2 = reader.ReadToEnd().Split(
new[] { "\r\n", "\r", "\n" },
StringSplitOptions.None
);
while (line < File1.Length)
{
if (astFile2[line] != astFile1[line])
break;
line++;
}
bool ret = line == astFile1.Length && line == astFile2.Length - 1;
if (ret )
{
using (var dstFile = new FileStream(dstFilePath, FileMode.Create, FileAccess.Write))
{
stream.Seek(0, SeekOrigin.Begin);
stream.CopyTo(dstFile );
}
}
return ret;
}
The problem is that I can't take the content of the variable "stream" anymore using the command:
"stream.CopyTo(dstFile ); "
The content is empty!
The "position" of the stream using the stream.Seek(0, SeekOrigin.Begin); is correct.
I have used the stream.Seek(0, SeekOrigin.Begin); command after the ReadToEnd() line, but it did not help.
I have used also the stream.Position = 0, and did not help too.
That why I think there should be no problem by position.
What is it missed in this code?
Why the ReadToEnd() command results in emptying the stream?
For test, I also did the following:
string[] File2 = reader.ReadToEnd().Split(
new[] { "\r\n", "\r", "\n" },
StringSplitOptions.None
);
stream.Position = 0;
string[] File3 = reader.ReadToEnd().Split(
new[] { "\r\n", "\r", "\n" },
StringSplitOptions.None
);
The content of "File3" is still empty!

It seems that you are not showing us the actual faulty code.
I have paraphrased your code to a compilable console app, which does NOT reproduce the problem:
using System.IO;
namespace Demo
{
public static class Program
{
static void Main()
{
string sourceFile = #"D:\tmp\source.txt";
string destFile = #"D:\tmp\dest.txt";
MemoryStream ms = new MemoryStream();
var data = File.ReadAllBytes(sourceFile);
ms.Write(data, 0, data.Length);
ms.Position = 0;
func(ms, sourceFile, destFile);
}
public static void func(Stream stream, string tempFilePath, string dstFilePath)
{
File.ReadAllLines(tempFilePath);
stream.Seek(0, SeekOrigin.Begin);
var reader = new StreamReader(stream);
reader.ReadToEnd();
stream.Position = 0;
using (var dstFile = new FileStream(dstFilePath, FileMode.Create, FileAccess.Write))
{
stream.CopyTo(dstFile);
}
}
}
}
So I'm afraid that the answer to the question "what is missed in this code?" is "a reproducable problem".
It is crucial that you tell us the concrete type of the Stream that you are passing to func(). It is likely that is the cause of your problem - for example, it's a stream that can only be read once (although in that case, trying to change the stream position should really throw an UnsupportedOperationException).

I always make sure I call stream.Flush() before doing a ReadToEnd().

Prevent JsonTextReader from consuming the stream during deserialization

I'm using Json.Net to consume some seekable streams.
// reset the input stream, in case it was previously read
inputStream.Position = 0;
using (var textReader = new StreamReader(inputStream))
{
using (var reader = new JsonTextReader(textReader))
{
deserialized = serializer.Deserialize(reader, expectedType);
}
}
However, this method 'consumes' the stream, meaning the first contained valid Json token is removed from the stream.
That it very annoying. And meaningless, stream Position is provided to emulate a consumption, and 'reading' generally implies 'not modifying'.
Of course, I can dump the stream into a MemoryStream to protect my precious source stream, but that's a huge overhead, especially when doing trial-and-error on a deserialization.
If there is a way to to just 'read' and not 'read-and-consume', thanks for your help, I could not find documentation about that (and I hope this post will help others to google the solution ^^).

JsonTextReader is a forward-only reader, meaning it cannot be set back to a position earlier in the JSON to re-read a portion of it, even if the underlying stream supports seeking. However, the reader does not actually "consume" the stream, as you said. If you set the CloseInput property on the reader to false to prevent it from closing the underlying reader and stream when it is disposed, you can position the stream back to the beginning and open a new reader on the same stream to re-read the JSON. Here is a short program to demonstrate reading the same stream twice:
class Program
{
static void Main(string[] args)
{
string json = #"{ ""name"": ""foo"", ""size"": ""10"" }";
MemoryStream inputStream = new MemoryStream(Encoding.UTF8.GetBytes(json));
JsonSerializer serializer = new JsonSerializer();
using (var textReader = new StreamReader(inputStream))
{
for (int i = 0; i < 2; i++)
{
inputStream.Position = 0;
using (var reader = new JsonTextReader(textReader))
{
reader.CloseInput = false;
Widget w = serializer.Deserialize<Widget>(reader);
Console.WriteLine("Name: " + w.Name);
Console.WriteLine("Size: " + w.Size);
Console.WriteLine();
}
}
}
}
}
class Widget
{
public string Name { get; set; }
public int Size { get; set; }
}
Output:
Name: foo
Size: 10
Name: foo
Size: 10
Fiddle: https://dotnetfiddle.net/fftZV7

A stream may be consumed once read. The solution could be to copy it to a memory or file stream as below:
MemoryStream ms = new MemoryStream();
inputStream.CopyTo(ms);
ms.Position = 0;
using (var textReader = new StreamReader(ms))
(...)
Please let me know if it works.