Develop Reference

Serialization takes too long

I serialize all the objects contained in a list each time a new object is added (kind of an history if the application crashes).
Serialization is done within a millisecond when adding the first ~20 objects, but from now on, each newly added object is going to take longer and longer (if I add 10 more it can take 10 minutes to serialize).
As I said, it's kind of an history. So if I restart app, every object in the JSON file is added back to the list. Let's admit I had 20 objects before closing app.
Now I can add 20 more objects, so it will serialize 40 within a millisecond.
Once again, if I add even more objects, I'll have to close and restart app.
I don't understand why multiple serializations in the same instance of the app takes that long.
Here's the serialization code:
public static void SerializeAll()
{
string output = JsonConvert.SerializeObject(ListOfModelToSerialize, Formatting.Indented);
IO.StreamWriter writer = new IO.StreamWriter("A:\history.json");
writer.Write(output);
writer.Close();
}
Deserialization code:
public static List<ModelToSerialize> DeserializeAll()
{
if (IO.File.Exists("A:\history.json"))
{
string input = IO.File.ReadAllText("A:\history.json");
var result = JsonConvert.DeserializeObject<List<ModelToSerialize>>(input);
return result;
}
else
{
return new List<ModelToSerialize>();
}
}
And I only serialize 4 properties from my model. Here's the serialized model output:
{
"an_integer": 1,
"a_string": "...",
"a_list_of_string": [],
"another_list_of_string": []
}
UPDATE 1:
It seems like the integer (ID) I'm serializing is the problem.
In fact, this ID is not assigned at the model creation because it changes everytime a new object is added to the list, as the list is sorted alphabetically. So to get the ID, I do this:
[JsonProperty(PropertyName = "id")]
public int Id
{
get
{
if (_id > 0)
{
return _id;
}
else
{
int id = Properties.Settings.Default.PREVIOUS_MAX_ID + 1;
foreach (File file in _directory.Files)
{
if (file == this)
{
return id;
}
else if (file.Id == id)
{
id += 1;
}
}
return 0;
}
}
}
So the reason why it doesn't take longer to serialize the first 20 objects after the app restart is that the ID is directly assigned at deserialization.
So I have to adapt the ID retrieval.
UPDATE 2:
As the ID retrieval was the performance issue, I fixed it by assigning the ID to each object each time a new object is added.
Like that, when the object ID is called, there is no more iteration in the model.
The following block is my custom Add() method for my list of objects:
public new void Add(File file)
{
if (!base.Contains(file))
{
base.Add(file);
base.Sort(new Comparer());
for (int i = 0; i < Count; i++)
{
this[i].Id = Properties.Settings.Default.PREVIOUS_MAX_ID + i;
}
}
}
I added ~100 new objects to the current instance of the app, and serialization is not taking years anymore.

Serialization (JSON etc) is best when used to communicate a discrete bit of data from one medium to another, such as making network API calls. It can also be used to store data to disk as you are doing.
But it's not a great option when you are frequently updating that data because the most popular formats (JSON, XML) are not structured so that new data can just be appended to existing data. That's why it needs to be serialized every time. And so of course as the data gets larger serialization will take longer.
For your purpose I would suggest a database like SQLite. You could still serialize each individual row to JSON if the data doesn't lend itself well to columns, but you'll be able to append new data to the database without having to rewrite the old. And if you use transactions you can also ensure that the data keeps its integrity even if your app crashes.

For any kind of performance issue I would recommend to measure, ideally using a performance profiler that can tell you exactly what part of the code is slow.
Some general suggestions
You probably want to have a limit on how much history you keep. Otherwise there is a risk that your application will get slower and slower as you accumulate objects. This also lets you test worst case performance.
Use a faster serializer, while json.net is good, using some variant of binary serialization, like protobuf.net, typically results in less data and time (benchmark), at the cost of not being humanly readable. You might also consider a fast compressor like lz4 to reduce the file size and therefore disk-performance.
Change your model so you do not have to overwrite all data. Either serialize to different files, or use some serialization method that lets you append messages to the same file.
Limit save-frequency. You might not need to save history for each and every change, it might be sufficient to save every 5s or so.
If you have not already done so, ensure that at least saving data is done asynchronously so it does not block the UI thread. This might require some way to access your data in a thread safe manner so ensure it cannot change while it is being serialized.

Need to update list item and then put it in textfile

So I am doing this c# program which basically is a coffee machine. Asks the user to register and adds the instance in a textfile, then login, then asks which coffee he wants from a provided list, and finally increments a property named "coffee points". The problem is that I have no idea how I am going to update the coffee points inside my textfile. I can get the coffee points and increment them by 1 and then print them in console, however, I dont know how to update just the coffee points in my textfile of a certain user.
Below is the code I am using to retrieve the coffee points of the logged in user and increment them by 1, and store them in the variable "newCoffeePoints".
int newCoffeePoints = 0;
for (int i = 0; i < customerList.Count; i++)
{
if (customerList[i].iUsername == cu2.iUsername)
{
customerList[i].iCoffeePoints += 1;
newCoffeePoints = customerList[i].iCoffeePoints;
break;
}
}

Well, for this exact purpose we have so many different types of databases such as Mongo, SQL etc.
But if you're trying to serialize/deserialize object and save it to a file among other types of serialization you might want to consider JSON format. The libarary Newtonsoft.js is what we (C# developers) usually use for that.
You could create a virtual structure that reflects what you need.
Since we might want to find object by key a Dictionary<TKey, TValue> collection suits much better than List<T>.
Do whatever operation you need to do with users in a Dictionary<key, value> and then serialize it to JSON and save it to disc using the File.WriteAllText(json) method.
Whether to do it after each update (so that if app crashes all the data is saved) or do it once when the app is about to finish execution will depend on your needs.
The field with dictionary could be something along the lines of:
// CKeyType here is a type of customer's id field such as int, string etc.
private static readonly CustomerDict = new Dictionary<CKeyType, Customer>();

You can easily solve this problem by using JSON and Newtonsoft Library. (Fabjan mentioned above)
Here is a tutorial link from Microsoft Virtual Academy that you can follow.
https://mva.microsoft.com/en-US/training-courses/introduction-to-json-with-c-12742?l=xxtX274UB_8805494542

Looking for a viable solution to using XML file as constant data source

I am looking for suggestions on how to best attack my problem.
I have a web application that will utilize a database and XML file as its data source.
Updating the DB is no issue, dealing with the XML file is.
First understand the web app is a bypass to a current(and bad performing production system). So there is no flexibility there.
So in case of the XML file the following steps will need to be performed if a user adds/update/delete a field.
Pull XML file from linux server
Parse XML file into POCOs for the UI
Recreate the XML file from the POCOs with the modifications
Push XML file to the linux server
The only good thing in my favor is that I won't have very many users using this app at a given time.

I would highly recommend looking at the DataContractSerializer for your serialization needs. Your objects would have to follow the same format as the XML, but there is no reason that you can't make those objects your model. Here is an example of marking up a POCO to serialize automatically to and from XML:
[DataContract]
public class Account
{
[DataMember, XmlAttribute]
public string Name { get; set; }
[DataMember]
public double Balance { get; set; }
}
And then you use the serialization code like this:
string serializedAccount = null;
XmlSerializer serializer = new XmlSerializer(typeof(Account));
using(StringWriter writer = new StringWriter())
{
serializer.Serialize(writer, myAccountObject);
serializedAccount = writer.ToString();
}
// serializedAccount should look something like this:
//
// <Account Name="MyName">
// <Balance>100</Balance>
// </Account>
//
// Each of the DataMember and DataContract attributes allow you to override
// names so that you can make them lower case, or provide a more meaningful
// name in your POCO.
Account deserializedAccount = null;
using(StringReader reader = new StringReader(serializedAccount))
{
deserializedAccount = serializer.DeSerialize(reader) as Account;
}
A couple things to keep in mind:
DataContract is an opt in policy. Members are ignored unless you explicitly mark them as ignored.
If you want to explicitly mark a field or property as ignored, use the [IgnoreDataMember] attribute. It can be handy even just for documentation purposes.
Nothing in the serializer prevents you from implementing INotifyPropertyChanged so all your properties can be bindable (handy for WPF)
If your XML format is not what you want to use as a model for your UI, you still may need to map properties in one object to properties in another object. This is still orders of magnitude easier than hand writing the XML reading/writing code yourself.
NOTE: if your server implements a proper SOAP web API, then WCF can take the WSDL and generate the bindings to the API and the objects that need to be serialized. There's a bit more work involved if it's a REST API, but that's also supported.

Basic Booksleeve plus Protobuf-net plus Lists/SortedSets, implementations?

I have some fairly basic questions regarding the use of booksleeve in conjunction with protobuf-net. Now I have implemented a singleton class to manage the connections so I am reusing the same connection many times as recommended. Now I several questions regarding actual use of the combo:
What is the difference/importance of the "db" int and the "key" string?
How would I serialize a bunch of objects into a SortedSet/List using protobuf-net?
How would I deserialize a bunch of objects from a SortedSet/List using protobuf-net?
I was thinking that I should use the Range() method for retrieval:
public IList<T> RetrieveAllAsList()
{
var conn = RedisConnectionManager.Current.GetConnection();
conn.Open();
int length = (int)conn.Lists.GetLength(10, "idk").Result;
byte[][] data = conn.Lists.Range(10, "idk", 0, length-1).Result;
List<T> output = new List<T>();
for (int i = 0; i < data.Length; i++)
{
using (MemoryStream ms = new MemoryStream(data[i]))
{
output.Add(Serializer.Deserialize<T>(ms));
}
}
conn.Close(false);
return output;
}
I would appreciate any help in this matter. Thank you.

What is the difference/importance of the "db" int and the "key" string?
Redis allows you to partition your data into several databases. For example, on the stackexchange websites, you may store stackoverflow.com related stuff in db=0, and programmers.se.com related stuff in db = 1.
Within each database, you have key = value pairs. The key is always a String. The value can be one of the following five data types - String, List, Set, Sorted Set or a Map.
A traditional Redis client does not force you to provide the database number. If you don't provide one, it assumes db=0. But Booksleeve requires you to provide the database number. If you don't care, just pass 0 to all API calls.
The string key must be unique though, and is totally application specific. For example, to store a user object, the usual technique is to have a key like user:1190099, and the value as a Map with key=value pairs like {"name":"singlelabs", "id":1190099 ... }
How would I serialize a bunch of objects into a SortedSet/List using protobuf-net?
How would I deserialize a bunch of objects from a SortedSet/List using protobuf-net?
First, you need to decide whether you want to use protocol buffers or not.
Protocol Buffers can serialize/deserialize complex objects into a binary blob. This blob is opaque to Redis. In other words, you can set or get this binary blob, but you cannot use any other Redis features.
If you really want to use Redis' lists and sorted sets, you should use the APIs provided by BookSleeve. I haven't used Booksleeve, but the following two interfaces should explain how to insert data into a list / sortedset.
http://code.google.com/p/booksleeve/source/browse/BookSleeve/ISortedSetCommands.cs
http://code.google.com/p/booksleeve/source/browse/BookSleeve/IListCommands.cs

Sripathi already did a good job of explaining what the DB key is. The reason it is required is that BookSleeve works as a multiplexer, so of you are using a DB it needs to be passed to avoid race conditions when different callers are using different databases. Just pass 0 if you aren't using multiple databases.
Re serialization/deserialization - this is deliberately done separately, so that the caller can use the appropriate serialization for them - so BookSleeve only knows about strings and blobs. Your serialiazation code looks ok though.
The only critique I would have is that there is no need to get the length first. The intellisense should illustrate the usage - I expect passing -1 or int.MaxValue will list all the items without needing to get the length first (apologies for being vague, but I'm in a departure lounge). This avoids a network trip.

Where to store configuration information

I have a console application that I am rebuilding from C to C#. This application has to be able to support the legacy method of storing information like parameters from a command-line and parameters from a file (called the system parameters) that customize each run. The system parameters file is in plain-text with a simple key-value structure.
My questions are:
Should I combine these different parameters into a single Configuration object?
How would I call this configuration object from the code to store parameters?
How would I call this configuration object from the code to retrieve parameters?
Should this object be strongly-typed?
I will need access to this structure from a lot of different places in the code. What is the most elegant way to retrieve the values in the object without passing the object itself around everywhere?
I have a feeling that it should be a single, strongly-typed object and that it should be an instantiated object that is retrieved from a repository with a static retrieval method however I really want validation of this method.

I would use a single configuration object like the following:
using System;
using System.IO;
using System.Reflection;
public sealed class Setting {
public static int FrameMax { get; set; }
public static string VideoDir { get; set; }
static readonly string SETTINGS = "Settings.ini";
static readonly Setting instance = new Setting();
Setting() {}
static Setting() {
string property = "";
string[] settings = File.ReadAllLines(SETTINGS);
foreach (string s in settings)
try {
string[] split = s.Split(new char[] { ':' }, 2);
if (split.Length != 2)
continue;
property = split[0].Trim();
string value = split[1].Trim();
PropertyInfo propInfo = instance.GetType().GetProperty(property);
switch (propInfo.PropertyType.Name) {
case "Int32":
propInfo.SetValue(null, Convert.ToInt32(value), null);
break;
case "String":
propInfo.SetValue(null, value, null);
break;
}
} catch {
throw new Exception("Invalid setting '" + property + "'");
}
}
}
Since this is a singleton, it will create one and only one instance of itself the first time a public static property is referenced from the Setting object.
When the object is created, it reads from the Settings.ini file. The settings file is a plain-text file with a simple key : value structure that might look like this:
FrameMax : 12
VideoDir : C:\Videos\Best
The object uses reflection to discover each property and to store its initial value. In this example, two properties have been defined:
public static int FrameMax { get; set; }
public static string VideoDir { get; set; }
The code as written handles Int32 and String types. By adding additional case statements to the switch statement, you could easily add support for types like Float and Decimal.
To change a setting, you would use something like:
Setting.FrameMax = 5;
To retrieve a setting, you would use something like:
if (Setting.FrameMax > 10) ...
You'll notice that all the properties are strongly-typed. Also, you don't have to pass the Setting object around, as all the Setting properties are static and always available everywhere.
I hope this idea is helpful.

I like using Settings. These can be generated automatically either by creating a settings file using the Add New File dialog box, or by adding a default settings file from project properties.
Each setting may be in user or application scope, which controls whether or not the user can change them or they are restricted to their default values. They are easily saved with the Save() method and loaded automatically into the static Default property.
This class seems to be for application or user-based settings. I'm looking for per-run settings. Would you still recommend using this class in that case? – x97mdr
Yes. If you have both user/application based settings and per-run settings you should use two different classes - the normal (saved) settings and the per-run settings.
As long as you don't save the per-run settings, you should be safe and settings are still quite easy to use. These are static settings though. If the same application run needs several instances - this is the wrong approach.

I find that whenever I have to deal with a legacy system, sticking with the old format almost always works best. Often times there are other people using the legacy formats for other tasks (like automation of the app, for example), so if you recode the way the application handles inputs, you might break other systems.
On the other hand, if you are pretty confident that you know all the people using the system, and they tell you that they don't care if you change these types of things, I would probably move everything to XML. Besides all the nice features of XML from an application point of view (like being in ASCII so it's easily modified by humans, being self-documenting, etc ...), XML is also time-saving, in that you don't have to write your own I/O or parser. There's already a wide variety of libraries out there, particularly in .NET 3.0/3.5, that do very well. (As you're moving to C#, I'm guessing you're already thinking along these lines :)
So ultimately, you'd have to base your decision on cost-to-implement: if you lower your cost of implementation by moving to XML or similar, make sure that you don't raise other people's cost of implementation to move to your new application framework.
Good luck!

XmlDocument - you can generate a class definition using XSD.exe