Develop Reference

C# comfortable way to construct byte array from different objects

I am looking for a way to fast and simple implementation of this paradigm:
MyByteArray mb = new MyByteArray();
mb.Add<byte>(bytevalue);
mb.Add<float>(floatvalue);
mb.Add<string>(str);
mb.Add<MyClass>(object);
And then get byte[] from mb to send it as a byte packet via RPC call (to be decoded on the other side using the same technique).
I've found MemoryStream, but it looks like too overheaded for this simple operation.
Can you help me? Thank you.

What are you looking for is BinaryWritter. But it still needs a Stream to write on for pure logic reason. And the only Stream that fits in your need is MemoryStream.
Are you afraid of performance overhead ? You can create your MemoryStream from an existing byte array ;
byte [] buffer = new byte[1024];
using (var memoryStream = new MemoryStream(buffer))
{
using (var binaryWriter = new BinaryWriter(memoryStream))
{
binaryWriter.Write(1.2F); // float
binaryWriter.Write(1.9D); // double
binaryWriter.Write(1); // integer
binaryWriter.Write("str"); // string
}
}
// buffer is filled with your data now.

A tricky way to achieve this is to use a combination of builtin class in .net
class Program
{
static void Main(string[] args)
{
Program program = new Program();
var listBytes = new List<byte>();
listBytes.Add( program.CastToBytes("test"));
listBytes.Add(program.CastToBytes(5));
}
Note
for a custom object you have to define an implicit operator on how the properties or all the object should be converted
public byte[] CastToBytes<T>(T value)
{
//this will cover most of primitive types
if (typeof(T).IsValueType)
{
return BitConverter.GetBytes((dynamic)value);
}
if (typeof(T) == typeof(string))
{
return Encoding.UTF8.GetBytes((dynamic) value);
}
//for a custom object you have to define the rules
else
{
var formatter = new BinaryFormatter();
var memoryStream = new MemoryStream();
formatter.Serialize(memoryStream, value);
return memoryStream.GetBuffer();
}
}
}

This looks like the case for Protocol Buffers, you could look at at protobuf-net.
First, let's decorate the classes.
[ProtoContract]
class User
{
[ProtoMember(1)]
public int Id { get; set; }
[ProtoMember(2)]
public string Name { get; set; }
}
[ProtoContract]
class Message
{
[ProtoMember(1)]
public byte Type { get; set; }
[ProtoMember(2)]
public float Value { get; set; }
[ProtoMember(3)]
public User Sender { get; set; }
}
Then we create our message.
var msg = new Message
{
Type = 1,
Value = 1.1f,
Sender = new User
{
Id = 8,
Name = "user"
}
};
And now, we can use ProtoBuf's serializer to do all our work.
// memory
using (var mem = new MemoryStream())
{
Serializer.Serialize<Message>(mem, msg);
var bytes = mem.GetBuffer();
}
// file
using (var file = File.Create("message.bin")) Serializer.Serialize<Message>(file, msg);

C# Task Parallel Library first time slow

Trying to perform a number of HTTP Get requests in parallel, one per task. If I do the Gets through internet explorer they return almost instantaneously, but when calling in code through tasks, the first time I fire them off they take a good few seconds to return, but running a second time they return as I would expect. So either I'm doing something which is blocking something, or for some reason the threads are not starting? It's my first time trying to use TPL.
Here's the basic lookup class:
public class Lookup
{
public string Name { get; set; }
public string URL { get; set; }
public Lookup(string Name, string URL)
{
this.Name = Name;
this.URL = URL;
}
public LookupReturn DoLookup()
{
LookupReturn d = new LookupReturn();
d.Name = this.Name;
d.URL = this.URL;
WebRequest wrGETURL;
wrGETURL = WebRequest.Create(this.URL);
Stream objStream;
objStream = wrGETURL.GetResponse().GetResponseStream();
StreamReader objReader = new StreamReader(objStream);
string sLine = objReader.ReadToEnd();
d.Result = sLine;
return d;
}
}
And the return type is simply:
public class LookupReturn
{
public string Name { get; set; }
public string Result { get; set; }
public string URL { get; set; }
}
So the attempt to run this in parallel - am testing from a Winforms GUI but eventually will be in a WCF service.
public partial class Form1 : Form
{
private List<Lookup> Lookups = new List<Lookup>();
private async void btnRunLookups_Click(object sender, EventArgs e)
{
Lookups.Add(new Lookup("Name1", "http://geturl1 "));
Lookups.Add(new Lookup("Name2", "http://geturl2 "));
// More lookups…
int workerThreads, complete;
ThreadPool.GetMinThreads(out workerThreads, out complete);
ThreadPool.SetMinThreads(100, complete);
btnRunLookups.Visible = false;
List <Task<LookupReturn>> lookupTasks = new List<Task<LookupReturn>>();
foreach(Lookup dl in Lookups)
{
lbLookups.Items.Add("Starting task for " + dl.URL);
Task<LookupReturn> t = new Task<LookupReturn>(() => dl.DoLookup() );
lookupTasks.Add(t);
t.Start();
}
//await Task.WhenAny(
// Task.WhenAll(lookupTasks.ToArray<Task<LookupReturn>>()),
// Task.Delay(3000)
// );
// This takes a good few seconds the first time
await Task.WhenAll(lookupTasks.ToArray<Task<LookupReturn>>());
// Now I need to see which ones completed and returned a result
foreach (var x in lookupTasks)
{
if (x.IsCompleted)
{
lbLookups.Items.Add(x.Result);
}
else
{
// lbLookups.Items.Add("Not finished " + x.Result.Name);
}
}
btnRunLookups.Visible = true;
}

Other people have noted that HttpWebRequest can take a long time on its first request because it's looking for proxy information. Try setting its Proxy property to null.
wrGETURL = WebRequest.Create(this.URL);
wrGETURL.Proxy = null;

Most likely the problem is that the program is doing some first-time setup stuff (DNS resolution, proxy detection, etc.) on the first call to GetResponse. The proxy detection, in particular, can take a good long while.

Why Is The Imgur API slow compared to the website

public sealed class ImgurUpload
{
public event EventHandler<UploadCompleteEventArgs> UploadComplete;
public void PostToImgur(string location, string key, string name = "", string caption = "")
{
try
{
using (var webClient = new WebClient())
{
NameValueCollection values = new NameValueCollection
{
{ "image", ConvertToBase64(location) },
{ "key", key },
{ "name", name },
{ "caption", caption}
};
webClient.UploadValuesAsync(new Uri("http://api.imgur.com/2/upload.xml"), "POST", values);
webClient.UploadValuesCompleted += ((sender, eventArgs) =>
{
byte[] response = eventArgs.Result;
XDocument result = XDocument.Load(XmlReader.Create(new MemoryStream(response)));
if (UploadComplete != null)
UploadComplete(this, new UploadCompleteEventArgs(result));
});
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
private string ConvertToBase64(string imageLocation)
{
byte[] imageData = null;
using (FileStream fileStream = File.OpenRead(imageLocation))
{
imageData = new byte[fileStream.Length];
fileStream.Read(imageData, 0, imageData.Length);
}
return Convert.ToBase64String(imageData);
}
}
public class UploadCompleteEventArgs : EventArgs
{
public string Original { get; private set; }
public string ImgurPage { get; private set; }
public string DeletePage { get; private set; }
public UploadCompleteEventArgs(XDocument xmlDoc)
{
var objLinks = from links in xmlDoc.Descendants("links")
select new
{
original = links.Element("original").Value,
imgurPage = links.Element("imgur_page").Value,
deletePage = links.Element("delete_page").Value
};
Original = objLinks.FirstOrDefault().original;
ImgurPage = objLinks.FirstOrDefault().imgurPage;
DeletePage = objLinks.FirstOrDefault().deletePage;
}
}
Above is a class I wrote to upload an image to imgur using the Anonymous API. I have used the API in the past and have always found it to be considerably slower than the website uploader and slower than other .NET applications that use web requests to effectively send data to the website directly rather than using the API.
I posted the full class above as it may be something I'm doing (or not doing) that's causing the issue. I'd really appreciate it if anybody can identify the issue for me.
I did some fair testing earlier today and one result for example, is as followed:
800kb image via the imgur website = 35 seconds
800kb image via using my class = 1minute 20 seconds

The one you are uploading is ~35% bigger because you're uploading it as a STRING.
Upload via bytes and it should be just as fast.

Use Protobuf-net to stream large data files as IEnumerable

I'm trying to use Protobuf-net to save and load data to disk but got stuck.
I have a portfolio of assets that I need to process, and I want to be able to do that as fast as possible. I can already read from a CSV but it would be faster to use a binary file, so I'm looking into Protobuf-Net.
I can't fit all assets into memory so I want to stream them, not load them all into memory.
So what I need to do is expose a large set of records as an IEnumerable. Is this possible with Protobuf-Net? I've tried a couple of things but haven't been able to get it running.
Serializing seems to work, but I haven't been able to read them back in again, I get 0 assets back. Could someone point me in the right direction please? Looked at the methods in the Serializer class but can't find any that covers this case. I this use-case supported by Protobuf-net? I'm using V2 by the way.
Thanks in advance,
Gert-Jan
Here's some sample code I tried:
public partial class MainWindow : Window {
// Generate x Assets
IEnumerable<Asset> GenerateAssets(int Count) {
var rnd = new Random();
for (int i = 1; i < Count; i++) {
yield return new Asset {
ID = i,
EAD = i * 12345,
LGD = (float)rnd.NextDouble(),
PD = (float)rnd.NextDouble()
};
}
}
// write assets to file
private void Write(string path, IEnumerable<Asset> assets){
using (var file = File.Create(path)) {
Serializer.Serialize<IEnumerable<Asset>>(file, assets);
}
}
// read assets from file
IEnumerable<Asset> Read(string path) {
using (var file = File.OpenRead(path)) {
return Serializer.DeserializeItems<Asset>(file, PrefixStyle.None, -1);
}
}
// try it
private void Test() {
Write("Data.bin", GenerateAssets(100)); // this creates a file with binary gibberish that I assume are the assets
var x = Read("Data.bin");
MessageBox.Show(x.Count().ToString()); // returns 0 instead of 100
}
public MainWindow() {
InitializeComponent();
}
private void button2_Click(object sender, RoutedEventArgs e) {
Test();
}
}
[ProtoContract]
class Asset {
[ProtoMember(1)]
public int ID { get; set; }
[ProtoMember(2)]
public double EAD { get; set; }
[ProtoMember(3)]
public float LGD { get; set; }
[ProtoMember(4)]
public float PD { get; set; }
}

figured it out. To deserialize use PrefixBase.Base128 wich apparently is the default.
Now it works like a charm!
GJ
using (var file = File.Create("Data.bin")) {
Serializer.Serialize<IEnumerable<Asset>>(file, Generate(10));
}
using (var file = File.OpenRead("Data.bin")) {
var ps = Serializer.DeserializeItems<Asset>(file, PrefixStyle.Base128, 1);
int i = ps.Count(); // got them all back :-)
}

c# serialized data

I have been using BinaryFormatter to serialise data to disk but it doesn't seem very scalable. I've created a 200Mb data file but am unable to read it back in (End of Stream encountered before parsing was completed). It tries for about 30 minutes to deserialise and then gives up. This is on a fairly decent quad-cpu box with 8Gb RAM.
I'm serialising a fairly large complicated structure.
htCacheItems is a Hashtable of CacheItems. Each CacheItem has several simple members (strings + ints etc) and also contains a Hashtable and a custom implementation of a linked list. The sub-hashtable points to CacheItemValue structures which is currently a simple DTO which contains a key and a value. The linked list items are also equally simple.
The data file that fails contains about 400,000 CacheItemValues.
Smaller datasets work well (though takes longer than i'd expect to deserialize and use a hell of a lot of memory).
public virtual bool Save(String sBinaryFile)
{
bool bSuccess = false;
FileStream fs = new FileStream(sBinaryFile, FileMode.Create);
try
{
BinaryFormatter formatter = new BinaryFormatter();
formatter.Serialize(fs, htCacheItems);
bSuccess = true;
}
catch (Exception e)
{
bSuccess = false;
}
finally
{
fs.Close();
}
return bSuccess;
}
public virtual bool Load(String sBinaryFile)
{
bool bSuccess = false;
FileStream fs = null;
GZipStream gzfs = null;
try
{
fs = new FileStream(sBinaryFile, FileMode.OpenOrCreate);
if (sBinaryFile.EndsWith("gz"))
{
gzfs = new GZipStream(fs, CompressionMode.Decompress);
}
//add the event handler
ResolveEventHandler resolveEventHandler = new ResolveEventHandler(AssemblyResolveEventHandler);
AppDomain.CurrentDomain.AssemblyResolve += resolveEventHandler;
BinaryFormatter formatter = new BinaryFormatter();
htCacheItems = (Hashtable)formatter.Deserialize(gzfs != null ? (Stream)gzfs : (Stream)fs);
//remove the event handler
AppDomain.CurrentDomain.AssemblyResolve -= resolveEventHandler;
bSuccess = true;
}
catch (Exception e)
{
Logger.Write(new ExceptionLogEntry("Failed to populate cache from file " + sBinaryFile + ". Message is " + e.Message));
bSuccess = false;
}
finally
{
if (fs != null)
{
fs.Close();
}
if (gzfs != null)
{
gzfs.Close();
}
}
return bSuccess;
}
The resolveEventHandler is just a work around because i'm serialising the data in one application and loading it in another (http://social.msdn.microsoft.com/Forums/en-US/netfxbcl/thread/e5f0c371-b900-41d8-9a5b-1052739f2521)
The question is, how can I improve this? Is data serialisation always going to be inefficient, am i better off writing my own routines?

I would personally try to avoid the need for the assembly-resolve; that has a certain smell about it. If you must use BinaryFormatter, then I'd simply put the DTOs into a separate library (dll) that can be used in both applications.
If you don't want to share the dll, then IMO you shouldn't be using BinaryFormatter - you should be using a contract-based serializer, such as XmlSerializer or DataContractSerializer, or one of the "protocol buffers" implementations (and to repeat Jon's disclaimer: I wrote one of the others).
200MB does seem pretty big, but I wouldn't have expected it to fail. One possible cause here is the object tracking it does for the references; but even then, this surprises me.
I'd love to see a simplified object model to see if it is a "fit" for any of the above.
Here's an example that attempts to mirror your setup from the description using protobuf-net. Oddly enough there seems to be a glitch working with the linked-list, which I'll investigate; but the rest seems to work:
using System;
using System.Collections.Generic;
using System.IO;
using ProtoBuf;
[ProtoContract]
class CacheItem
{
[ProtoMember(1)]
public int Id { get; set; }
[ProtoMember(2)]
public int AnotherNumber { get; set; }
private readonly Dictionary<string, CacheItemValue> data
= new Dictionary<string,CacheItemValue>();
[ProtoMember(3)]
public Dictionary<string, CacheItemValue> Data { get { return data; } }
//[ProtoMember(4)] // commented out while I investigate...
public ListNode Nodes { get; set; }
}
[ProtoContract]
class ListNode // I'd probably expose this as a simple list, though
{
[ProtoMember(1)]
public double Head { get; set; }
[ProtoMember(2)]
public ListNode Tail { get; set; }
}
[ProtoContract]
class CacheItemValue
{
[ProtoMember(1)]
public string Key { get; set; }
[ProtoMember(2)]
public float Value { get; set; }
}
static class Program
{
static void Main()
{
// invent 400k CacheItemValue records
Dictionary<string, CacheItem> htCacheItems = new Dictionary<string, CacheItem>();
Random rand = new Random(123456);
for (int i = 0; i < 400; i++)
{
string key;
CacheItem ci = new CacheItem {
Id = rand.Next(10000),
AnotherNumber = rand.Next(10000)
};
while (htCacheItems.ContainsKey(key = rand.NextString())) {}
htCacheItems.Add(key, ci);
for (int j = 0; j < 1000; j++)
{
while (ci.Data.ContainsKey(key = rand.NextString())) { }
ci.Data.Add(key,
new CacheItemValue {
Key = key,
Value = (float)rand.NextDouble()
});
int tail = rand.Next(1, 50);
ListNode node = null;
while (tail-- > 0)
{
node = new ListNode
{
Tail = node,
Head = rand.NextDouble()
};
}
ci.Nodes = node;
}
}
Console.WriteLine(GetChecksum(htCacheItems));
using (Stream outfile = File.Create("raw.bin"))
{
Serializer.Serialize(outfile, htCacheItems);
}
htCacheItems = null;
using (Stream inFile = File.OpenRead("raw.bin"))
{
htCacheItems = Serializer.Deserialize<Dictionary<string, CacheItem>>(inFile);
}
Console.WriteLine(GetChecksum(htCacheItems));
}
static int GetChecksum(Dictionary<string, CacheItem> data)
{
int chk = data.Count;
foreach (var item in data)
{
chk += item.Key.GetHashCode()
+ item.Value.AnotherNumber + item.Value.Id;
foreach (var subItem in item.Value.Data.Values)
{
chk += subItem.Key.GetHashCode()
+ subItem.Value.GetHashCode();
}
}
return chk;
}
static string NextString(this Random random)
{
const string alphabet = "abcdefghijklmnopqrstuvwxyz0123456789 ";
int len = random.Next(4, 10);
char[] buffer = new char[len];
for (int i = 0; i < len; i++)
{
buffer[i] = alphabet[random.Next(0, alphabet.Length)];
}
return new string(buffer);
}
}

Serialization is tricky, particularly when you want to have some degree of flexibility when it comes to versioning.
Usually there's a trade-off between portability and flexibility of what you can serialize. For example, you might want to use Protocol Buffers (disclaimer: I wrote one of the C# ports) as a pretty efficient solution with good portability and versioning - but then you'll need to translate whatever your natural data structure is into something supported by Protocol Buffers.
Having said that, I'm surprised that binary serialization is failing here - at least in that particular way. Can you get it to fail with a large file with a very, very simple piece of serialization code? (No resolution handlers, no compression etc.)

Something that could help is cascade serializing.
You call mainHashtable.serialize(), which return a XML string for example. This method call everyItemInYourHashtable.serialize(), and so on.
You do the same with a static method in every class, called 'unserialize(String xml)', which unserialize your objetcs and return an object, or a list of objects.
You get the point ?
Of course, you need to implement this method in every of your class you want to be serializable.
Take a look at ISerializable interface, which represent exaclty what I'm describing. IMO, this interface looks too "Microsoft" (no use of DOM, etc), so i created mine, but principle is the same : cascade.