Develop Reference

Streaming a large file into a database BLOB field

I have a large file on disk whose contents I need to store in a SqlServer database as a VARBINARY(MAX) field. By "contents", I mean using something like File.ReadAllBytes(). (I understand the pitfalls of this, but it's the current mode of operations, so I have to deal with it for now.)
I found this answer, which provides a way to stream a large Byte[] by using UPDATE.WRITE:
How to I serialize a large graph of .NET object into a SQL Server BLOB without creating a large buffer?
However, simply reading the contents of a large file into a Byte[] in memory leads to this problem:
OutOfMemoryException when I read 500MB FileStream
I might be overlooking something obvious, but I just can't work out how should I go about getting from a large file on disk, to the resulting storage into the database.

Using some hints I got from these two pages, I have an answer that works:
http://www.syntaxwarriors.com/2013/stream-varbinary-data-to-and-from-mssql-using-csharp/ (this looks a lot like the serialize SO answer, but there's more here...not sure who copied who!).
How do I copy the contents of one stream to another?
Basically, it uses the same methodology as the answer about serializing Blobs, but instead of using BinaryFormatter (a class I'm not fond of anyhow), it creates a FileStream that takes the path to the file, and an extension method to copy that stream into the target stream, or BlobStream, as the example named it.
Here's the extension:
public static class StreamEx
{
public static void CopyTo(this Stream Input, Stream Output)
{
var buffer = new Byte[32768];
Int32 bytesRead;
while ((bytesRead = Input.Read(buffer, 0, buffer.Length)) > 0)
Output.Write(buffer, 0, bytesRead);
}
}
So the trick was to link two streams, copying the data from one to another in chunked fashion, as noted in the comments.

Sending object from Android Client App to C# Server App

Java Code:
public class EMessage implements Serializable
{
private Bitmap image;
private String type;
EMessage()
{}
}
...
EMessage eMessage=new EMessage();
outToServer = new DataOutputStream(clientSocket.getOutputStream());
objectOutputStream=new ObjectOutputStream(outToServer);
objectOutputStream.writeObject(eMessage);
C# Code:
[Serializable]
class EMessage
{
private Bitmap image;
private String type;
EMessage()
{ }
}
client = server.AcceptTcpClient();
Connected = client.Connected;
ns = client.GetStream();
IFormatter formatter = new
System.Runtime.Serialization.Formatters.Binary.BinaryFormatter();
EMessage recievedmsg = (EMessage)formatter.Deserialize(ns);
When I send an object from Android Client App (java coded) and I recieve the object in C# Server App but with an Exception.
"The Input Stream is not a valid binary format. The Starting Content(in bytes) are:
00-05-73-72-00-1D-63-6F-6D-2E etc";
Please suggest any simple solution. My project isn't that much complex. I just need to send an EMessage object.

Serialization formats are specific to the platforms, and Java and .NET serialization aren't compatible with each other. Use JSON instead (and it's easier to debug as well).

Why not use SOAP, here's an article on exactly what you're doing (android to .net)
http://www.codeproject.com/Articles/29305/Consuming-NET-Web-Services-via-the-kSOAP-library

I suggest you drop the Serialization for the above mentioned reasons (Java serialization being different from C# serialization), and transfer your data between your Java and C# applications in plain byte arrays.
You can convert your Bitmap image to a byte array like so (taken from this post on SO):
Bitmap bmp = intent.getExtras().get("data");
ByteArrayOutputStream stream = new ByteArrayOutputStream();
bmp.compress(Bitmap.CompressFormat.PNG, 100, stream);
byte[] byteArray = stream.toByteArray();
Of course you could change the CompressFormat if circumstances so require. After that, you could convert your type string to a byte array too, and add a null-terminator to the end of it.
Once you're there, you can send your type string first, and add the byte array of the bitmap after it. On the C# end, you could read the incoming data until you reach the 0 terminator, at which point you'll know you've read the string portion of your EMessage object, and then read the rest of the bytes you've sent over and parse them into a Bitmap object.
That way you'll be sure that between your Java and C# implementations, you won't run into any compatibility issues. It may require a bit more code and a little more understanding to do, but it's far more reliable than serializing between two languages.

Cannot Serialize Neurotec NTemplate using c# - protobuf-net

I have a Neurotec NTemplate with one Finger record. Now i want to Serialize it with c# - protobuf-net. I dont getting any exeption but my MemoryStream is emplty.
what might be the problem?
code Is below:(where tenPrintTemplate is a NTemplate)
tenPrintTemplate.AddFingers(fingerPrintTemplate.Save());
//start Proto Buffer serialization
MemoryStream stream = new MemoryStream();
RuntimeTypeModel.Default.InferTagFromNameDefault = true;
RuntimeTypeModel.Default.Add(typeof(NTemplate), false);
ProtoBuf.Serializer.Serialize<NTemplate>(stream, tenPrintTemplate);

Here you've told it not to apply any standard pattern / configuration logic:
RuntimeTypeModel.Default.Add(typeof(NTemplate), false);
so you have basically told it "serialize nothing". If you specify false, it expects you to tell it how you want it to work, for example by using Add on the MetaType that is returned. I suspect you could also just specify true if it has suitable attributes.
Note that 0 is a perfectly reasonable length for protobuf-net and an object that doesn't have anything interesting to mention on the wire.

I have got the solution of serializing Neurotec's NTemplate using C# Protobuf-net. i'm adding the solution code below. if anyone face the same problem please use it as your solution.
//Its a NTemplate of TenPrint
tenPrintTemplate.AddFingers(fingerPrintTemplate.Save());
//start Proto Buffer serialization
MemoryStream stream = new MemoryStream();
int tenpritnTemplateSize = tenPrintTemplate.GetSize();
NBuffer buffer = new NBuffer(tenpritnTemplateSize);
// saving fingers template to buffer.
tenPrintTemplate.Save(buffer);
ProtoBuf.Serializer.Serialize<byte[]>(stream, buffer.ToArray());

How to deserialize all Soap serialized objects

I am soap serializing multiple objects and appending them to a single file,then I am desirializing to have all the objects to tree view
I am using this part of code of desirialization
FileStream fs = new FileStream(fName, FileMode.Open);
while (fs.Position < fs.Length)
{
arraylizt.Add(sf.Deserialize(fs));
}
It works well, but sometimes the last object in the file is not desirialized.
I am not getting why its not desirializing the final object sometimes
Please help me or suggest me any other way to deserialize

Have you ensure you've flushed your stream upon writing before close (or closed the writer rather than the underlying stream).

very large string in memory

I am writing a program for formatting 100s of MB String data (nearing a gig) into xml == And I am required to return it as a response to an HTTP (GET) request .
I am using a StringWriter/XmlWriter to build an XML of the records in a loop and returning the
using (StringWriter writer = new StringWriter())
using (writer = XmlWriter.Create(writer, settings)) //where settings are the xml props
writer.ToString()
during testing I saw a few --out of memory exceptions-- and quite clueless on how to find a solution? do you guys have any suggestions for a memory optimized delivery of the response?
is there a memory efficient way of encoding the data? or maybe chunking the data --
I just can not think of how to return it without building the whole thing into one HUGE string object
thanks
--
a few clarifications --
this is an asp .net webservices app over a gigabit ethernet link as josh noted. I am not very familiar with it so still a bit of a learning curve.
I am using XMLWriter to create the XML and create a string out of it using String
some stats --
response xml size = about 385 megs (my data size will grow very quickly to way more than this)
string object size as calculated by a memory profiler = peaked at 605MB
and thanks to everyone who responded...

Use XmlTextWriter wrapped around Reponse.OutputStream to send the XML to the client and periodically flush the response. This way you never have to have more than a few mb in memory at any one time (at least for sending to the client).

Can't you just stream the response to the client? XmlWriter doesn't require its underlying stream to be buffered in memory. If it's ASP.NET you can use the Response.OutputStream or if it's WCF, you can use response streaming.

HTTP get for 1 gig? that's a lot! Perhaps you should reconsider.
At least gziping the output could help.

You should not create XML using string manipulation.
Instead, you should use the XmlTextWriter, XmlDocument, or (in .Net 3.5) XElement classes to build an XML tree in memory, then write it directly to Response.OutputStream using an XmlTextWriter.
Writing directly to an XmlTextWriter that wraps Response.OutputStream wil be most efficient (you'll never have an entire element tree in memory at once), but will be somewhat more complicated.
By doing it this way, you will never have a single string (or array) containing the entire object, and should thus avoid OutOfMemoryExceptions.

Had a similar problem, hope this will help someone. My initial code was:
var serializer = new XmlSerializer(type);
string xmlString;
using (var writer = new StringWriter())
{
serializer.Serialize(writer, objectData, sn); // OutOfMemoryException here
xmlString = writer.ToString();
}
I ended up replaceing StringWriter with MemoryStream and this solved my problem
using (var mem = new MemoryStream())
{
serializer.Serialize(mem, objectData, sn);
xmlString = Encoding.UTF8.GetString(mem.ToArray());
}

You'll have to return each record (or a small group of records) on their own individual GETs.