Convert byte[] to Base64 string with buffer (to avoid OOM) - c#

I need to convert a byte array into a Base64 string. Quite an easy task, really. Just do something like this:
string resultBase64 = Convert.ToBase64String(byteArray);
And it's done, right? Well, not in my case. Apparently, the server this application has to run on has some severe restrictions and an attempt to convert a 20 MB file causes an System.OutOfMemoryException.
I tried to cut the byteArray into chunks and get a Base64 string through a buffer.
int bufferSize = 3000;
System.Text.StringBuilder builder = new System.Text.StringBuilder();
for (int i = 0; i < byteArray.Length; i += bufferSize)
{
int currentBufferSize = Math.Min(bufferSize, byteArray.Length - i);
byte[] buffer = new byte[currentBufferSize];
Array.Copy(byteArray, i, buffer, 0, currentBufferSize);
builder.Append(Convert.ToBase64String(buffer));
}
string resultBase64 = builder.ToString();
However, this throws an OOM as well (I believe, because StringBuilder creates a new string on .ToString, which combined with builder itself causes an OOM).
What is the best way to turn byte[] into base64 string with minimum use of memory?


If you're using Json.NET to turn your object into JSON, it turns out that this already supports turning a byte array into a base64-encoded string, and in a way which doesn't take too much memory.
The trick is to make sure that you can stream the generated JSON directly to the HTTP request, rather than generating the entire request in memory before sending it.
public class TestModel
{
public byte[] Bytes { get; set; }
}
public static void Main(string[] args)
{
var model = new TestModel() { Bytes = new byte[] { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14 } };
var request = WebRequest.CreateHttp("...");
request.Method = "POST";
request.SendChunked = true;
using (var streamWriter = new StreamWriter(request.GetRequestStream()))
using (var writer = new JsonTextWriter(streamWriter))
{
new JsonSerializer().Serialize(writer, model);
}
var response = request.GetResponse();
}
The important points here are the request.SendChunked = true, and the fact that we open a new JsonTextWriter over the stream returned from request.GetRequestStream(), and serialize using the JsonTextWriter.

Related

Cant decompress message from NetworkSteam(zlib)

I have problem.
I send XML request to NetworkStream, and i get answer [head] - not compressed, and [body] - compressed zlib.
When i used code
public static string UnZipStr(byte[] input)
{
using (MemoryStream inputStream = new MemoryStream(input))
{
using (DeflateStream gzip =
new DeflateStream(inputStream, CompressionMode.Decompress))
{
using (StreamReader reader =
new StreamReader(gzip, System.Text.Encoding.UTF8))
{
return reader.ReadToEnd();
}
}
}
}
I get error
System.IO.InvalidDataException: "Bad data detected during decoding."
I did delele [head](just dont read first 100 byte = size head), but result is same.
When i used it
private MemoryStream Deflate(byte[] data, int level, bool zlib)
{
var memoryStream = new MemoryStream();
var deflater = new Deflater(level, !zlib);
using (DeflaterOutputStream outStream = new DeflaterOutputStream(memoryStream, deflater))
{
outStream.IsStreamOwner = false;
outStream.Write(data, 0, data.Length);
outStream.Flush();
outStream.Finish();
}
return memoryStream;
}
My message is not true read. I dont know level, because i used all - from 1 to 9. And two time - with flags true and false.
Piece of uncompress message:
(��]�r�F\u0012}�W�����n��r�5\0F���ƅ\n��uT[�J�-���_��(Q$H��,��v�+���i#����\u0003��Vl˔h���ӧ/x������ã��\u000e��z��-g���O��\u001f��ۭ<�~\U00017b7f=���������_8Γ�\aG��;t���\u000f/
)
How can i decompress my message?
p.s. when server send me non-compress message - i get true message.
all my code. I just dont know, where can i mistake?
public string postXMLData(string request)
{
int port = XXXXX;
string ip = "XXX.XXX.XXX.XX";
int idClient = XXX;
StringBuilder response = new StringBuilder();
int byteSizeBuf = 1024;
byte[] buff = new byte[byteSizeBuf * 100];
byte[] byteRequest = Encoding.UTF8.GetBytes(request);
ConnectionMode connectionMode = ConnectionMode.Plain;
QueryHeader test = new QueryHeader(idClient, byteRequest.Length, connectionMode);
byte[] data = new byte[byteSizeBuf];
byte[] res = new byte[test.MessageLength + byteRequest.Length];
test.HeaderData.CopyTo(res, 0);
byteRequest.CopyTo(res, test.HeaderData.Length);
TcpClient tcpClient = new TcpClient();
tcpClient.Connect(ip, port);
NetworkStream netStream = tcpClient.GetStream();
netStream.Write(res, 0, res.Length);
string final = response.ToString(); //куда записывать
int pos = 0;
do
{
int bytes = netStream.Read(data, 0, data.Length);
response.Append(Encoding.UTF8.GetString(data, 0, bytes));
data.CopyTo(buff, pos);
pos += data.Length;
}
while (netStream.DataAvailable);
string decodedStr = UnZipStr(buff);
netStream.Close();
tcpClient.Close();
return final;
}

I can't see any problem with your posted code. The problem must be in the code you didn't post. So, whatever is done in between of Deflate(...) and UnZipStr(...) it must have changed your compressed data.
Just try it yourself by directly using the output of Deflate(...) as the input of UnZipStr(...) :
Console.WriteLine(UnZipStr(
Deflate(System.Text.Encoding.UTF8.GetBytes("Hello World!"), 9, false).ToArray()
));

C# - Is there a limit to the size of an httpWebRequest stream?

I am trying to build an application that downloads a small binary file (20-25 KB) from a custom webserver using httpwebrequests.
This is the server-side code:
Stream UpdateRequest = context.Request.InputStream;
byte[] UpdateContent = new byte[context.Request.ContentLength64];
UpdateRequest.Read(UpdateContent, 0, UpdateContent.Length);
String remoteVersion = "";
for (int i = 0;i < UpdateContent.Length;i++) { //check if update is necessary
remoteVersion += (char)UpdateContent[i];
}
byte[] UpdateRequestResponse;
if (remoteVersion == remotePluginVersion) {
UpdateRequestResponse = new byte[1];
UpdateRequestResponse[0] = 0; //respond with a single byte set to 0 if no update is required
} else {
FileInfo info = new FileInfo(Path.Combine(Directory.GetCurrentDirectory(), "remote logs", "PointAwarder.dll"));
UpdateRequestResponse = File.ReadAllBytes(Path.Combine(Directory.GetCurrentDirectory(), "remote logs", "PointAwarder.dll"));
//respond with the updated file otherwise
}
//this byte is past the threshold and will not be the same in the version the client recieves
Console.WriteLine("5000th byte: " + UpdateRequestResponse[5000]);
//send the response
context.Response.ContentLength64 = UpdateRequestResponse.Length;
context.Response.OutputStream.Write(UpdateRequestResponse, 0, UpdateRequestResponse.Length);
context.Response.Close();
After this the array UpdateRequestResponse contains the entire file and has been sent to the client.
The client runs this code:
//create the request
WebRequest request = WebRequest.Create(url + "pluginUpdate");
request.Method = "POST";
//create a byte array of the current version
byte[] requestContentTemp = version.ToByteArray();
int count = 0;
for (int i = 0; i < requestContentTemp.Length; i++) {
if (requestContentTemp[i] != 0) {
count++;
}
}
byte[] requestContent = new byte[count];
for (int i = 0, j = 0; i < requestContentTemp.Length; i++) {
if (requestContentTemp[i] != 0) {
requestContent[j] = requestContentTemp[i];
j++;
}
}
//send the current version
request.ContentLength = requestContent.Length;
Stream dataStream = request.GetRequestStream();
dataStream.Write(requestContent, 0, requestContent.Length);
dataStream.Close();
//get and read the response
WebResponse response = request.GetResponse();
Stream responseStream = response.GetResponseStream();
byte[] responseBytes = new byte[response.ContentLength];
responseStream.Read(responseBytes, 0, (int)response.ContentLength);
responseStream.Close();
response.Close();
//if the response containd a single 0 we are up-to-date, otherwise write the content of the response to file
if (responseBytes[0] != 0 || response.ContentLength > 1) {
BinaryWriter writer = new BinaryWriter(File.Open(Path.Combine(Directory.GetCurrentDirectory(), "ServerPlugins", "PointAwarder.dll"), FileMode.Create));
writer.BaseStream.Write(responseBytes, 0, responseBytes.Length);
writer.Close();
TShockAPI.Commands.HandleCommand(TSPlayer.Server, "/reload");
}
The byte array responseBytes on the client should be identical to the array UpdateRequestResponse on the server, but it isn't. after about 4000 bytes every byte after that is set to 0 rather than what it should be (responseBytes[3985] is the last non-zero byte).
Does this happen because httpWebRequest has a size limit? I can't see any bug in my code that could be causing it and the same code works in other instances where I only have to pass around short sequences of data (less than 100 bytes).
The MSDN pages don't mention any size limit like this.

It's not that it has any artificial limit, this is a byproduct of the Streaming nature of what you're attempting to do. I have a feeling the following line is the offender:
responseStream.Read(responseBytes, 0, (int)response.ContentLength);
I've had this issue in the past (with TCP streams), it doesn't read all of the contents of the array, because they haven't all been sent over the wire yet. This is what I would try instead.
for (int i = 0; i < response.ContentLength; i++)
{
responseBytes[i] = responseStream.ReadByte();
}
That way, it will make sure to read all the way until the end of the stream.
EDIT
usr's BinaryReader based solution is much more efficient. Here is the relevant solution:
BinaryReader binReader = new BinaryReader(responseStream);
const int bufferSize = 4096;
byte[] responseBytes;
using (MemoryStream ms = new MemoryStream())
{
byte[] buffer = new byte[bufferSize];
int count;
while ((count = binReader.Read(buffer, 0, buffer.Length)) != 0)
ms.Write(buffer, 0, count);
responseBytes = ms.ToArray();
}

You are assuming that Read is reading as many bytes as you request. But the requested count is just an upper limit. You must tolerate reading small chunks.
You can use var bytes = new BinaryReader(myStream).ReadBytes(count); to read an exact number. Don't call ReadByte too often because that is very CPU intensive.
The best solution would be to step away from the fairly manual HttpWebRequest and use HttpClient or WebClient. All of this is automated for you and you get back a byte[].

Decompressing GZIP stream

I am trying to decompress a GZipped string which is part of response from a webservice. The string that I have is:
"[31,-117,8,0,0,0,0,0,0,0,109,-114,65,11,-62,48,12,-123,-1,75,-50,-61,-42,-127,30,122,21,111,-126,94,60,-119,-108,-72,102,44,-48,-75,-93,-21,100,56,-6,-33,-19,20,20,101,57,37,95,-14,94,-34,4,-63,-5,-72,-73,-44,-110,-117,-96,38,-88,26,-74,38,-112,3,117,-7,25,-82,5,24,-116,56,-97,-44,108,-23,28,24,-44,-85,83,34,-41,97,-88,24,-99,23,36,124,-120,94,99,-120,15,-42,-91,-108,91,45,-11,70,119,60,-110,21,-20,12,-115,-94,111,-80,-93,89,-41,-65,-127,-82,76,41,51,-19,52,90,-5,69,-85,76,-96,-128,64,22,35,-33,-23,-124,-79,-55,-1,-2,-10,-87,0,55,-76,55,10,-57,122,-9,73,42,-45,98,-44,5,-77,101,-3,58,-91,39,38,51,-15,121,21,1,0,0]"
I'm trying to decompress that string using the following method:
public static string UnZip(string value)
{
// Removing brackets from string
value = value.TrimStart('[');
value = value.TrimEnd(']');
//Transform string into byte[]
string[] strArray = value.Split(',');
byte[] byteArray = new byte[strArray.Length];
for (int i = 0; i < strArray.Length; i++)
{
if (strArray[i][0] != '-')
byteArray[i] = Convert.ToByte(strArray[i]);
else
{
int val = Convert.ToInt16(strArray[i]);
byteArray[i] = (byte)(val + 256);
}
}
//Prepare for decompress
System.IO.MemoryStream ms = new System.IO.MemoryStream(byteArray);
System.IO.Compression.GZipStream sr = new System.IO.Compression.GZipStream(ms,
System.IO.Compression.CompressionMode.Decompress);
//Reset variable to collect uncompressed result
byteArray = new byte[byteArray.Length];
//Decompress
int rByte = sr.Read(byteArray, 0, byteArray.Length);
//Transform byte[] unzip data to string
System.Text.StringBuilder sB = new System.Text.StringBuilder(rByte);
//Read the number of bytes GZipStream red and do not a for each bytes in
//resultByteArray;
for (int i = 0; i < rByte; i++)
{
sB.Append((char)byteArray[i]);
}
sr.Close();
ms.Close();
sr.Dispose();
ms.Dispose();
return sB.ToString();
}
The method is a modified version of the one in the following link:
http://www.codeproject.com/Articles/27203/GZipStream-Compress-Decompress-a-string
Sadly, the result of that method is a corrupted string. More specifically, I know that the input string contains a compressed JSON object and the output string has only some of the expected string:
"{\"rootElement\":{\"children\":[{\"children\":[],\"data\":{\"fileUri\":\"file:////Luciano/e/orto_artzi_2006_0_5_pixel/index/shapefiles/index_cd20/shp_all/index_cd2.shp\",\"relativePath\":\"/i"
Any idea what could be the problem and how to solve it?

Try
public static string UnZip(string value)
{
// Removing brackets from string
value = value.TrimStart('[');
value = value.TrimEnd(']');
//Transform string into byte[]
string[] strArray = value.Split(',');
byte[] byteArray = new byte[strArray.Length];
for (int i = 0; i < strArray.Length; i++)
{
byteArray[i] = unchecked((byte)Convert.ToSByte(strArray[i]));
}
//Prepare for decompress
using (System.IO.MemoryStream output = new System.IO.MemoryStream())
{
using (System.IO.MemoryStream ms = new System.IO.MemoryStream(byteArray))
using (System.IO.Compression.GZipStream sr = new System.IO.Compression.GZipStream(ms, System.IO.Compression.CompressionMode.Decompress))
{
sr.CopyTo(output);
}
string str = Encoding.UTF8.GetString(output.GetBuffer(), 0, (int)output.Length);
return str;
}
}
The MemoryBuffer() doesn't "duplicate" the byteArray but is directly backed by it, so you can't reuse the byteArray.
I'll add that I find funny that they "compressed" a json of 277 characters to a stringized byte array of 620 characters.
As a sidenote, the memory occupation of this method is out-of-the-roof... The 620 character string (that in truth is a 277 byte array) to be decompressed causes the creation of strings/arrays for a total size of 4887 bytes (including the 620 initial character string) (disclaimer: the GC can reclaim part of this memory during the execution of the method). This is ok for byte arrays of 277 bytes... But for bigger ones the memory occupation will become quite big.

Following on from Xanatos's answer in C# slightly modified to return a simple byte array. This takes a gzip compressed byte array and returns the inflated gunzipped array.
public static byte[] Decompress(byte[] compressed_data)
{
var outputStream = new MemoryStream();
using (var compressedStream = new MemoryStream(compressed_data))
using (System.IO.Compression.GZipStream sr = new System.IO.Compression.GZipStream(
compressedStream, System.IO.Compression.CompressionMode.Decompress))
{
sr.CopyTo(outputStream);
outputStream.Position = 0;
return outputStream.ToArray();
}
}

Retrieving POST data in C# ASP.NET

I'm having troubles to make my program work correctly - here I explain :
I have, on one hand a C# WinForms app which launches an instance of IE by using the "Navigate" method : myWebBrowser.Navigate(myUrl, "_blank", intarray, "");, with intarray defined like this : byte[] intarray = BitConverter.GetBytes(id);. On this side, it works.
On the other side, I have an ASP .NET WebForms application which has to retrieve this intarray. I've tried this.
if (HttpContext.Current != null)
{
if (Session["Authenticated"] == null)
{
var current = HttpContext.Current;
byte[] postdata = getpostdata(current);
}
}
private byte[] getpostdata(HttpContext CurrentContext)
{
MemoryStream ms = new MemoryStream();
CurrentContext.Request.InputStream.CopyTo(ms);
byte[] postdata = ms.ToArray();
return postdata;
}
// Convert a byte array to an Object
public int ByteArrayToInt(byte[] arrBytes)
{
if (BitConverter.IsLittleEndian) Array.Reverse(arrBytes);
int i = BitConverter.ToInt32(arrBytes, 0);
return i;
}
The problem seems to be in retrieving the Data in the getpostdata(HttpContext) function... I get a byte array with length = 0 instead of the one which is sent with length = 4...
Does anyone know how to make it work ?
Yann

var current = HttpContext.Current;
var sr = new StreamReader(Request.InputStream, Encoding.Default);
var postdata = sr.ReadToEnd();
above

Can't download complete image file from skydrive using REST API

I'm working on a quick wrapper for the skydrive API in C#, but running into issues with downloading a file. For the first part of the file, everything comes through fine, but then there start to be differences in the file and shortly thereafter everything becomes null. I'm fairly sure that it's just me not reading the stream correctly.
This is the code I'm using to download the file:
public const string ApiVersion = "v5.0";
public const string BaseUrl = "https://apis.live.net/" + ApiVersion + "/";
public SkyDriveFile DownloadFile(SkyDriveFile file)
{
string uri = BaseUrl + file.ID + "/content";
byte[] contents = GetResponse(uri);
file.Contents = contents;
return file;
}
public byte[] GetResponse(string url)
{
checkToken();
Uri requestUri = new Uri(url + "?access_token=" + HttpUtility.UrlEncode(token.AccessToken));
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(requestUri);
request.Method = WebRequestMethods.Http.Get;
WebResponse response = request.GetResponse();
Stream responseStream = response.GetResponseStream();
byte[] contents = new byte[response.ContentLength];
responseStream.Read(contents, 0, (int)response.ContentLength);
return contents;
}
This is the image file I'm trying to download
And this is the image I am getting
These two images lead me to believe that I'm not waiting for the response to finish coming through, because the content-length is the same as the size of the image I'm expecting, but I'm not sure how to make my code wait for the entire response to come through or even really if that's the approach I need to take.
Here's my test code in case it's helpful
[TestMethod]
public void CanUploadAndDownloadFile()
{
var api = GetApi();
SkyDriveFolder folder = api.CreateFolder(null, "TestFolder", "Test Folder");
SkyDriveFile file = api.UploadFile(folder, TestImageFile, "TestImage.png");
file = api.DownloadFile(file);
api.DeleteFolder(folder);
byte[] contents = new byte[new FileInfo(TestImageFile).Length];
using (FileStream fstream = new FileStream(TestImageFile, FileMode.Open))
{
fstream.Read(contents, 0, contents.Length);
}
using (FileStream fstream = new FileStream(TestImageFile + "2", FileMode.CreateNew))
{
fstream.Write(file.Contents, 0, file.Contents.Length);
}
Assert.AreEqual(contents.Length, file.Contents.Length);
bool sameData = true;
for (int i = 0; i < contents.Length && sameData; i++)
{
sameData = contents[i] == file.Contents[i];
}
Assert.IsTrue(sameData);
}
It fails at Assert.IsTrue(sameData);

This is because you don't check the return value of responseStream.Read(contents, 0, (int)response.ContentLength);. Read doesn't ensure that it will read response.ContentLength bytes. Instead it returns the number of bytes read. You can use a loop or stream.CopyTo there.
Something like this:
WebResponse response = request.GetResponse();
MemoryStream m = new MemoryStream();
response.GetResponseStream().CopyTo(m);
byte[] contents = m.ToArray();

As LB already said, you need to continue to call Read() until you have read the entire stream.
Although Stream.CopyTo will copy the entire stream it does not ensure that read the number of bytes expected. The following method will solve this and raise an IOException if it does not read the length specified...
public static void Copy(Stream input, Stream output, long length)
{
byte[] bytes = new byte[65536];
long bytesRead = 0;
int len = 0;
while (0 != (len = input.Read(bytes, 0, Math.Min(bytes.Length, (int)Math.Min(int.MaxValue, length - bytesRead)))))
{
output.Write(bytes, 0, len);
bytesRead = bytesRead + len;
}
output.Flush();
if (bytesRead != length)
throw new IOException();
}

Categories

Resources