Develop Reference

Uploading Large Files to WCF from Xamarin Android App Crashes

I'm trying to upload a large video (1 GB+) from my xamarin app and it keeps crashing once it reaches about 0.5 GB of my file. The only way I've found to get the videos to post to my WCF service while sending data along with it is using the MultiPart logic but I'm not sure if I'm running out of memory or what because even in debug mode, it simply crashes without any real error message.
I'm trying to run it on a native device (not a sim) and it's a Samsung Galaxy S9 with Android 9.
Here's the upload code that I'm using: (p.s. - as a test, I tried putting the WriteAsync into a for loop thinking that maybe trying to write the whole gig was the problem, but the result was the same. That's why you'll see the MAXFILESIZEPART constant in there which is just an int equal to 10000000.)
private async Task<byte[]> GetMultipartFormDataAsync(Dictionary<string, object> postParameters, string boundary)
{
try
{
using (Stream formDataStream = new System.IO.MemoryStream())
{
bool needsCLRF = false;
foreach (var param in postParameters)
{
// Thanks to feedback from commenters, add a CRLF to allow multiple parameters to be added.
// Skip it on the first parameter, add it to subsequent parameters.
if (needsCLRF)
await formDataStream.WriteAsync(Encoding.UTF8.GetBytes("\r\n"), 0, Encoding.UTF8.GetByteCount("\r\n"));
needsCLRF = true;
if (param.Value is FileParameter)
{
FileParameter fileToUpload = (FileParameter)param.Value;
// Add just the first part of this param, since we will write the file data directly to the Stream
string header = string.Format("--{0}\r\nContent-Disposition: form-data; name=\"{1}\"; filename=\"{2}\"\r\nContent-Type: {3}\r\n\r\n",
boundary,
param.Key,
fileToUpload.FileName ?? param.Key,
fileToUpload.ContentType ?? "application/octet-stream");
await formDataStream.WriteAsync(Encoding.UTF8.GetBytes(header), 0, Encoding.UTF8.GetByteCount(header));
// Write the file data directly to the Stream, rather than serializing it to a string.
if (fileToUpload.File.Length > MAXFILESIZEPART)
{
for (var i = 0; i < fileToUpload.File.Length; i += MAXFILESIZEPART)
{
var len = i + MAXFILESIZEPART > fileToUpload.File.Length
? fileToUpload.File.Length - i
: MAXFILESIZEPART;
await formDataStream.WriteAsync(fileToUpload.File, i, len);
}
}
else
{
await formDataStream.WriteAsync(fileToUpload.File, 0, fileToUpload.File.Length);
}
}
else
{
string postData = string.Format("--{0}\r\nContent-Disposition: form-data; name=\"{1}\"\r\n\r\n{2}",
boundary,
param.Key,
param.Value);
await formDataStream.WriteAsync(Encoding.UTF8.GetBytes(postData), 0, Encoding.UTF8.GetByteCount(postData));
}
}
// Add the end of the request. Start with a newline
string footer = "\r\n--" + boundary + "--\r\n";
await formDataStream.WriteAsync(Encoding.UTF8.GetBytes(footer), 0, Encoding.UTF8.GetByteCount(footer));
// Dump the Stream into a byte[]
formDataStream.Position = 0;
byte[] formData = new byte[formDataStream.Length];
formDataStream.Read(formData, 0, formData.Length);
return formData;
}
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
}
And it's eventually failing on the following line
await formDataStream.WriteAsync(fileToUpload.File, i, len);
but only after a certain point (about 500MB) so I'm assuming it's a memory issue but it doesn't say so. Is there a better way to accomplish this task? I'm doing it so that it also records the progress as the upload happens. I'm trying to accomplish something similar to uploading large videos via the facebook app so that it will upload in the background while you continue working. It works great when working with smaller files (i.e. - < 500 MB) but this is the first time I've tried a file that was almost a gig in size.
NOTE: This happens BEFORE it starts posting anything to the server so it's not IIS or WCF related. This code crashes just writing the bytes to the memory stream.
Any suggestions?
Thanks!

According to your description, the service will stop at a certain time point, and because the file you transfer is about 1G, it is likely to be sendtimeout.No transfer completed within the specified time, causing exception。The SendTimeout that specifies how long the write operation has to complete before timing out. The default value is 1 minute.
I set sendtimeout to 15 seconds in my configuration file.If the data takes more than 15 seconds, an exception will occur. You can set it to a higher value to avoid timeout and exception.
For information about sendtimeout, please refer to the following link:
https://learn.microsoft.com/en-us/dotnet/api/system.servicemodel.channels.binding.sendtimeout?view=dotnet-plat-ext-3.1
UPDATE
I think it might be a memory overflow problem.Large file may cause memory overflow, unable to read at the same time.
You can refer to the following links for solutions
https://learn.microsoft.com/en-us/archive/blogs/johan/are-you-getting-outofmemoryexceptions-when-uploading-large-files

Mailkit - Get HTML and Text parts without having full mime message

I want to download text and html parts from mime message and store it in the database and later if needed to download attachments. I need this because I don't want to store the attachments in my database to save disk space and bandwidth. They will be downloaded on demand later. I am not sure if I can do that and still be able to use MimeParser from MimeKit
I am planning to do that:
Get message body structure and find the text and html parts
Download text and html body parts using ImapFolder.GetStream and store in database preserving mime tree structure by saving section names along with headers. Attachment will not be downloaded
Later I want to show the message in the UI but I want to delay parsing until the mail message needs to be shown in the UI.
This is my progress so far
var msgSummaries = remoteFolder.Fetch(new int[] { remoteMessage.Index }, MessageSummaryItems.BodyStructure);
var stream = remoteFolder.GetStream(remoteMessage.Index, msgSummaries[0].HtmlBody.PartSpecifier);
//at this point i am saving the stream to the database and later i am trying to convert it to mime entity like that
var parser = new MimeParser(ParserOptions.Default, stream, true);
var mimeEntity = parser.ParseEntity(cancellationToken);
Unfortunatelly the stream doesn't contain mime part headers and cannot be parsed and I don't see an option to request headers inside GetStream method like this
FETCH 1 (BODY.PEEK[2.MIME] BODY.PEEK[2])
Any suggestions?

Well, first of all, have you tried:
var mimeEntity = remoteFolder.GetBodyPart (remoteMessage.Index, msgSummaries[0].HtmlBody);
or, if you really want to use streams:
var headerStream = remoteFolder.GetStream (remoteMessage.Index, msgSummaries[0].HtmlBody.PartSpecifier + ".MIME");
var contentStream = remoteFolder.GetStream (remoteMessage.Index, msgSummaries[0].HtmlBody.PartSpecifier);
var stream = new MemoryStream ();
headerStream.CopyTo (stream);
headerStream.Dispose ();
contentStream.CopyTo (stream);
contentStream.Dispose ();
stream.Position = 0;
//at this point i am saving the stream to the database and later i am trying to convert it to mime entity like that
var parser = new MimeParser(ParserOptions.Default, stream, true);
var mimeEntity = parser.ParseEntity(cancellationToken);

Sending image with byte array convertion, from java to c#

I am trying to send a .jpg file which is on my android device to my server computer.
To do this, I am converting the picture into a byte array by a java android application, and sending it as an argument to my server computer. I`m doing this by a web service call.
The first function is edited:
public static byte[] ImageConvertion(){
File inputFile = new File("/storage/emulated/0/IFSpictures/icon-si_strapclamp.jpg");
byte[] data;
try{
FileInputStream input = new FileInputStream(inputFile);
ByteArrayOutputStream output = new ByteArrayOutputStream ();
byte[] buffer = new byte[65535];
int l;
while ((l = input.read(buffer)) > 0)
output.write (buffer, 0, l);
input.close();
output.close();
data = output.toByteArray();
return data;
} catch (IOException e) {
System.err.println(e);
data=null;
}
return data;
}
My web-service is written in ASP.NET (C#) language, and there is a function that takes the byte array as an argument and converts it back into an image on server computer.
[WebMethod]
public void ByteArrayToPicture(byte[] imageData)
{
using (var ms = new MemoryStream(imageData))
{
Image image = Image.FromStream(ms);
image.Save(#"C:\newImage.jpg");
}
}
However, I couldn`t do it because of the web-service side. I have debugged it that and it seems that the problem is because of the Image.FromStream() function.
I definitely don`t have any problems with passing the arguments. I think either, the language conflict or the conversion image to byte and vice-verse may be leading the problem. Does anyone has any idea or see something wrong?
I muchly appropriate any help.
Thanks.

sorry for my incomplete question, however I want to give some tips whoever is trying to do the same thing.
If anyone is trying to send an image to a server and both side has different platforms, then do not convert the image into byte array!
The reason is, in my case the image which is converted into byte array on Java differs from the byte array on C#. Therefore according to my research it is not possible to gather the image on the server side. The byte array created on Java wont have the right format on C#.
Hence anyone wants data transferring from one language to another, use Base64 encoding. Convert the image into Base64 string on one side and send it as string to the other language. Since Base64 format is same on every language there wont be any problem to reproduce it.
I sold the problem with the following codes:
Bitmap ourbitmap = BitmapFactory.decodeStream(imageStream, null, options);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ourbitmap.compress(Bitmap.CompressFormat.JPEG, 100, baos);
byte[] b = baos.toByteArray();
test = Base64.encodeToString(b, Base64.DEFAULT);
This is the code where I get the image and convert it into Base64 string on Java android application,
byte[] imageBytes = Convert.FromBase64String(Base64ImageData);
MemoryStream ms = new MemoryStream(imageBytes, 0,
imageBytes.Length);
ms.Write(imageBytes, 0, imageBytes.Length);
Image image = Image.FromStream(ms, true);
image.Save(#"D:\tmpImage.jpg");
The code above takes the Base64 type string and converts back into an image. This is written in C#.

With such an incomplete code example and such a vague problem description, it's difficult to know for sure what the problem is.
However, reviewing the code you did post, I see one bug that would be significant if this is really the code you are using. In your Java methodConvertion() method, you have this statement:
data = output.toByteArray();
The problem is that all that does is create a new byte[] object and assign the reference to your local variable named data. That object never leaves the method.
Presumably you have some other code which, after calling methodConvertion(), sends the byte[] object that is referenced by the argument you passed to that method. But that object is just going to be whatever it was before you called the method.
You should instead change your Java code so that it looks like this:
public static byte[] methodConvertion(){
File inputFile = new File("/storage/emulated/0/IFSpictures/icon-si_strapclamp.jpg");
try{
FileInputStream input = new FileInputStream(inputFile);
ByteArrayOutputStream output = new ByteArrayOutputStream ();
byte [] buffer = new byte [65536];
int l;
while ((l = input.read(buffer)) > 0)
output.write (buffer, 0, l);
input.close();
output.close();
return output.toByteArray();
} catch (IOException e) {
System.err.println(e);
return null;
}
}
And then in the caller, you should check the return value and only proceed if the value is not null, reporting the error somehow otherwise.
If that doesn't address your question, you should edit the question so that it has a better code example, and so that you are much more specific about what's wrong. What happens when you run the code, and how is that different from what you expected? Be sure to clearly state any error messages, quoting them exactly, and including any stack traces from exceptions.

Ruby base64 to C# base64

I'm writing a Ruby/Rhomobile application that takes an image, encodes it in base64 and sends to the server(that is running C# on ASP.NET), as part of a bigger XML.
However, once I try to decode the base64 and save it to the hard disk, the resulting file does not work as an image.
Here's the relevant ruby code:
image_element = REXML::Element.new("image")
image_element.text = Base64.encode64(open(Rho::RhoApplication::get_blob_path(self.image_uri)) { |io| io.read })
form_element.add_element(image_element)
And here is my C# code:
var doc = new XmlDocument();
doc.LoadXml(Server.UrlDecode(Request.Form[0]));
var imageBase64 = doc.SelectNodes("//image")[0];
var imageBytes = imageBase64.InnerText;
using(var imgWriter = new FileStream(#"c:\img.jpg",FileMode.Create))
{
imgWriter.Write(imageBytes,0,imageBytes.Length);
}

I would investigate your call to Server.UrlDecode. It seems like that could corrupt your data.
It seems like the "+" sign is of specific concern, per this SO question. Server.UrlDecode uses HttpServerUtility.UrlDecode, and here's the documentation for it.

BlockingSenderDestination.sendReceive() UTF-8 issue

In my Blackberry application I am loading JSON using the following method.
private static Object loadJson(String uriStr){
Object _json = null;
Message response = null;
BlockingSenderDestination bsd = null;
try
{
bsd = (BlockingSenderDestination)
DestinationFactory.getSenderDestination
("CommAPISample", URI.create(uriStr));
if(bsd == null)
{
bsd =
DestinationFactory.createBlockingSenderDestination
(new Context("CommAPISample"),
URI.create(uriStr), new JSONMessageProcessor()
);
}
response = bsd.sendReceive();
_json = response.getObjectPayload();
}
catch(Exception e)
{
System.out.println(e.toString());
}
finally
{
if(bsd != null)
{
bsd.release();
}
}
return _json;
}
This is working fine. But the problem is when I am getting JSON, Arabic characters show as junk
(الرئيس التنÙ) . I submitted this issue to Blackberry support form
Arabic shows corrupted in the JSON output
As per the discussion, I encode the Arabic character into \uxxxx format(In my server side application) and it was working. But now I have to use a JSON from somebody else where I can’t change the server side code.
They are using asp.net C# , as per them they are sending the data like the following.
JsonResult result = new JsonResult();
result.ContentEncoding = System.Text.Encoding.UTF8;
result.JsonRequestBehavior = JsonRequestBehavior.AllowGet;
result.Data = “Data Object (Contains Arabic) comes here”
return result;
So my question is, If the server provide the data in the above manner, BlockingSenderDestination.sendReceive method can get a utf-8 data? Or it is expecting only \uxxxx encoded data for non-ascii. Or I have to do something else (like sending some header to server) so that I can directly use the utf-8 data.
In debug mode I check the value of 'response'. It is already showing junk characters.
Except from JSON I am able to handle Arabic everywhere else.
Yesterday I posted this issue in Blackberry form . But till now no reply.
I am new to blackberry and Java. So I am sorry if this is silly question.
Thanks in advance.

What is the content type in the response? Is the server explicitly defining the UTF-8 character encoding in the HTTP header? e.g.:
Content-Type: text/json; charset=UTF-8
If the API is ignoring the charset in the HTTP content type, an easier way to do the String conversion is by determining whether the Message received is a ByteMessage or a StreamMessage. Retrieve the message as a byte array and then convert to a string using the UTF-8 encoding
i.e.:
Message msg = bsd.sendReceive();
byte[] msgBytes = null;
if (msg instanceof ByteMessage) {
msgBytes = ((ByteMessage) msg).getBytePayload();
}
else { /* StreamMessage */
// TODO read the bytes from the stream into a byte array
}
return new String(msgBytes,"UTF-8");

At last I found the solution myself.
The data sending from server was in UTF-8 which uses double byte to show single character. But BlockingSenderDestination.sendReceive() is not able to identify that. So it is creating one character for each byte. So the solution was to get each character and get the byte from that character and add to a byte array. From that byte array create a string with UTF8 encoding.
If anyone know to use BlockingSenderDestination.sendReceive() for utf-8 please post here. So that we can avoid this extra conversion method.