I have a big problem dealing with data I try to download in my Application over the internet via HttpWebResponse. My code looks like that:
myWebRequest.Timeout = 10000;
using (HttpWebResponse myWebResponse = (HttpWebResponse)myWebRequest.GetResponse())
{
using (Stream ReceiveStream = myWebResponse.GetResponseStream())
{
Encoding encode = Encoding.GetEncoding("utf-8");
StreamReader readStream = new StreamReader(ReceiveStream, encode);
// Read 1024 characters at a time.
Char[] read = new Char[1024];
int count = readStream.Read(read, 0, 1024);
int break_counter = 0;
while (count > 0 && break_counter < 10000)
{
String str = new String(read, 0, count);
buffer += str;
count = readStream.Read(read, 0, 1024);
break_counter++;
}
}
}
This code runs in a few instances in separated threads so it's a little bit hard to debug. The problem is this method got stuck and I blame it on the poor connection to the data.
As you can see I already set a timeout and was hoping the code would just terminate after the timeout time has expired. It does not! At least not all the time. Sometimes I get a WebException/Timeout but a few times it just got stuck.
What is a timeout exactly? When is it called?
Lets say the HttpWebResponse starts to receive data but it got stuck somewhere in the middle of transmission. Do I get a timeout? For me it looks like I don't because my application got stuck too and no timeout exception is raised.
What can I do to patch this up or how can I get further information about what is going wrong here?
Try setting HttpWebRequest.ReadWriteTimeout Property
The number of milliseconds before the
writing or reading times out. The
default value is 300,000 milliseconds
(5 minutes).
Related
I'm trying to upload a large video (1 GB+) from my xamarin app and it keeps crashing once it reaches about 0.5 GB of my file. The only way I've found to get the videos to post to my WCF service while sending data along with it is using the MultiPart logic but I'm not sure if I'm running out of memory or what because even in debug mode, it simply crashes without any real error message.
I'm trying to run it on a native device (not a sim) and it's a Samsung Galaxy S9 with Android 9.
Here's the upload code that I'm using: (p.s. - as a test, I tried putting the WriteAsync into a for loop thinking that maybe trying to write the whole gig was the problem, but the result was the same. That's why you'll see the MAXFILESIZEPART constant in there which is just an int equal to 10000000.)
private async Task<byte[]> GetMultipartFormDataAsync(Dictionary<string, object> postParameters, string boundary)
{
try
{
using (Stream formDataStream = new System.IO.MemoryStream())
{
bool needsCLRF = false;
foreach (var param in postParameters)
{
// Thanks to feedback from commenters, add a CRLF to allow multiple parameters to be added.
// Skip it on the first parameter, add it to subsequent parameters.
if (needsCLRF)
await formDataStream.WriteAsync(Encoding.UTF8.GetBytes("\r\n"), 0, Encoding.UTF8.GetByteCount("\r\n"));
needsCLRF = true;
if (param.Value is FileParameter)
{
FileParameter fileToUpload = (FileParameter)param.Value;
// Add just the first part of this param, since we will write the file data directly to the Stream
string header = string.Format("--{0}\r\nContent-Disposition: form-data; name=\"{1}\"; filename=\"{2}\"\r\nContent-Type: {3}\r\n\r\n",
boundary,
param.Key,
fileToUpload.FileName ?? param.Key,
fileToUpload.ContentType ?? "application/octet-stream");
await formDataStream.WriteAsync(Encoding.UTF8.GetBytes(header), 0, Encoding.UTF8.GetByteCount(header));
// Write the file data directly to the Stream, rather than serializing it to a string.
if (fileToUpload.File.Length > MAXFILESIZEPART)
{
for (var i = 0; i < fileToUpload.File.Length; i += MAXFILESIZEPART)
{
var len = i + MAXFILESIZEPART > fileToUpload.File.Length
? fileToUpload.File.Length - i
: MAXFILESIZEPART;
await formDataStream.WriteAsync(fileToUpload.File, i, len);
}
}
else
{
await formDataStream.WriteAsync(fileToUpload.File, 0, fileToUpload.File.Length);
}
}
else
{
string postData = string.Format("--{0}\r\nContent-Disposition: form-data; name=\"{1}\"\r\n\r\n{2}",
boundary,
param.Key,
param.Value);
await formDataStream.WriteAsync(Encoding.UTF8.GetBytes(postData), 0, Encoding.UTF8.GetByteCount(postData));
}
}
// Add the end of the request. Start with a newline
string footer = "\r\n--" + boundary + "--\r\n";
await formDataStream.WriteAsync(Encoding.UTF8.GetBytes(footer), 0, Encoding.UTF8.GetByteCount(footer));
// Dump the Stream into a byte[]
formDataStream.Position = 0;
byte[] formData = new byte[formDataStream.Length];
formDataStream.Read(formData, 0, formData.Length);
return formData;
}
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
}
And it's eventually failing on the following line
await formDataStream.WriteAsync(fileToUpload.File, i, len);
but only after a certain point (about 500MB) so I'm assuming it's a memory issue but it doesn't say so. Is there a better way to accomplish this task? I'm doing it so that it also records the progress as the upload happens. I'm trying to accomplish something similar to uploading large videos via the facebook app so that it will upload in the background while you continue working. It works great when working with smaller files (i.e. - < 500 MB) but this is the first time I've tried a file that was almost a gig in size.
NOTE: This happens BEFORE it starts posting anything to the server so it's not IIS or WCF related. This code crashes just writing the bytes to the memory stream.
Any suggestions?
Thanks!
According to your description, the service will stop at a certain time point, and because the file you transfer is about 1G, it is likely to be sendtimeout.No transfer completed within the specified time, causing exception。The SendTimeout that specifies how long the write operation has to complete before timing out. The default value is 1 minute.
I set sendtimeout to 15 seconds in my configuration file.If the data takes more than 15 seconds, an exception will occur. You can set it to a higher value to avoid timeout and exception.
For information about sendtimeout, please refer to the following link:
https://learn.microsoft.com/en-us/dotnet/api/system.servicemodel.channels.binding.sendtimeout?view=dotnet-plat-ext-3.1
UPDATE
I think it might be a memory overflow problem.Large file may cause memory overflow, unable to read at the same time.
You can refer to the following links for solutions
https://learn.microsoft.com/en-us/archive/blogs/johan/are-you-getting-outofmemoryexceptions-when-uploading-large-files
A client has a page which when called starts a long running process and intermittently spits out its progress as it goes
In the format
[dd/MM/yyyy hh:mm:ss] - Process Started
[dd/MM/yyyy hh:mm:ss] - CSV Imported
[dd/MM/yyyy hh:mm:ss] - Process 10% complete
Then 30 seconds later it might write out :
[dd/MM/yyyy hh:mm:ss] - User x Created
[dd/MM/yyyy hh:mm:ss] - User y Created
[dd/MM/yyyy hh:mm:ss] - Process 20% complete
etc... it takes 10-20 minutes to run, we dont have access to the code for this page.
What I have been asked to do is to call this page from one of the other applications, consume the output and give a realtime update on our dashboard.
My first thought was was to use an http client call .GetStreamAsync() and have a loop reading the stream intermittently and reporting back on the last thing that was written out.
This was my first attempt :
using (HttpClient httpClient = new HttpClient())
{
httpClient.Timeout = TimeSpan.FromMilliseconds(Timeout.Infinite);
var requestUri = "http://localhost:64501/Page1.aspx";
var stream = httpClient.GetStreamAsync(requestUri).Result;
using (var reader = new StreamReader(stream))
{
while (!reader.EndOfStream)
{
var currentLine = reader.ReadLine();
Thread.Sleep(1000);
}
}
}
However
var currentLine = reader.ReadLine();
Appears to block and wait for the response to complete before returning anything..
I need to be able to read the stream as it comes in.. Is this possible?
The problem lies in ReadLine, the server may be not sending lines (something logic as it seems to be prepared to be sent to a web page where newlines are ignored) so you need to read chuncks of data and convert those to strings:
using (HttpClient httpClient = new HttpClient())
{
httpClient.Timeout = TimeSpan.FromMilliseconds(Timeout.Infinite);
var requestUri = "http://localhost:64501/Page1.aspx";
var stream = httpClient.GetStreamAsync(requestUri).Result;
string read = "";
byte[] buffer = new byte[1024];
while(!stream.EndOfStream)
{
int readed = stream.Read(buffer, 0, 1024);
read += Encoding.UTF8.GetString(buffer, 0, readed);
//Do whatever you need to do with the string.
}
}
i have a simplistic file server\client application ive written in c#. but i commonly run into the problem that my stream writes two different reads into a single buffer. i have a synchronized stream, still isnt helping. any suggestions? thanks!
System.Threading.Thread.Sleep(25);
receive_fspos = new byte[30];
int bytesread = stream_1.Read(receive_fspos, 0, receive_fspos.Length);//this is where it gets combined
if (bytesread == 0)
{
finished = true;
System.Threading.Thread.Sleep(25);
}
string string_1 = utf.GetString(receive_fspos).TrimEnd(new char[] { (char)0 });
int fsposition = (int)Convert.ToInt64(string_1);
bytestosend = fsposition;
filestream.Position = fsposition;
byte[] buffer_1 = new byte[bufsize];
int bytesreadfromfs = filestream.Read(buffer_1, 0, buffer_1.Length);
stream_1.Write(buffer_1, 0, buffer_1.Length);
Console.Write("\rSent " + fsposition + " / " + length + " bytes");
finished = true;
I would not recommend writing your own stream method if you do not fully understand it.
The problem that you are having is because the incoming data is a stream of bytes that does not give you a way of knowing how many bytes in length that the message is.
In the code below you are stating that you would like to read "receive_fspos.Length" bytes of the stream. Since "receive_fspos.Length" is 30, the amount of bytes that will be read will be anywhere from 0 to 30.
If there is only 15 bytes that have been received by the connection. It will give you 15 bytes. If the message was 20 bytes long. Then the message is now split up into different segments.
If the first message was 4 bytes and the second message is 12 bytes. Now you have 2 messages and a set of 16 blank bytes at the end. Even worse those 16 "blank" bytes could be the beginning of a third message coming in to the stream.
If the message is 50 bytes long. Then you will only receive half of the message. Now you would need to add the bytes that were read to a seperate buffer. Read from the stream again. Then repeat this until you have determined that you have read the exact amount of bytes that are needed to complete the entire message. Then concat all of the read bytes back to a single byte[].
receive_fspos = new byte[30];
int bytesread = stream_1.Read(receive_fspos, 0, receive_fspos.Length);//this is where it gets combined
Instead of rolling your own loop please use the BCL methods. It sounded like you are using strings so this would be the preferred method.. I would suggest the following.
using(NetworkStream networkStream = tcpClient.GetStream())
using(StreamReader streamReader = new StreamReader(networkStream))
using(StreamWriter streamWriter = new StreamWriter(networkStream))
{
networkStream.ReadTimeout = timeout; //Set a timeout to stop the stream from reading indefinately
//To receive a string
string incomingString = stream.ReadLine();
//To send a string
stream.WriteLine(messageToSend);
stream.Flush();
}
Your answer clarified that you are trying to send a file. For this I would recommend sending an array of bytes[]. Using this method you can send anything that can be serialized. This includes a file. Please note that the size of the file is limited since it must be kept in memory. To write a larger file you would want to save the data in chunks as it is being streamed in.
//Please note that if the file size is large enough. It may be preferred to use a stream instead of holding the entire file in memory.
byte[] fileAsBytes = File.ReadAllBytes(fileName);
using(NetworkStream networkStream = tcpClient.GetStream())
using(BinaryReader binaryReader = new BinaryReader(networkStream))
using(BinaryWriter binaryWriter = new BinaryWriter(networkStream))
{
networkStream.ReadTimeout = timeout; //Set a timeout to stop the stream from reading indefinately
//To receive a byte array
int incomingBytesLength = BinaryReader.ReadInt32(); //The header is 4 bytes that lets us know how large the incoming byte[] is.
byte[] incomingBytes = BinaryReader.ReadBytes(incomingBytesLength);
//To send a byte array
BinaryWriter.Write(fileAsBytes.Length); //Send a header of 4 bytes that lets the listener know how large the incoming byte[] is.
BinaryWriter.Write(fileAsBytes);
}
got it working, code > 30000 chars :\
it is a little messy but hey, it's functional.
server : https://www.dropbox.com/s/2wyccxpjbja10z3/Program.cs?m
client : https://www.dropbox.com/s/yp78nx4ubacsz6f/Program.cs?m
I've been learning C# by creating an app and i've hit a snag i'm really struggling with.
Basicly i have the code below which is what im using to read from a network stream I have setup. It works but as its only reading 1 packet for each time the sslstream.Read() unblocks. It's causes a big backlog of messages.
What im looking at trying to do is if the part of the stream read contains multiple packets read them all.
I've tried multiple times to work it out but i just ended up in a big mess of code.
If anyone could help out I'd appreciate it!
(the first 4bytes of each packet is the size of the packet.. packets range between 8 bytes and 28,000 bytes)
SslStream _sslStream = (SslStream)_sslconnection;
int bytes = -1;
int nextread = 0;
int byteslefttoread = -1;
byte[] tmpMessage;
byte[] buffer = new byte[3000000];
do
{
bytes = _sslStream.Read(buffer, nextread, 8192);
int packetSize = BitConverter.ToInt32(buffer, 0);
nextread += bytes;
byteslefttoread = packetSize - nextread;
if (byteslefttoread <= 0)
{
int leftover = Math.Abs(byteslefttoread);
do
{
tmpMessage = new byte[packetSize];
Buffer.BlockCopy(buffer, 0, tmpMessage, 0, packetSize);
PacketHolder tmpPacketHolder = new PacketHolder(tmpMessage, "in");
lock (StaticMessageBuffers.MsglockerIn)
{
//puts message into the message queue.. not very oop... :S
MessageInQueue.Enqueue(tmpPacketHolder);
}
}
while (leftover > 0);
Buffer.BlockCopy(buffer, packetSize , buffer, 0, leftover);
byteslefttoread = 0;
nextread = leftover;
}
} while (bytes != 0);
If you are using .Net 3.5 or later I would highly suggest you look into Windows Communication Foundation (wcf). It will simply anything you are trying to do over a network.
On the other hand, if you are doing this purely for educational purposes.
Take a look at this link. Your best bet is to read from the stream in somewhat smaller increments, and feed that data into another stream. Once you can identify the length of data you need for a message, you can cut the second stream off into a message. You can setup an outside loop where available bytes are being checked and wait until its value is > 0 to start the next message. Also should note, that any network code should be running on its own thread, so as to not block the UI thread.
I get an Out of Memory Exception when using Http.Put of a large file. I am using an asynchronous model as shown in the code. I am trying to send 8K blocks of data to a Windows 2008 R2 server. The exception consistently occurs when I attempt to write a block of data that exceeds 536,868,864 bytes. The exception occurs on the requestStream.Write method in the code snippet below.
Looking for reasons why?
Note: Smaller files are PUT OK. Logic also works if I write to a local FileStream. Running VS 2010, .Net 4.0 on Win 7 Ultimate client computer.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("Http://website/FileServer/filename");
request.Method = WebRequestMethods.Http.Put;
request.SendChunked = true;
request.AllowWriteStreamBuffering = true;
...
request.BeginGetRequestStream( new AsyncCallback(EndGetStreamCallback), state);
...
int chunk = 8192; // other values give same result
....
private static void EndGetStreamCallback(IAsyncResult ar) {
long limit = 0;
long fileLength;
HttpState state = (HttpState)ar.AsyncState;
Stream requestStream = null;
// End the asynchronous call to get the request stream.
try {
requestStream = state.Request.EndGetRequestStream(ar);
// Copy the file contents to the request stream.
FileStream stream = new FileStream(state.FileName, FileMode.Open, FileAccess.Read, FileShare.None, chunk, FileOptions.SequentialScan);
BinaryReader binReader = new BinaryReader(stream);
fileLength = stream.Length;
// Set Position to the beginning of the stream.
binReader.BaseStream.Position = 0;
byte[] fileContents = new byte[chunk];
// Read File from Buffer
while (limit < fileLength)
{
fileContents = binReader.ReadBytes(chunk);
// the next 2 lines attempt to write to network and server
requestStream.Write(fileContents, 0, chunk); // causes Out of memory after 536,868,864 bytes
requestStream.Flush(); // I get same result with or without Flush
limit += chunk;
}
// IMPORTANT: Close the request stream before sending the request.
stream.Close();
requestStream.Close();
}
}
You apparently have this documented problem. When AllowWriteStreamBuffering is true, it buffers all the data written to the request! So, the "solution" is to set that property to false:
To work around this issue, set the HttpWebRequest.AllowWriteStreamBuffering property to false.