I am uploading multiple files using the BeginGetRequestStream of HttpWebRequest but I want to update the progress control I have written whilst I post up the data stream.
How should this be done, I have tried calling Dispatch.BeginInvoke (as below) from within the loop that pushes the data into the stream but it locks the browser until its finished so it seems to be in some sort of worker/ui thread deadlock.
This is a code snippet of pretty much what I am doing:
class RequestState
{
public HttpWebRequest request; // holds the request
public FileDialogFileInfo file; // store our file stream data
public RequestState( HttpWebRequest request, FileDialogFileInfo file )
{
this.request = request;
this.file = file;
}
}
private void UploadFile( FileDialogFileInfo file )
{
UriBuilder ub = new UriBuilder( app.receiverURL );
ub.Query = string.Format( "filename={0}", file.Name );
// Open the selected file to read.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create( ub.Uri );
request.Method = "POST";
RequestState state = new RequestState( request, file );
request.BeginGetRequestStream( new AsyncCallback( OnUploadReadCallback ), state );
}
private void OnUploadReadCallback( IAsyncResult asynchronousResult )
{
RequestState state = (RequestState)asynchronousResult.AsyncState;
HttpWebRequest request = (HttpWebRequest)state.request;
Stream postStream = request.EndGetRequestStream( asynchronousResult );
PushData( state.file, postStream );
postStream.Close();
state.request.BeginGetResponse( new AsyncCallback( OnUploadResponseCallback ), state.request );
}
private void PushData( FileDialogFileInfo file, Stream output )
{
byte[] buffer = new byte[ 4096 ];
int bytesRead = 0;
Stream input = file.OpenRead();
while( ( bytesRead = input.Read( buffer, 0, buffer.Length ) ) != 0 )
{
output.Write( buffer, 0, bytesRead );
bytesReadTotal += bytesRead;
App app = App.Current as App;
int totalPercentage = Convert.ToInt32( ( bytesReadTotal / app.totalBytesToUpload ) * 100 );
// enabling the following locks up my UI and browser
Dispatcher.BeginInvoke( () =>
{
this.ProgressBarWithPercentage.Percentage = totalPercentage;
} );
}
}
I was going to say that, I didn't think that Silverlight 2's HttpWebRequest supported streaming, because the request data gets buffered into memory entirely. It had been a while since the last time I looked at it though, therefore I went back to see if Beta 2 supported it. Well turns out it does. I am glad I went back and read before stating that. You can enable it by setting AllowReadStreamBuffering to false. Did you set this property on your HttpWebRequest? That could be causing your block.
MSDN Reference
File upload component for Silverlight and ASP.NET
Edit, found another reference for you. You may want to follow this approach by breaking the file into chunks. This was written last March, therefore I am not sure if it will work in Beta 2 or not.
Thanks for that, I will take a look at those links, I was considering chunking my data anyway, seems to be the only way I can get any reasonable progress reports out of it.
Related
I'm trying to upload a large video (1 GB+) from my xamarin app and it keeps crashing once it reaches about 0.5 GB of my file. The only way I've found to get the videos to post to my WCF service while sending data along with it is using the MultiPart logic but I'm not sure if I'm running out of memory or what because even in debug mode, it simply crashes without any real error message.
I'm trying to run it on a native device (not a sim) and it's a Samsung Galaxy S9 with Android 9.
Here's the upload code that I'm using: (p.s. - as a test, I tried putting the WriteAsync into a for loop thinking that maybe trying to write the whole gig was the problem, but the result was the same. That's why you'll see the MAXFILESIZEPART constant in there which is just an int equal to 10000000.)
private async Task<byte[]> GetMultipartFormDataAsync(Dictionary<string, object> postParameters, string boundary)
{
try
{
using (Stream formDataStream = new System.IO.MemoryStream())
{
bool needsCLRF = false;
foreach (var param in postParameters)
{
// Thanks to feedback from commenters, add a CRLF to allow multiple parameters to be added.
// Skip it on the first parameter, add it to subsequent parameters.
if (needsCLRF)
await formDataStream.WriteAsync(Encoding.UTF8.GetBytes("\r\n"), 0, Encoding.UTF8.GetByteCount("\r\n"));
needsCLRF = true;
if (param.Value is FileParameter)
{
FileParameter fileToUpload = (FileParameter)param.Value;
// Add just the first part of this param, since we will write the file data directly to the Stream
string header = string.Format("--{0}\r\nContent-Disposition: form-data; name=\"{1}\"; filename=\"{2}\"\r\nContent-Type: {3}\r\n\r\n",
boundary,
param.Key,
fileToUpload.FileName ?? param.Key,
fileToUpload.ContentType ?? "application/octet-stream");
await formDataStream.WriteAsync(Encoding.UTF8.GetBytes(header), 0, Encoding.UTF8.GetByteCount(header));
// Write the file data directly to the Stream, rather than serializing it to a string.
if (fileToUpload.File.Length > MAXFILESIZEPART)
{
for (var i = 0; i < fileToUpload.File.Length; i += MAXFILESIZEPART)
{
var len = i + MAXFILESIZEPART > fileToUpload.File.Length
? fileToUpload.File.Length - i
: MAXFILESIZEPART;
await formDataStream.WriteAsync(fileToUpload.File, i, len);
}
}
else
{
await formDataStream.WriteAsync(fileToUpload.File, 0, fileToUpload.File.Length);
}
}
else
{
string postData = string.Format("--{0}\r\nContent-Disposition: form-data; name=\"{1}\"\r\n\r\n{2}",
boundary,
param.Key,
param.Value);
await formDataStream.WriteAsync(Encoding.UTF8.GetBytes(postData), 0, Encoding.UTF8.GetByteCount(postData));
}
}
// Add the end of the request. Start with a newline
string footer = "\r\n--" + boundary + "--\r\n";
await formDataStream.WriteAsync(Encoding.UTF8.GetBytes(footer), 0, Encoding.UTF8.GetByteCount(footer));
// Dump the Stream into a byte[]
formDataStream.Position = 0;
byte[] formData = new byte[formDataStream.Length];
formDataStream.Read(formData, 0, formData.Length);
return formData;
}
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
}
And it's eventually failing on the following line
await formDataStream.WriteAsync(fileToUpload.File, i, len);
but only after a certain point (about 500MB) so I'm assuming it's a memory issue but it doesn't say so. Is there a better way to accomplish this task? I'm doing it so that it also records the progress as the upload happens. I'm trying to accomplish something similar to uploading large videos via the facebook app so that it will upload in the background while you continue working. It works great when working with smaller files (i.e. - < 500 MB) but this is the first time I've tried a file that was almost a gig in size.
NOTE: This happens BEFORE it starts posting anything to the server so it's not IIS or WCF related. This code crashes just writing the bytes to the memory stream.
Any suggestions?
Thanks!
According to your description, the service will stop at a certain time point, and because the file you transfer is about 1G, it is likely to be sendtimeout.No transfer completed within the specified time, causing exception。The SendTimeout that specifies how long the write operation has to complete before timing out. The default value is 1 minute.
I set sendtimeout to 15 seconds in my configuration file.If the data takes more than 15 seconds, an exception will occur. You can set it to a higher value to avoid timeout and exception.
For information about sendtimeout, please refer to the following link:
https://learn.microsoft.com/en-us/dotnet/api/system.servicemodel.channels.binding.sendtimeout?view=dotnet-plat-ext-3.1
UPDATE
I think it might be a memory overflow problem.Large file may cause memory overflow, unable to read at the same time.
You can refer to the following links for solutions
https://learn.microsoft.com/en-us/archive/blogs/johan/are-you-getting-outofmemoryexceptions-when-uploading-large-files
I get an Out of Memory Exception when using Http.Put of a large file. I am using an asynchronous model as shown in the code. I am trying to send 8K blocks of data to a Windows 2008 R2 server. The exception consistently occurs when I attempt to write a block of data that exceeds 536,868,864 bytes. The exception occurs on the requestStream.Write method in the code snippet below.
Looking for reasons why?
Note: Smaller files are PUT OK. Logic also works if I write to a local FileStream. Running VS 2010, .Net 4.0 on Win 7 Ultimate client computer.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("Http://website/FileServer/filename");
request.Method = WebRequestMethods.Http.Put;
request.SendChunked = true;
request.AllowWriteStreamBuffering = true;
...
request.BeginGetRequestStream( new AsyncCallback(EndGetStreamCallback), state);
...
int chunk = 8192; // other values give same result
....
private static void EndGetStreamCallback(IAsyncResult ar) {
long limit = 0;
long fileLength;
HttpState state = (HttpState)ar.AsyncState;
Stream requestStream = null;
// End the asynchronous call to get the request stream.
try {
requestStream = state.Request.EndGetRequestStream(ar);
// Copy the file contents to the request stream.
FileStream stream = new FileStream(state.FileName, FileMode.Open, FileAccess.Read, FileShare.None, chunk, FileOptions.SequentialScan);
BinaryReader binReader = new BinaryReader(stream);
fileLength = stream.Length;
// Set Position to the beginning of the stream.
binReader.BaseStream.Position = 0;
byte[] fileContents = new byte[chunk];
// Read File from Buffer
while (limit < fileLength)
{
fileContents = binReader.ReadBytes(chunk);
// the next 2 lines attempt to write to network and server
requestStream.Write(fileContents, 0, chunk); // causes Out of memory after 536,868,864 bytes
requestStream.Flush(); // I get same result with or without Flush
limit += chunk;
}
// IMPORTANT: Close the request stream before sending the request.
stream.Close();
requestStream.Close();
}
}
You apparently have this documented problem. When AllowWriteStreamBuffering is true, it buffers all the data written to the request! So, the "solution" is to set that property to false:
To work around this issue, set the HttpWebRequest.AllowWriteStreamBuffering property to false.
I have a big problem dealing with data I try to download in my Application over the internet via HttpWebResponse. My code looks like that:
myWebRequest.Timeout = 10000;
using (HttpWebResponse myWebResponse = (HttpWebResponse)myWebRequest.GetResponse())
{
using (Stream ReceiveStream = myWebResponse.GetResponseStream())
{
Encoding encode = Encoding.GetEncoding("utf-8");
StreamReader readStream = new StreamReader(ReceiveStream, encode);
// Read 1024 characters at a time.
Char[] read = new Char[1024];
int count = readStream.Read(read, 0, 1024);
int break_counter = 0;
while (count > 0 && break_counter < 10000)
{
String str = new String(read, 0, count);
buffer += str;
count = readStream.Read(read, 0, 1024);
break_counter++;
}
}
}
This code runs in a few instances in separated threads so it's a little bit hard to debug. The problem is this method got stuck and I blame it on the poor connection to the data.
As you can see I already set a timeout and was hoping the code would just terminate after the timeout time has expired. It does not! At least not all the time. Sometimes I get a WebException/Timeout but a few times it just got stuck.
What is a timeout exactly? When is it called?
Lets say the HttpWebResponse starts to receive data but it got stuck somewhere in the middle of transmission. Do I get a timeout? For me it looks like I don't because my application got stuck too and no timeout exception is raised.
What can I do to patch this up or how can I get further information about what is going wrong here?
Try setting HttpWebRequest.ReadWriteTimeout Property
The number of milliseconds before the
writing or reading times out. The
default value is 300,000 milliseconds
(5 minutes).
Edit: This is done in the Compact Framework, I don't have access to WebClient therefore it has to be done with HttpWebRequests.
I am creating a download manager application that will be able to have concurrent downloads (more than one download at once) and the ability to report the percentage completed and resume the downloads.
This means that I am downloading some bytes into a buffer and then writing the buffer to disk. I just wanted to check what recommended algorithm/procedure is for this.
This is what I have thus far for the main download method:
private void StartDownload()
{
HttpWebRequest webReq = null;
HttpWebResponse webRes = null;
Stream fileBytes = null;
FileStream saveStream = null;
try
{
webReq = (HttpWebRequest)HttpWebRequest.Create(_url);
webReq.Headers.Add(HttpRequestHeader.Cookie, "somedata");
webRes = (HttpWebResponse)webReq.GetResponse();
byte[] buffer = new byte[4096];
long bytesRead = 0;
long contentLength = webRes.ContentLength;
if (File.Exists(_filePath))
{
bytesRead = new FileInfo(_filePath).Length;
}
fileBytes = webRes.GetResponseStream();
fileBytes.Seek(bytesRead, SeekOrigin.Begin);
saveStream = new FileStream(_filePath, FileMode.Append, FileAccess.Write);
while (bytesRead < contentLength)
{
int read = fileBytes.Read(buffer, 0, 4096);
saveStream.Write(buffer, 0, read);
bytesRead += read;
}
//set download status to complete
//_parent
}
catch
{
if (Thread.CurrentThread.ThreadState != ThreadState.AbortRequested)
{
//Set status to error.
}
}
finally
{
saveStream.Close();
fileBytes.Close();
webRes.Close();
saveStream.Dispose();
fileBytes.Dispose();
saveStream = null;
fileBytes = null;
webRes = null;
webReq = null;
}
}
Should I be downloading a larger buffer? Should I be writing the buffer to file so often (every 4KB?) Should there be some thread sleeping in there to ensure not all the CPU is used? I think reporting the progress change every 4KB is stupid so I was planning to do it every 64KB downloaded.
Looking for some general tips or anything that is wrong with my code so far.
First thing, I would get rid of the finally clause and change the code to use "USING" clauses.
Anything that implements IDisposable should be programmed that way to make sure garbage collection occurs correctly and when it is supposed to.
For example:
using (HttpWebRequest webReq = (HttpWebRequest)HttpWebRequest.Create(_url)) {
/* more code here... */
}
Second, I wouldn't instantiate my variables at the head with null values (ala Pascal style). See above example.
Third, the download should be in it's own thread which sync's with a call back function in the main thread to report status. Put the sync call in the middle of your while loop.
In the full framework, the simplest way to do this is to use the WebClient class's DownloadFile method, like this:
using(var wc = new WebClient()) {
wc.DownloadFile(url, filePath);
}
EDIT: To report the download progress, call DownloadFileAsync and listen for the DownloadProgressChanged event. You can also cancel the download by calling the CancelAsync method.
From a user experience perspective, you should be able to answer a lot of these questions by looking at an application like Internet Explorer or Firefox. For example;
In Internet Explorer, new data is reported every few kilobytes, up to the one megabyte mark. After that, it is reported in 100 kilobyte increments.
How often you write to the buffer depends on whether you're allowing recovery when the connection is dropped. If you're like IE and force the user to start from scratch, it doesn't really matter how often you save your buffer as long as you do it eventually. Set your saving based on "acceptable loss".
Your application should obviously not take 100% of the CPU, since that isn't good etiquette in the programming world. Have your threads at least sleep long enough not to dominate the CPU.
Your code, generally speaking, is functional, though it could stand a lot of refactoring to make it a little cleaner/easier to read. You might consider using the WebClient class in .NET, too, but if this is a learning exercise, you're doing it the right way.
Good luck! You're on the right track.
You can get all the tips you need from existing code thats been published and is freely available.
Such as here: http://www.codeproject.com/KB/IP/MyDownloader.aspx
Better yet, take the existing code and modify it for your needs. (That's why the code gets posted there in the first place.)
If you need to track the progress, use WebClient.DownloadFileAsync along with the DownloadProgressChanged and DownloadFileCompleted events
WebClient wc = new WebClient();
wc.DownloadProgressChanged += wc_DownloadProgressChanged;
wc.DownloadFileCompleted += wc_DownloadFileCompleted;
wc.DownloadFileAsync(sourceUri, localPath);
...
private void wc_DownloadProgressChanged(object sender, DownloadProgressChangedEventArgs e)
{
...
}
private void wc_DownloadFileCompleted(object sender, AsyncCompletedEventArgs e)
{
...
}
I need to download a large file (2 GB) over HTTP in a C# console application. Problem is, after about 1.2 GB, the application runs out of memory.
Here's the code I'm using:
WebClient request = new WebClient();
request.Credentials = new NetworkCredential(username, password);
byte[] fileData = request.DownloadData(baseURL + fName);
As you can see... I'm reading the file directly into memory. I'm pretty sure I could solve this if I were to read the data back from HTTP in chunks and write it to a file on disk.
How could I do this?
If you use WebClient.DownloadFile you could save it directly into a file.
The WebClient class is the one for simplified scenarios. Once you get past simple scenarios (and you have), you'll have to fall back a bit and use WebRequest.
With WebRequest, you'll have access to the response stream, and you'll be able to loop over it, reading a bit and writing a bit, until you're done.
From the Microsoft documentation:
We don't recommend that you use WebRequest or its derived classes for
new development. Instead, use the System.Net.Http.HttpClient class.
Source: learn.microsoft.com/WebRequest
Example:
public void MyDownloadFile(Uri url, string outputFilePath)
{
const int BUFFER_SIZE = 16 * 1024;
using (var outputFileStream = File.Create(outputFilePath, BUFFER_SIZE))
{
var req = WebRequest.Create(url);
using (var response = req.GetResponse())
{
using (var responseStream = response.GetResponseStream())
{
var buffer = new byte[BUFFER_SIZE];
int bytesRead;
do
{
bytesRead = responseStream.Read(buffer, 0, BUFFER_SIZE);
outputFileStream.Write(buffer, 0, bytesRead);
} while (bytesRead > 0);
}
}
}
}
Note that if WebClient.DownloadFile works, then I'd call it the best solution. I wrote the above before the "DownloadFile" answer was posted. I also wrote it way too early in the morning, so a grain of salt (and testing) may be required.
You need to get the response stream and then read in blocks, writing each block to a file to allow memory to be reused.
As you have written it, the whole response, all 2GB, needs to be in memory. Even on a 64bit system that will hit the 2GB limit for a single .NET object.
Update: easier option. Get WebClient to do the work for you: with its DownloadFile method which will put the data directly into a file.
WebClient.OpenRead returns a Stream, just use Read to loop over the contents, so the data is not buffered in memory but can be written in blocks to a file.
i would use something like this
The connection can be interrupted, so it is better to download the file in small chunks.
Akka streams can help download file in small chunks from a System.IO.Stream using multithreading. https://getakka.net/articles/intro/what-is-akka.html
The Download method will append the bytes to the file starting with long fileStart. If the file does not exist, fileStart value must be 0.
using Akka.Actor;
using Akka.IO;
using Akka.Streams;
using Akka.Streams.Dsl;
using Akka.Streams.IO;
private static Sink<ByteString, Task<IOResult>> FileSink(string filename)
{
return Flow.Create<ByteString>()
.ToMaterialized(FileIO.ToFile(new FileInfo(filename), FileMode.Append), Keep.Right);
}
private async Task Download(string path, Uri uri, long fileStart)
{
using (var system = ActorSystem.Create("system"))
using (var materializer = system.Materializer())
{
HttpWebRequest request = WebRequest.Create(uri) as HttpWebRequest;
request.AddRange(fileStart);
using (WebResponse response = request.GetResponse())
{
Stream stream = response.GetResponseStream();
await StreamConverters.FromInputStream(() => stream, chunkSize: 1024)
.RunWith(FileSink(path), materializer);
}
}
}