I am using _webClient.OpenReadAsync(myURI) to download files, it works fine to download files. I want to download the files in sequence starting from 0-20. the 1st file should be downloaded, then the 2nd and so on.
I am using below to download, but it's not what I am expecting.
foreach (string s in files)
_webClient.OpenReadAsync(new Uri(string.Format("{0}{1}", selectedReciter.DownloadURL, s)));
The for loop should only continue to 2nd, 3rd and so on, if 1st file is downloaded, then 2nd, then 3rd and so on.
You are opening URL for reading asynchronously, that word has heavy meaning. What will happen is that the function won't complete when the file has began reading, but rather it will return much sooner.
What you need to do there is to await the result, something like this:
async Task DownloadAll(List<string> addresses)
{
var wc = new WebClient();
foreach(var address in addresses)
await wc.OpenReadTaskAsync(address);
}
Don't forget to add the NuGet package: Microsoft.Bcl.Async first.
Use Background file transfer to download files. Background file transfer allows you to download files also in background means when your application are deactivate or in background.
Here is more about Background file transfer. And Here is an example how to use background file transfer.
You could use the non-async version of the same method that blocks execution until the OpenRead stream is complete.
Related
I'm looking for an archive file type that I can use for processing large archive files in AWS lambda. The entries in the archive are not so large by themselves, the largest maybe 100mb, but there could be a lot of them. My strategy is to create a lambda for processing each entry, where the parameters to my the lambda are a path to the file in s3, as well as a byte range for the entry inside the archive. This would allow for processing each entry without needing to load the entire file. I can write a format to handle this, but I figure something like this probably already exists.
Not required, but hoping to work with these files in C#.
As long as your files are not that big, I can suggest the following approach.
Function invoked
If there is a file in /tmp GoTo Step 4.
If there is no file in /tmp download a new file from S3.
Pop data from the file in chunks making sure that the remaining file shrinks while you process it.
Process the popped chunks of data.
If the function is about to timeout, stop processing file and invoke yourself again (call sibling). It may spawn in the same container or in a different one and will either start processing another file (remaining from some other run) or continue the same one.
When file is completely processed - mark it in some way (Tag) in S3.
There are some limitations here:
- You should not care about the order of processing the files and the rows inside files.
- Occasional multiple processing of same chunks of data should not cause any problem.
- You probably want to keep track of processed files also somewhere externally
A pretty similar approach is used in the Scheduler class of the sosw package. This is a Python package not C#, but idea could help you.
I have some large zip files that I'm downloading and then unzipping in my program. Performance is important, and one direction I started thinking about was whether it was possible to start the download and then begin unzipping the data as it arrives, instead of waiting for the download to complete and then start unzipping. Is this possible? From what I understand of DEFLATE, it should be theoretically possible right?
I'm currently using DotNetZip as my zip library, but it refuses to act on a non-seekable stream.
Code would be something like this:
// HTTP Get the application from the server
var request = (System.Net.HttpWebRequest)System.Net.WebRequest.Create(url);
request.Method = "GET";
Directory.CreateDirectory(localPath);
using (var response = (HttpWebResponse)request.GetResponse())
using (Stream input = response.GetResponseStream())
{
// Unzip being some function which will start unzipping and
// return when unzipping is done
return Unzip(input, localPath);
}
I started thinking about was whether it was possible to start the download and then begin unzipping the data as it arrives, instead of waiting for the download to complete and then start unzipping. Is this possible?
If you want to start unzipping whilst the response body is still downloading, you can't really do this.
In a ZIP file, the Central Directory Record, which contains the list of files in the ZIP file, is located at the very end of the ZIP file. It will be the last thing you download. Without it, you can't reliably determine where the individual file records are located in your ZIP file.
This would also explain why DotNetZip needs a seekable stream. It needs to be able to read the Central Directory Record at the end of the file first, then jump back to earlier sections to read information about individual ZIP entries to extract them.
If you have very specific ZIP files you could make certain assumptions about the layout of those individual file records and extract them by hand, without seeking backwards, but it would not be broadly compatible with ZIP files in general.
You could use a async Task to unzip
await Task.Run(() => ZipFile.ExtractToDirectory(localPath + #"\" + fileName, destinationPath));
If you want to unpack the vast majority of zipfiles, they contain only file records followed by compressed data, repeated until you hit the central directory. So it is very much possible to do streaming decompression like asked in this question. The fflate JavaScript library does it for example.
It is possible to create a) a self executing zipfile, or b) some other weird ass zipfile that isn't formatted like this, but you'd be hard pressed to find one in the wild.
I have a mix python-C# code that scans list of directories and manipulate it files in a loop.
Sometime there is a download directly to the income directory and the program start manipulating the file before the download completed.
Is there any way to detect if the file finish downloading?
A simple way to detect if the file is done downloading is to compare file size. If you always keep a previous "snapshot" of the files in the current directory you will be able to see which files exist and which don't at a given moment in time. Once you see an new file you know that the file has started to download. From this point you can compare the file size of that file and once the previous file size is equal to the current file size you know the file has finished downloading. Each time you would take a new "snapshot" it would be, for example 1ms after the previous. This may not be simple to implement depending on your knowledge of python or C# but I think this algorithm would get you what you want.
When you download, you get file size. You can check file size before writing to file. If file size is same download size then allow writing.
Camera will capture image and store in a folder(.bmp file).Simultaneously our application needs to get one by one image file from the folder for processing.(captured images will update the folder dynamically.If we use FileSystemWatcher , it may cause problem for us.(We are selecting the directory before processing of the image,so updated images will not process.Is there any solution for this problem)how to get one by one image from the folder at run time?
One solution would be to use a FileSystemWatcher to get notified when a new file is put into the folder. Then, process the file(s).
To get the file names of all BMP files in a folder you can use:
string[] fileNames = Directory.GetFiles("<path>", "*.bmp");
Then you can load the bitmaps normally.
You can use FileSystemWatcher but you need to be aware of a few quirks. The watcher will raise a Created event when a file is first created and one or more Modified events as data is written to the files. There is no Close event so you can't know when the camera has stopped writing to the file.
As long as the file is open for writing, you won't be able to read it for processing and any attempt to read it will raise an exception
There are several ways you can handle this:
You can log all events in a list and periodically process all files in the list. If you fail to open a file, just leave it in the list to process in the next processing round.
You can delay reading the file hoping that the camera will finish writing by the time you try to read the file. If you still receive an exception, wait for a while and then retry.
You can keep track of the Created and Modified events eg. in a dictionary keyed by the file name and try to open the file only if no event is received for a file after a specific timeout.
Polling and timeouts can be implemented using timers.
If you don't like the delay introduced by polling, you can use a separate thread to wait on the list of events and process new events as they appear. You can use the functionality of the BlockingCollection in C# 4 to do this in a relatively simple way.
Use FileSystemWatcher http://msdn.microsoft.com/en-us/library/system.io.filesystemwatcher.aspx
Any image development will require to know how to load an image in C#.
FromFile
There are two direct ways to read an image file and load it into either a Bitmap or Image. Behold the C# code:
Image myImg = Image.FromFile("path here");
Bitmap myBmp = Bitmap.FromFile("path here");
Alternatively a Bitmap object can also be loaded with:
Bitmap myBmp = new Bitmap("path here");
The code above does not work with Image objects though, so it is best to stick with FromFile.
Dialog Box
Finally, to write an application that loads an image from a file, your C# program needs a dialogbox to select files. Using the .Net OpenFileDialog is simple enough. Just apply the image-loading code to the Filename chosen by the user, so for example:
Bitmap loadedBitmap = Bitmap.FromFile(openFileDialog1.Filename);
Of course, you don't have to necessarily load an image from a file in this manner, but it is a useful thing to know.
For more information HERE
I need to create a proxy-like solution that "forwards" an uploaded file to somewhere else. The problem I am facing is that with large uploads the file needs to be fully uploaded before my forwarding code gets control of it in Page_Loaded of an ASP.NET application.
Is there a way to get control of a Stream of a file being uploaded as soon as the upload begins? The idea is to start forwarding as soon as first file bytes start coming in without waiting for the entire upload to finish.
Can IHttpModule be used to achieve this goal?
Some time ago, I had to deal with images and some heavy files upload. We used this project:
http://silverlightfileupld.codeplex.com/
It is basically a silverlight file uploader, we added some feautures like image compression. Also the key part, is that we used this uploader to send chunks of the file instead of the whole file, so the new file was recreated as a temp file and then renamed. Maybe you can follow that approach and in the ashx file, you can transfer the bytes to the real server.
Hope this helps