I have to add a progress Bar in my AV and the problem is that I don't know in advance how many files to scan.So I don't know the total count.If I count the files to be scanned,then the soft pauses for a while since it has to go through many files and in case of Full System Scan,this pause if of long duration which is not desired.Hence I dropped that code.I heard that avast dynamically changes progress bar value i.e. if the value approaches 100 and more files are found to scan then it pushes the value to like 50.Coding is in C#.
It would be foolish to try to get a list of all the files in one pass, before updating the UI at all.
Use threads; one to locate files and add their locations to a queue, and update a total file count.
Another to do the scanning, and update the count of scanned files.
and the UI thread will update itself based on that; so the upper bound will constantly be increasing... or, becoming more accurate.
Just count the folders 5-level deep, and calculate the progress based on how many of those folders were scanned. Increase dept to increase accuracy (at the cost of pre-calculation time). Use this article to learn how to enumerate directories as fast as possible: MSDN How to: Enumerate Directories and Files
Since AV scans should happen on a regular basis can you just remember how many files were in the last scan and then pad it by 5%? That would give the user a rough idea of how long it would take without having to do an up to date full scan right now. In order to get an accurate number for the first scan you would have to do a full directory search and suffer the pause that you described. As for custom individual folders you could either store the # of files for the first 2-3 levels of folders and then just do a directory scan below that as there would be a reduced number of files in each sub-directory.
I would recommend checking how much occupied space there is on the disk and evaluating number of files basing on density: files per GB. At first You have to make an assumption how much files do get stored in one GB (for my system it is 3860) During the scan you will know what is real density of scanned hard drive so you can adjust number of files.
Adjusting total number of files during the scan is the best method I can think off, and being user of you app I would like it much more, than waiting until you count all the files on the machine.
Related
I have 2 folders with files on Windows:
Local folder e.g C:\MyFolder
Network folder e.g. \server\MyFolder
I need to check whether \server\MyFolder has got any updates in compare to my local one (e.g. updated file etc.).
I only need to know if there is any update in the network folder, I don't need to know what are the differences.
I tried implementing MD5 check sums on the folders, but this solution is too slow. Creating MD5 checksum on the network folder is just too slow (in my case 3-4 minutes, but I need this comparison to be made in max. few seconds) [we are talking about folders with few hundred files, and few hundred MB in size]
I tried implementing some basic rules (e.g. if number of files differs, names are different then it is a quick response), but in most of the cases number of files remain the same, and the only change is in the content of the file, so as long as the first files changes then it is quick response, but in the worst case scenario (or in the scenario that folders are the same) I end up iterating through all files, which is very slow (few minutes).
What other approaches could I try ?
(Note, that network drive is read only to me, so I can't store any information, checksums in the folder whenever it changes).
Here's a solution :
You loop your folder's file and everytime check this variable.
var lastModified = System.IO.File.GetLastWriteTime("Server/Folder");
if lastModified > the time of your code execution recurrence. then you should return the true or false
Problem
I need to make a dialog, in which there are two progress bars: One for total progress (if multiple files copied / moved / deleted) and one for the current file's progress.
I basically need to have some way of copying, moving and deleting files and know the progress of the corresponding process in order to update my visuals.
I already found a question about the copying problem here, but this does not really work for moving, because if you move files around on the same disk, they usually do not get copied but rather the directory of them gets changed. (Moving to other disk results in copying and deleting) Also I do not know how to achieve the deleting process.
What I have already tried
Well, given that I posted a link to a solution for the copying process, I already solved the copying part (kind of) by modifying the code so that it suits my needs.
What is asked for?
A way to know the process of copying, moving or deleting one file. If I know this, I can figure the rest of it out.
Thanks
I once had the need to show a progress bar for a mix of copy, move and delete. For copying computing progress is easy. You base the computation on bytes. But what about moves and deletes?
I based progress on "synthetic bytes". The cost for a move or a delete comes mainly from the disk seeks required to address the relevant directory entries. I therefore counted them as a synthetic number of bytes that was roughly equivalent to one disk seek. For a magnetic disk a seek is ~10ms. At 100MB/sec that's a synthetic cost of 1MB. But I think multiple moves can be processed more efficiently by the OS so, if I recall correctly, I just made the cost of a move/delete 64KB.
The progress bar seemed fairly reasonable and the concept is very extensible to new operation types and new disk types (SSD).
I hope this applies to your case.
I am developing a winform application that moves files between directories in a SAN. The app searches a list of files in directories with over 200,000 files (per directory) in the SAN's Directory, and then moves the found files to another directory.
For example I have this path:
\san\xxx\yyy\zzz
I perform the search in \zzz and then move the files to \yyy, but while I'm moving the files, another app is inserting more files into the xxx, yyy and zzz directories.
I don't want to impact the access or performance to other applications that use these directories.
The file's integrity is also a big concern, because when the application moves the files to \yyy directory another app uses those files.
All of these steps should run in parallel.
What methods could I use to achieve this?
How can I handle concurrency?
Thanks in advance.
Using the published .net sources, one can see how the move file operation works Here. Now if you follow the logic you'll find there's a call to KERNEL32. The MSDN documentation for this states
To perform this operation as a transacted operation, use the MoveFileTransacted function.
Based on that I'd say the move method is a safe/good way to move a file.
Searching through a large number of files is always going to take time. If you need to increase the speed of this operation, I'd suggest think out of side of the box to achieve this. For instance, I've seen cache buckets produced for both letters and dates. Using some sort of cache system, you could search for files in a quicker manner, say by asking the cache bucket for files starting with "a" for "these files like this starting with a" file, or files between these dates - etc.
Presumably, these files are used by something. For the purposes of this conversation, let's assume they are indexed in an SQL Database.
TABLE: Files
File ID Filename
-------------- ----------------------------
1 \\a\b\aaaaa.bin
2 \\a\b\bbbbb.bin
To move them to \\a\c\ without impacting access, you can execute the following:
Copy the file from \\a\b\aaaa.bin to \\a\c\aaaa.bin
Update the database to point to \\a\c\aaaa.bin
Once all open sessions have been closed for \\a\b\aaaa.bin, delete the old file
If the files are on the same volume, you could also use a hardlink instead of a copy.
I wrote a code in C++ CLI which can loop through all files in the system. I want to know the number of files are existing in the system programmatically without counting the files one by one!
Also...Would that number be useful in scanning progress bar?
If you are using NTFS, you can call the FSCTL_GET_NTFS_VOLUME_DATA IoControl to get a whole bunch of data for your volume. The number of files will be at most MftValidDataLength / BytesPerFileRecordSegment. Keep in mind that this is a maximum number of files that can be on the volume, so the actual number will always be less.
I would also hope that you'd be using FSCTL_ENUM_USN_DATA to read the files rather than walking the directory tree (which is inaccurate and error-prone). This thread has an example of somebody doing it in C#: http://social.msdn.microsoft.com/Forums/en-US/csharpgeneral/thread/c1550294-d121-4511-ac32-31551497f64e/
That would depend highly on the filesystem in use (and the administrative privileges held by the impersonated user :))
You might find something here or you can enhance your question with more details
I have been trying to figure out a way to streamline copying files from one drive (network or external in my case) to my main system drives. Short of creating a program that I will have to activate each time then choosing all the info in it, I have no really good design to bypass it.
I have been wondering if there was a way to intercept a straight-forward windows copy with a program to do it my way. The basic design would be that upon grabbing the copy (actually for this to be efficient, a group of different copies), the program would organize all the separate copies into a single stream of copies.
I've been wanting to do this as recently I have been needing to make backups of a lot of data and move it a lot as all my drives seem to be failing the past few months.
EDIT
For the moment, let's assume I am copying from a fully defragmented external drive (because I was). Let's also assume I am copying to a 99% defragmented drive (because I was). If I attempt to move every file at once, then the Windows copy method works great as it only needs to stream a contiguous file from one drive to the other (likely keeping the copy contiguous). This is basically best case, I need everything from there to move to here. So the copy method gets to grab the files in some algorithmicly logical order (typically by filename) and read then write the data to the new location.
Now for the reality of the situation, I instead copy a bunch of files from different locations (let's still assume fully defragmented as this is the case of my external), to different file locations on the second drive. I don't wish to wait for each one to finish before starting the next one, so I go through the folders picking and choosing what goes where and soon my screen is covered in COPY windows displaying the status of all the copies attempting to run at the same time. This presents a slight slowing down of the copy as a whole as each copy progresses independently of the others. That means that instead of a single contiguous file taking the total bandwidth available, all these files share the bandwidth approximately equally (this also adds in read and write delays to the disks as now the system has to grab a little from each file before circling back to the first). Still assuming best case scenario, this slower than if all the files were in one location and moved to one location.
What I want to do is intercept all the separate downloads and organize them into a list of single downloads that would happen one after another.
The only way you can intercept nicely is the FileSystemWatcher.
This is probably what you are looking for
detect Windows Explorer copy operation
You need to create a Shell Extension according to this post.