Listing large number of files in multi level directories - c#

I've got quite a big problem. I have a drive with nearly 600,000 files on it within various levels of folders and I need to create an SQL table with a list of filepaths but not for all of the files.
The structure is:
J:/
J:/test
J:/XCR
J:/554
J:/B1
I need to keep all the files in the folders test & XCR but only files within certain directories in the other folders. I have a list of these folders uploaded to an SQL table.
I've written a recursive function that uses Directory.GetFiles() and Directory.GetDirectories but I encounter problems when the directory levels increase and the filenames get longer. I've also have the following error that I'm not sure about...
System.IO.IOException: Element not found. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath
I think I'm going about this the wrong way. Can someone point me in the right direction?
Any help would be greatly appreciated!
EDIT:
public void copyFiles(string dirName, int level)
{
try
{
if (level == 1)
{
foreach (string f in Directory.GetFiles(dirName))
{
Response.Write(f + "<br />");
// insTbl.Rows.Add(f);
}
}
foreach (string d in Directory.GetDirectories(dirName))
{
foreach (string f in Directory.GetFiles(d))
{
Response.Write(f + "<br />");
// insTbl.Rows.Add(f);
}
level++;
copyFiles(d,level);
}
}
catch (Exception ex)
{
Response.Write(ex);
}
}

Related

Recursive Deletion of Folders in C#

I am trying to recursively delete all empty subdirectories in a root directory and (not files)using C#. I have referred this post (Recursive delete of files and directories in C#)
and wrote the function as follows.
private void DeleteEmptySubFolders()
{
DirectoryInfo root = new DirectoryInfo(folderPath);
RecursiveDelete(root,true);
}
public static void RecursiveDelete(DirectoryInfo baseDir,bool isRootDir)
{
if (!baseDir.Exists)
return;
foreach (var dir in baseDir.EnumerateDirectories())
{
RecursiveDelete(dir,false);
}
try
{
if (!isRootDir)
baseDir.Delete(false);
}
catch(Exception ex)
{
}
}
My question is baseDir.Delete(false); will give me an exception when a directory is not empty, and I am just passing that exception without handling it(I just want to skip those ones, don't have to log). Is there a better way to do this?
Instead of try/catch, check if any files exist:
bool isEmpty = !Directory.EnumerateFileSystemEntries(baseDir.FullName).Any();
Now you can avoid trying to delete non-empty folders:
if (!isRootDir && isEmpty)
baseDir.Delete(false);

Search for ".myox" files on Computer

trying to write a small windows application for my company. the part i am stuck at the moment is trying to search the computer for ".myox" files (or say any file type). Below pasted is the code i have worked out. I am an amateur programmer trying to get started with coding. The issue am having at the moment with the code below is its skipping almost all locations on the computer with the exception coming up as "access denied". I have run the VS as admin, and i am an admin on the computer as well. Not sure what i am missing, but if someone can point me in the right direction, that would be amazing.
private void FindAllFiles()
{
int drvCount;
int drvSearchCount = 0;
DriveInfo[] allDrives = DriveInfo.GetDrives();
drvCount = allDrives.Count();
foreach (DriveInfo dr in allDrives)
{
lbAllFiles.Items.Clear();
drvSearchCount++;
//removable drives
if (!dr.IsReady)
break;
foreach (string dir in Directory.GetDirectories(dr.ToString()))
{
try
{
foreach (string files in Directory.GetFiles(dir, "*.myox"))
{
lbAllFiles.Items.Add(files);
}
}
catch (Exception Error)
{
}
}
if (drvSearchCount == drvCount)
break;
}
MessageBox.Show("Done searching your computer");
}
Thanks in Advance.
-Manu
I see few "potential" issues and will list them below.
First is that you're doing this on main ( UI ) thread which will block whole application giving you no feedback about current state. You can use Thread to get rid of this problem. Outcome from this operation will produce another issue which is accessing lbAllFiles because ( as i think ) it's part of the UI. You can easily get rid of this problem making a List<string> that can be filled during FindAllFiles operation and then assigned into lbAllFiles.Items.
Second issue is :
foreach (string files in Directory.GetFiles(dir, "*.myox"))
{
lbAllFiles.Items.Add(files);
}
Directory.GetFiles(...) will return only the files that are matching your pattern parameter so you can simply do :
var files = Directory.GetFiles(dir, "*.myox");
if ( files != null && files.Length > 0 )
lblAllFiles.Items.AddRange(files);
And finaly to get ( or check ) permission you can Demand() permissions as I've posted in the comment :
foreach (string dir in Directory.GetDirectories(dr.ToString()))
{
FileIOPermission permission = new FileIOPermission(FileIOPermissionAccess.Read, dir);
try
{
permission.Demand();
var files = Directory.GetFiles(dir, "*.myox");
if ( files != null && files.Length > 0 )
lblAllFiles.Items.AddRange(files);
}
catch (Exception Error)
{
}
}
Let me know if that helped you. If not I'll try to update my answer with another solution.
One thing i noticed in your code, is that you're not navigating through ALL directories and sub-directories. For that, where you call the GetDirectories function, not only send the path, but use the enumerator Alldirectories:
foreach (string dir in Directory.GetDirectories(dr.ToString(),System.IO.SearchOption.AllDirectories))

C# File Permissions

I am currently writing a program in C# that will copy all user profile files to an external device (in this case, my home server).
When my code iterates through my files and folders, it throws an UnauthorizedAccessException.
I have Googled this and searched StackOverflow, but I am unable to find a clear solution that doesn't involve terminating my process. The idea is that it should copy the files and folders that have read permissions.
I had this when I first started, but I easily fixed it by limiting what directories I would backup (though I would prefer a full backup).
Here is my code:
FileInfo f = new FileInfo(_configuration.Destination);
if (!f.Directory.Exists)
{
f.Directory.Create();
}
string[] backupDirectories = new string[]
{
"Desktop",
"Documents",
"Downloads",
"Favorites",
"Links",
"Pictures",
"Saved Games",
"Searches",
"Videos",
".git",
".android",
".IdealC15",
".nuget",
".oracle_jre_usage",
".vs",
"Contacts"
};
foreach (string dirPath in backupDirectories)
{
DirectoryInfo dirInfo = new DirectoryInfo(_path + "\\" + dirPath);
if (dirInfo.Exists)
{
foreach (string dirP in Directory.GetDirectories(dirInfo.FullName, "*", SearchOption.AllDirectories))
{
DirectoryInfo dirI = new DirectoryInfo(dirP);
if (dirI.Exists)
{
string dir = dirP.Replace(_path, _configuration.Destination);
try
{
Directory.CreateDirectory(dir);
textBox2.Invoke((MethodInvoker) delegate
{
textBox2.AppendText("Create Directory: " + dir + Environment.NewLine);
});
} catch (Exception e)
{
textBox2.Invoke((MethodInvoker) delegate
{
textBox2.AppendText("Could NOT Create Directory: " + dir + Environment.NewLine);
});
continue;
}
foreach (FileInfo theFile in dirInfo.GetFiles("*", SearchOption.AllDirectories))
{
string newPath = theFile.FullName;
string file = newPath.Replace(_path, _configuration.Destination);
try
{
File.Copy(newPath, file, true);
textBox2.Invoke((MethodInvoker) delegate
{
textBox2.AppendText("Create File: " + file + Environment.NewLine);
});
} catch (Exception ex)
{
textBox2.Invoke((MethodInvoker) delegate
{
textBox2.AppendText("Could NOT Create File: " + file + Environment.NewLine);
});
}
}
}
}
}
}
I apologise if the code is messy, but I will describe sort of what it is doing. The first bit checks if the backup folder exists on the external drive.
The second part says what folders I need to backup (if you're able to fix this and make it backup all directories with permissions, please help me in doing so).
The first loop starts the iteration for each of the backupDirectories. The second loop starts the iteration for each of the directories in the backup directory. The third loop starts the iteration for each of the folders in the backup directory.
The exception is thrown at Directory.GetDirectories(dirInfo.FullName, "*", SearchOption.AllDirectories), and it is trying to access C:\Users\MyName\Documents\My Music. Attempting to access it in explorer does give me a permissions error, though it isn't listed in explorer when I try going to "Documents" (I am in Windows 10 Pro).
As I recommended, since the Operating System authority is higher than the application, it is likely that you cannot do more than what the Operating System would allow you to do (that is to access or not to access certain folder).
Thus, folders' accessibility is best solved in the Operating System level.
But you could still do two things in the program level to minimize the damage when you search for the items.
To use Directory.AccessControl to know the access level of a directory before you do any query on it. This is not so easy, and there are elaborated answers about this here and also here.
To minimize the damage made by unauthorized access issues by using SearchOption.TopDirectoryOnly instead of SearchOption.AllDirectories, combined with recursive search for all the accessible directories. This is how you can code it
public static List<string> GetAllAccessibleDirectories(string path, string searchPattern) {
List<string> dirPathList = new List<string>();
try {
List<string> childDirPathList = Directory.GetDirectories(path, searchPattern, SearchOption.TopDirectoryOnly).ToList(); //use TopDirectoryOnly
if (childDirPathList == null || childDirPathList.Count <= 0) //this directory has no child
return null;
foreach (string childDirPath in childDirPathList) { //foreach child directory, do recursive search
dirPathList.Add(childDirPath); //add the path
List<string> grandChildDirPath = GetAllAccessibleDirectories(childDirPath, searchPattern);
if (grandChildDirPath != null && grandChildDirPath.Count > 0) //this child directory has children and nothing has gone wrong
dirPathList.AddRange(grandChildDirPath.ToArray()); //add the grandchildren to the list
}
return dirPathList; //return the whole list found at this level
} catch {
return null; //something has gone wrong, return null
}
}
The function above minimize the damage caused by the unauthorized access only to the sub-directories which have the issue. All other accessible directories can be returned.

getfiles is ignoring current user folder?

The folder C:\Users contain 3 subfolders :
C:\Users\hacen
C:\Users\_rafi_000
C:\Users\Public
However, when I call :
DirSearch(#"C:\Users\", "*.jpg");
It outputs all jpg filenames from Public and hacen, but not from _rafi_000 which is the folder of current user.
Here is the function :
static void DirSearch(string dir, string pattern)
{
try
{
foreach (string f in Directory.GetFiles(dir, pattern))
{
Console.WriteLine(f);
}
foreach (string d in Directory.GetDirectories(dir))
{
DirSearch(d, pattern);
}
}
catch (System.Exception ex)
{
//MessageBox.Show(ex.Message);
}
}
EDIT:
I tried with the code below and it works. So it isn't an access denied problem :
DirSearch("C:\Users\_rafi_000\","*.jpg");
What I noticed so far is that unlike other subfolders, the folder _rafi_000 cannot be ranamed when I press F2
Would this work?
void DirSearch(string dir, "*.JPG")
{
foreach (string f in Directory.GetFiles(dir, "*.JPG"))
{
Console.WriteLine(f);
}
foreach (string d in Directory.GetDirectories(dir))
{
DirSearch(d);
}
}
Could be related to where the jpg are stored and how reparse points work in later windows version.
I suggest looking at:
Directory Searching: http://msdn.microsoft.com/en-us/library/bb513869.aspx
Reparse Point Info: http://msdn.microsoft.com/en-us/library/aa365503(VS.85).aspx
I ran your code and it works fine in windows XP:
C:\Users\hacen\bar.jpg
C:\Users\Public\bar1.jpg
C:\Users\_rafi_000\bar2.jpg
Your code is correct.
Perhaps Process Monitor can help?
If the code is fine, it must be something else. I understand that you can run the code directly against the directory which is causing you a problem (which is surprising) - but I think Process Mon could help.

C# code to copy files, can this snippet be improved?

While copying around 50 GB of data via local LAN share, due to connectivity issue copy failed at around 10 GB copied.
I have renamed copied 10GB of data directory to localRepository and then written a C# program to copy files from the remote server to destination, only if it is not found in local repository. If found move file from local repository to destination folder.
Although the code worked fine and accomplishes the task very well. I wonder, have I written the most efficient code? Can you find any improvements?
string destinationFolder = #"C:\DataFolder";
string remoteRepository = #"\\RemoteComputer\DataFolder";
string localRepository = #"\\LocalComputer\LocalRepository";
protected void Page_Load(object sender, EventArgs e)
{
foreach (string remoteSrcFile in Directory.EnumerateFiles(remoteRepository, "*.*", SearchOption.AllDirectories))
{
bool foundInLocalRepo = false; ;
foreach (var localSrcFile in Directory.EnumerateFiles(localRepository, "*.*", SearchOption.AllDirectories))
{
if (Path.GetFileName(remoteSrcFile).Equals(Path.GetFileName(localSrcFile)))
{
FileInfo localFile = new FileInfo(localSrcFile);
FileInfo remoteFile = new FileInfo(remoteSrcFile);
//copy this file from local repository
if (localFile.Length == remoteFile.Length)
{
try
{
File.Move(localSrcFile, PrepareDestinationPath(remoteSrcFile));
Debug.WriteLine(remoteSrcFile + " moved from local repo");
}
catch (Exception ex)
{
Debug.WriteLine(remoteSrcFile + " did not move");
}
foundInLocalRepo = true;
break;
}
}
}
if (!foundInLocalRepo)
{
//copy this file from remote repository
try
{
File.Copy(remoteSrcFile, PrepareDestinationPath(remoteSrcFile), false);
Debug.WriteLine(remoteSrcFile + " copied from remote repo");
}
catch (Exception ex)
{
Debug.WriteLine(remoteSrcFile + " did not copy");
}
}
}
}
private string PrepareDestinationPath(string remoteSrcFile)
{
string relativePath = remoteSrcFile.Split(new string[] { "DataFolder" }, StringSplitOptions.None)[1];
string copyPath = Path.GetFullPath(destinationFolder + relativePath);
Directory.CreateDirectory(Path.GetDirectoryName(copyPath));
return copyPath;
}
EDIT:
Based on answer given by Thomas I am attempting to zip the file.
Traditionally as an end user we use to zip a file and then copy. As a programmer can we zip and copy the file parallel? I mean the portion which has been zipped send it over the wire?
You are doing far too much work with the nested loop.
You should remove the inner "foreach" and replace it with some code that:
(1) Constructs the name of the file that you are looking for and
(2) Uses File.Exists() to see if exists, then
(3) Continues with the same block of code that you currently have following the "if (Path.GetFileName(remoteSrcFile)..." condition.
Something like this:
foreach (string remoteSrcFile in Directory.EnumerateFiles(remoteRepository, "*.*", SearchOption.AllDirectories))
{
string localSrcFile = Path.Combine(localRepository, Path.GetFileName(remoteSrcFile));
if (File.Exists(localSrcFile))
{
...
}
}
I would suggest zipping the files before moving. Try take a look at the very simple http://dotnetzip.codeplex.com/
Try zipping 1000 files a time, in that way, you don't have to run the for-loop that many times and establish new connections etc each time.

Categories

Resources