Develop Reference

Zip Windows 10 Documents folder without including reparse points using System.IO.Compression

I'm writing a simple desktop application to copy files from one PC to another. Having trouble with the Windows 10 reparse points, specifically My Music. I thought was going to get away with one simple line of code:
ZipFile.CreateFromDirectory(documentsFolder, docSavePath + #"\Documents.zip", CompressionLevel.Optimal, false);
But not so, it crashes on the My Music folder. I've also tried a bunch of different ways of doing this, all with the same result - access denied. Can copying and/or zipping the Documents folder really be this hard? I doubt it, I'm just missing something. I tried elevating privileges and that didn't work, either. Anyone have an example of how to do this?

I was able figure out how to check for the ReparsePoint attribute, which was relatively easy, but then had to piece together how to loop through all the files and add them to the ZipArchive. The credit for the RecurseDirectory goes to this answer.
Then I added in what I learned about the reparse file attributes.
private void documentBackup(string docSavePath)
{
if (File.Exists(docSavePath + #"\Documents.zip")) File.Delete(docSavePath + #"\Documents.zip");
using (ZipArchive docZip = ZipFile.Open(docSavePath + "\\Documents.zip", ZipArchiveMode.Create))
{
foreach (FileInfo goodFile in RecurseDirectory(documentsFolder))
{
var destination = Path.Combine(goodFile.DirectoryName, goodFile.Name).Substring(documentsFolder.ToString().Length + 1);
docZip.CreateEntryFromFile(Path.Combine(goodFile.Directory.ToString(), goodFile.Name), destination);
}
}
}
public IEnumerable<FileInfo> RecurseDirectory(string path, List<FileInfo> currentData = null)
{
if (currentData == null)
currentData = new List<FileInfo>();
var directory = new DirectoryInfo(path);
foreach (var file in directory.GetFiles())
currentData.Add(file);
foreach (var d in directory.GetDirectories())
{
if ((d.Attributes & FileAttributes.ReparsePoint) == FileAttributes.ReparsePoint)
{
continue;
}
else
{
RecurseDirectory(d.FullName, currentData);
}
}
return currentData;
}
It takes longer than I'd like to run - but after looking at this dang problem for days I'm just happy it works!

How to prevent Directory.GetFiles to "check" recycle bin and other "unsafe" places?

So guys, I have a function in my application which to search for certain file in certain directory using GetFiles method
System.IO.Directory.GetFiles(string path, string searchPattern, System.IO.SearchOption)
It works fine, until when I choose drive directory (D:\ or C:\ and such) to be searched, because it's also accessing the Recycle Bin, and then restricted
Access to the path 'D:\$RECYCLE.BIN\S-1-5-21-106145493-3722843178-2978326776-1010' is denied.
It's also need to be able to search subfolders (SearchOption.AllDirectories) too.
How to SKIP such place to be searched? Because there may be any other folder which access also denied.
I capitalize SKIP because if I use try catch and an exception caught, then the entire search will also fail.
Thanks. Please clarify anything you need.

EDITed for more clarity.
When recursively scanning a directory tree, say using a recursive method which takes the directory to start with as a parameter, you can get the attributes of the directory. Then check whether it's a system directory AND NOT a root directory like "C:\" - in that case you want to skip that directory, as it may be, for instance, the recycle bin.
Here's some code that does this, and also catches some common exceptions which occurred when I fiddled with directory scanning.
void scan_dir(string path)
{
// Exclude some directories according to their attributes
string[] files = null;
string skipReason = null;
var dirInfo = new DirectoryInfo( path );
var isroot = dirInfo.Root.FullName.Equals( dirInfo.FullName );
if ( // as root dirs (e.g. "C:\") apparently have the system + hidden flags set, we must check whether it's a root dir, if it is, we do NOT skip it even though those attributes are present
(dirInfo.Attributes.HasFlag( FileAttributes.System ) && !isroot) // We must not access such folders/files, or this crashes with UnauthorizedAccessException on folders like $RECYCLE.BIN
)
{ skipReason = "system file/folder, no access";
}
if ( null == skipReason )
{ try
{ files = Directory.GetFiles( path );
}
catch (UnauthorizedAccessException ex)
{ skipReason = ex.Message;
}
catch (PathTooLongException ex)
{ skipReason = ex.Message;
}
}
if (null != skipReason)
{ // perhaps do some error logging, stating skipReason
return; // we skip this directory
}
foreach (var f in files)
{ var fileAttribs = new FileInfo( f ).Attributes;
// do stuff with file if the attributes are to your liking
}
try
{ var dirs = Directory.GetDirectories( path );
foreach (var d in dirs)
{ scan_dir( d ); // recursive call
}
}
catch (PathTooLongException ex)
{ Trace.WriteLine(ex.Message);
}
}

C# code to copy files, can this snippet be improved?

While copying around 50 GB of data via local LAN share, due to connectivity issue copy failed at around 10 GB copied.
I have renamed copied 10GB of data directory to localRepository and then written a C# program to copy files from the remote server to destination, only if it is not found in local repository. If found move file from local repository to destination folder.
Although the code worked fine and accomplishes the task very well. I wonder, have I written the most efficient code? Can you find any improvements?
string destinationFolder = #"C:\DataFolder";
string remoteRepository = #"\\RemoteComputer\DataFolder";
string localRepository = #"\\LocalComputer\LocalRepository";
protected void Page_Load(object sender, EventArgs e)
{
foreach (string remoteSrcFile in Directory.EnumerateFiles(remoteRepository, "*.*", SearchOption.AllDirectories))
{
bool foundInLocalRepo = false; ;
foreach (var localSrcFile in Directory.EnumerateFiles(localRepository, "*.*", SearchOption.AllDirectories))
{
if (Path.GetFileName(remoteSrcFile).Equals(Path.GetFileName(localSrcFile)))
{
FileInfo localFile = new FileInfo(localSrcFile);
FileInfo remoteFile = new FileInfo(remoteSrcFile);
//copy this file from local repository
if (localFile.Length == remoteFile.Length)
{
try
{
File.Move(localSrcFile, PrepareDestinationPath(remoteSrcFile));
Debug.WriteLine(remoteSrcFile + " moved from local repo");
}
catch (Exception ex)
{
Debug.WriteLine(remoteSrcFile + " did not move");
}
foundInLocalRepo = true;
break;
}
}
}
if (!foundInLocalRepo)
{
//copy this file from remote repository
try
{
File.Copy(remoteSrcFile, PrepareDestinationPath(remoteSrcFile), false);
Debug.WriteLine(remoteSrcFile + " copied from remote repo");
}
catch (Exception ex)
{
Debug.WriteLine(remoteSrcFile + " did not copy");
}
}
}
}
private string PrepareDestinationPath(string remoteSrcFile)
{
string relativePath = remoteSrcFile.Split(new string[] { "DataFolder" }, StringSplitOptions.None)[1];
string copyPath = Path.GetFullPath(destinationFolder + relativePath);
Directory.CreateDirectory(Path.GetDirectoryName(copyPath));
return copyPath;
}
EDIT:
Based on answer given by Thomas I am attempting to zip the file.
Traditionally as an end user we use to zip a file and then copy. As a programmer can we zip and copy the file parallel? I mean the portion which has been zipped send it over the wire?

You are doing far too much work with the nested loop.
You should remove the inner "foreach" and replace it with some code that:
(1) Constructs the name of the file that you are looking for and
(2) Uses File.Exists() to see if exists, then
(3) Continues with the same block of code that you currently have following the "if (Path.GetFileName(remoteSrcFile)..." condition.
Something like this:
foreach (string remoteSrcFile in Directory.EnumerateFiles(remoteRepository, "*.*", SearchOption.AllDirectories))
{
string localSrcFile = Path.Combine(localRepository, Path.GetFileName(remoteSrcFile));
if (File.Exists(localSrcFile))
{
...
}
}

I would suggest zipping the files before moving. Try take a look at the very simple http://dotnetzip.codeplex.com/
Try zipping 1000 files a time, in that way, you don't have to run the for-loop that many times and establish new connections etc each time.

Get all files and directories in specific path fast

I am creating a backup application where c# scans a directory. Before I use to have something like this in order to get all the files and subfiles in a directory:
DirectoryInfo di = new DirectoryInfo("A:\\");
var directories= di.GetFiles("*", SearchOption.AllDirectories);
foreach (FileInfo d in directories)
{
//Add files to a list so that later they can be compared to see if each file
// needs to be copid or not
}
The only problem with that is that sometimes a file could not be accessed and I get several errors. an example of an error that I get is:
As a result I created a recursive method that will scan all files in the current directory. If there where directories in that directory then the method will be called again passing that directory. The nice thing about this method is that I could place the files inside a try catch block giving me the option to add those files to a List if there where no errors and adding the directory to another list if I had errors.
try
{
files = di.GetFiles(searchPattern, SearchOption.TopDirectoryOnly);
}
catch
{
//info of this folder was not able to get
lstFilesErrors.Add(sDir(di));
return;
}
So this method works great the only problem is that when I scan a large directory it takes to much times. How could I speed up this process? My actual method is this in case you need it.
private void startScan(DirectoryInfo di)
{
//lstFilesErrors is a list of MyFile objects
// I created that class because I wanted to store more specific information
// about a file such as its comparePath name and other properties that I need
// in order to compare it with another list
// lstFiles is a list of MyFile objects that store all the files
// that are contained in path that I want to scan
FileInfo[] files = null;
DirectoryInfo[] directories = null;
string searchPattern = "*.*";
try
{
files = di.GetFiles(searchPattern, SearchOption.TopDirectoryOnly);
}
catch
{
//info of this folder was not able to get
lstFilesErrors.Add(sDir(di));
return;
}
// if there are files in the directory then add those files to the list
if (files != null)
{
foreach (FileInfo f in files)
{
lstFiles.Add(sFile(f));
}
}
try
{
directories = di.GetDirectories(searchPattern, SearchOption.TopDirectoryOnly);
}
catch
{
lstFilesErrors.Add(sDir(di));
return;
}
// if that directory has more directories then add them to the list then
// execute this function
if (directories != null)
foreach (DirectoryInfo d in directories)
{
FileInfo[] subFiles = null;
DirectoryInfo[] subDir = null;
bool isThereAnError = false;
try
{
subFiles = d.GetFiles();
subDir = d.GetDirectories();
}
catch
{
isThereAnError = true;
}
if (isThereAnError)
lstFilesErrors.Add(sDir(d));
else
{
lstFiles.Add(sDir(d));
startScan(d);
}
}
}
Ant the problem if I try to handle the exception with something like:
DirectoryInfo di = new DirectoryInfo("A:\\");
FileInfo[] directories = null;
try
{
directories = di.GetFiles("*", SearchOption.AllDirectories);
}
catch (UnauthorizedAccessException e)
{
Console.WriteLine("There was an error with UnauthorizedAccessException");
}
catch
{
Console.WriteLine("There was antother error");
}
Is that if an exception occurs then I get no files.

This method is much faster. You can only tel when placing a lot of files in a directory. My A:\ external hard drive contains almost 1 terabit so it makes a big difference when dealing with a lot of files.
static void Main(string[] args)
{
DirectoryInfo di = new DirectoryInfo("A:\\");
FullDirList(di, "*");
Console.WriteLine("Done");
Console.Read();
}
static List<FileInfo> files = new List<FileInfo>(); // List that will hold the files and subfiles in path
static List<DirectoryInfo> folders = new List<DirectoryInfo>(); // List that hold direcotries that cannot be accessed
static void FullDirList(DirectoryInfo dir, string searchPattern)
{
// Console.WriteLine("Directory {0}", dir.FullName);
// list the files
try
{
foreach (FileInfo f in dir.GetFiles(searchPattern))
{
//Console.WriteLine("File {0}", f.FullName);
files.Add(f);
}
}
catch
{
Console.WriteLine("Directory {0} \n could not be accessed!!!!", dir.FullName);
return; // We alredy got an error trying to access dir so dont try to access it again
}
// process each directory
// If I have been able to see the files in the directory I should also be able
// to look at its directories so I dont think I should place this in a try catch block
foreach (DirectoryInfo d in dir.GetDirectories())
{
folders.Add(d);
FullDirList(d, searchPattern);
}
}
By the way I got this thanks to your comment Jim Mischel

In .NET 4.0 there's the Directory.EnumerateFiles method which returns an IEnumerable<string> and is not loading all the files in memory. It's only once you start iterating over the returned collection that files will be returned and exceptions could be handled.

There is a long history of the .NET file enumeration methods being slow. The issue is there is not an instantaneous way of enumerating large directory structures. Even the accepted answer here has its issues with GC allocations.
The best I've been able to do is wrapped up in my library and exposed as the FindFile (source) class in the CSharpTest.Net.IO namespace. This class can enumerate files and folders without unneeded GC allocations and string marshalling.
The usage is simple enough, and the RaiseOnAccessDenied property will skip the directories and files the user does not have access to:
private static long SizeOf(string directory)
{
var fcounter = new CSharpTest.Net.IO.FindFile(directory, "*", true, true, true);
fcounter.RaiseOnAccessDenied = false;
long size = 0, total = 0;
fcounter.FileFound +=
(o, e) =>
{
if (!e.IsDirectory)
{
Interlocked.Increment(ref total);
size += e.Length;
}
};
Stopwatch sw = Stopwatch.StartNew();
fcounter.Find();
Console.WriteLine("Enumerated {0:n0} files totaling {1:n0} bytes in {2:n3} seconds.",
total, size, sw.Elapsed.TotalSeconds);
return size;
}
For my local C:\ drive this outputs the following:
Enumerated 810,046 files totaling 307,707,792,662 bytes in 232.876 seconds.
Your mileage may vary by drive speed, but this is the fastest method I've found of enumerating files in managed code. The event parameter is a mutating class of type FindFile.FileFoundEventArgs so be sure you do not keep a reference to it as it's values will change for each event raised.

I know this is old, but... Another option may be to use the FileSystemWatcher like so:
void SomeMethod()
{
System.IO.FileSystemWatcher m_Watcher = new System.IO.FileSystemWatcher();
m_Watcher.Path = path;
m_Watcher.Filter = "*.*";
m_Watcher.NotifyFilter = m_Watcher.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite | NotifyFilters.FileName | NotifyFilters.DirectoryName;
m_Watcher.Created += new FileSystemEventHandler(OnChanged);
m_Watcher.EnableRaisingEvents = true;
}
private void OnChanged(object sender, FileSystemEventArgs e)
{
string path = e.FullPath;
lock (listLock)
{
pathsToUpload.Add(path);
}
}
This would allow you to watch the directories for file changes with an extremely lightweight process, that you could then use to store the names of the files that changed so that you could back them up at the appropriate time.

(copied this piece from my other answer in your other question)
Show progress when searching all files in a directory
Fast files enumeration
Of course, as you already know, there are a lot of ways of doing the enumeration itself... but none will be instantaneous. You could try using the USN Journal of the file system to do the scan. Take a look at this project in CodePlex: MFT Scanner in VB.NET... it found all the files in my IDE SATA (not SSD) drive in less than 15 seconds, and found 311000 files.
You will have to filter the files by path, so that only the files inside the path you are looking are returned. But that is the easy part of the job!

Maybe it will be helpfull for you.
You could use "DirectoryInfo.EnumerateFiles" method and handle UnauthorizedAccessException as you need.
using System;
using System.IO;
class Program
{
static void Main(string[] args)
{
DirectoryInfo diTop = new DirectoryInfo(#"d:\");
try
{
foreach (var fi in diTop.EnumerateFiles())
{
try
{
// Display each file over 10 MB;
if (fi.Length > 10000000)
{
Console.WriteLine("{0}\t\t{1}", fi.FullName, fi.Length.ToString("N0"));
}
}
catch (UnauthorizedAccessException UnAuthTop)
{
Console.WriteLine("{0}", UnAuthTop.Message);
}
}
foreach (var di in diTop.EnumerateDirectories("*"))
{
try
{
foreach (var fi in di.EnumerateFiles("*", SearchOption.AllDirectories))
{
try
{
// Display each file over 10 MB;
if (fi.Length > 10000000)
{
Console.WriteLine("{0}\t\t{1}", fi.FullName, fi.Length.ToString("N0"));
}
}
catch (UnauthorizedAccessException UnAuthFile)
{
Console.WriteLine("UnAuthFile: {0}", UnAuthFile.Message);
}
}
}
catch (UnauthorizedAccessException UnAuthSubDir)
{
Console.WriteLine("UnAuthSubDir: {0}", UnAuthSubDir.Message);
}
}
}
catch (DirectoryNotFoundException DirNotFound)
{
Console.WriteLine("{0}", DirNotFound.Message);
}
catch (UnauthorizedAccessException UnAuthDir)
{
Console.WriteLine("UnAuthDir: {0}", UnAuthDir.Message);
}
catch (PathTooLongException LongPath)
{
Console.WriteLine("{0}", LongPath.Message);
}
}
}

You can use this to get all directories and sub-directories. Then simply loop through to process the files.
string[] folders = System.IO.Directory.GetDirectories(#"C:\My Sample Path\","*", System.IO.SearchOption.AllDirectories);
foreach(string f in folders)
{
//call some function to get all files in folder
}

DirectoryNotFoundException when calling Directory.GetDirectories on Environment.SpecialFolder.Favorites due to Domain Folder Redirection

I have some C# code that tries to get the Favorites for the currently logged in user. The code is part of a Taskbar Toolbar that gets loaded into the Windows Explorer process. I have a user who is using Windows Vista with UAC enabled on a domain that either has Roaming Profiles or Folder Redirection setup and enabled. When calling Directory.GetDirectories on the Favorites path, it throws "System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\Users\\Favorites\". Other users on other domains that do not have Roaming Profiles or Folder Redirection setup do not have this issue.
The user also reported that copying the path from the failed logs into the run prompt fails to load the path, but if they navigate to the path directly using explorer and then copy and paste that path into the run prompt, it works. He sent me both paths and they are exactly identical which doesn't make any sense at all.
My theory is that this is caused by the Folder Redirection where that path is actually pointing to a share on the server but the redirection is failing when trying to access the subdirectories (of the directoryInfo returned from Directory.GetDirectories). The initial directory works but all subdirectories of the initial directory fail to redirect correctly.
Has anyone come across a situation like this and/or know a workaround to gain proper access to redirected folders?
private void GetFavorites()
{
try
{
System.IO.DirectoryInfo dirInfo = new System.IO.DirectoryInfo(Environment.GetFolderPath(Environment.SpecialFolder.Favorites));
AddFavorites(dirInfo);
}
catch
{
}
}
private void AddFavorites(DirectoryInfo dirInfo)
{
foreach (System.IO.FileInfo fileInfo in dirInfo.GetFiles("*.url"))
{
//string alias = fileInfo.Name.Replace(".url", "");
if (!ItemsBookmarks.ContainsKey(fileInfo.Name))
ItemsBookmarks.Add(fileInfo.Name, fileInfo.Name);
}
foreach (System.IO.FileInfo fileInfo in dirInfo.GetFiles("*.lnk"))
{
if (!ItemsBookmarks.ContainsKey(fileInfo.Name))
ItemsBookmarks.Add(fileInfo.Name, fileInfo.Name);
}
foreach (System.IO.DirectoryInfo objDir in dirInfo.GetDirectories())
{
AddFavorites(objDir);
}
}
Thanks,
John

I believe the problem you are experiencing is related to Reparse Points.
See: http://msdn.microsoft.com/en-us/library/bb513869.aspx
See: What is the best way to check for reparse point in .net (c#)?
The problem can be avoided by using the following syntax:
private void AddFavorites(string dirPath)
{
try
{
foreach (string fileName in Directory.GetFiles(dirPath, "*.*", SearchOption.TopDirectoryOnly))
{
//string alias = fileInfo.Name.Replace(".url", "");
if (!ItemsBookmarks.ContainsKey(fileInfo.Name))
{
ItemsBookmarks.Add(fileName);
}
}
foreach (string subDirName in Directory.GetDirectories(dirPath, "*.*", SearchOption.TopDirectoryOnly))
{
AddFavorites(objDir);
}
}
catch
{
//error getting files or subdirs... permissions issue?
//throw
}
}