first of all - don't look at the code and say it's too long
it only looks that way.
I'm writing a program that will search my computer and delete files based on their MD5 value (and to speed things up i don't want to search all the files, just those that have specific file names).
I am sending a FileInfo to a method named ConditionallyDeleteNotWantedFile, it then takes that file's name and trys to find it in the dictionary - retrieve that file's MD5 and computes the current FileInfo MD5 to see if they are the same.
If it does - delete the file.
the problem? exception is thrown when i try to delete... even tho no other process uses it. when i try to delete the file using windows explorer it says vshost (meaning:VS...)
what am i missing ?
public static bool ConditionallyDeleteNotWantedFile(FileInfo fi)
{
string dictMD5;
if (NotWanted.TryGetValue(fi.Name, out dictMD5))
{
string temp = ComputeMD5Hash(fi.FullName);
// temp will only be null if we couldn't open the file for
// read in the md5 calc operation. probably file is in use.
if (temp == null)
return false;
if (temp == dictMD5)
{
try
{
fi.Delete();
}
catch { fi.Delete(); // this exception is raised with
// "being used by another process"
}
return true;
}
}
return false;
}
public static string ComputeMD5Hash(string fileName)
{
return ComputeHash(fileName, new MD5CryptoServiceProvider());
}
public static string ComputeHash(string fileName, HashAlgorithm
hashAlgorithm)
{
try
{
FileStream stmcheck = File.OpenRead(fileName);
try
{
stmcheck = File.OpenRead(fileName);
byte[] hash = hashAlgorithm.ComputeHash(stmcheck);
string computed = BitConverter.ToString(hash).Replace("-", "");
stmcheck.Close();
return computed;
}
finally
{
stmcheck.Close();
}
}
catch
{
return null;
}
}
I don't know if that's the key, but you're opening the stream twice in ComputeHash, and there's a path that does not close it. May I suggest this:
public static string ComputeHash(string fileName, HashAlgorithm hashAlgorithm)
{
string hashFixed = null;
try
{
using (FileStream stmcheck = File.OpenRead(fileName))
{
try
{
byte[] hash = hashAlgorithm.ComputeHash(stmcheck);
hashFixed = BitConverter.ToString(hash).Replace("-", "");
}
catch
{
//logging as needed
}
finally
{
stmcheck.Close();
}
}
}
catch
{
//logging as needed
}
return hashFixed;
}
Related
I am new to object-oriented programming and I am working on a small personal project with some SQL scripts.
I have a scenario where a SQL script calls a static method with a file path as input.
queries = Select Query from Table where Utils.ContainsKeyword(Query, #Path1) AND NOT Utils.ContainsKeyword(Query, #Path2);
I had initially created a static class that does the following:
public static class Utils
{
public static bool ContainsKeyword(string query, string path)
{
var isQueryInFile = false;
var stringFromFile = GetStringFromFile(path);
List<Regex>regexList = GetRegexList(stringFromFile);
if(regexList!= null)
{
isQueryInFile = regexList.Any(pattern => pattern.IsMatch(query));
}
return isQueryInFile;
}
private static string GetStringFromFile(string path)
{
var words = String.Empty;
if(!string.IsNullOrEmpty(path))
{
try
{
using (StreamReader sr = File.OpenText(path))
{
words = sr.ReadToEnd().Replace(Environment.Newline, "");
}
}
catch { return words; }
}
return words;
}
private static List<Regex> GetRegexList(string words)
{
if(string.IsNullOrEmpty(words)) { return null; }
return words.Split(',').Select(w=> new Regex(#"\b" + Regex.Escape(w) + #'\b', RegexOptions.Compiled | RegexOptions.IgnoreCase)).ToList();
}
}
My problem is that I neither want to read from the file every time the ContainsKeyword static method is called nor do I want to create a new RegexList every time. Also, I cannot change the SQL script and I have to send the path to the file as an input parameter for the method call in the SQL script since the path might change in the future.
Is there a way to make sure I only read the contents from the input path only once, store them in a string, and use the string for the match with different input queries?
To read the content only once, saving in memory will probaby be needed. Memory capacity could be an issue.
public Dictionary<string, string> FileContentCache { get; set; } // make sure that gets initialized
public string GetFileContentCache(string path)
{
if (FileContentCache == null) FileContentCache = new Dictionary<string, string>();
if (FileContentCache.ContainsKey(path))
return FileContentCache[path];
var fileData = GetStringFromFile(path);
FileContentCache.Add(path, fileData);
return fileData;
}
I am using C# and AWSSDK v3 to upload files into an S3 bucket. The file is encrypted using ServerSideEncryptionCustomerMethod. I can upload the file, but if I check if the file exists using S3FileInfo().Exists, an error is thrown as a (400) Bad Request. However, if I comment out the lines that specify encryption in the upload routine, the S3FileInfo().Exists finds the file without throwing an error. What I am doing wrong? Or is there a different way to check if a file exists when it is encrypted?
Here is my upload routine:
public static string wfUpload(Stream pFileStream, string pBucketName, string pKeyName, string pCryptoKey) {
string retVal = "";
try {
using (var lS3Client = new AmazonS3Client()) {
Aes aesEncryption = Aes.Create();
aesEncryption.KeySize = 256;
aesEncryption.GenerateKey();
string lCryptoKey = Convert.ToBase64String(aesEncryption.Key);
PutObjectRequest request = new PutObjectRequest {
BucketName = pBucketName,
Key = pKeyName,
ServerSideEncryptionCustomerMethod = ServerSideEncryptionCustomerMethod.AES256,
ServerSideEncryptionCustomerProvidedKey = lCryptoKey,
};
request.InputStream = pFileStream;
PutObjectResponse response = lS3Client.PutObject(request);
retVal = lCryptoKey;
}
}
catch (AmazonS3Exception s3Exception) {
Console.WriteLine(s3Exception.Message,
s3Exception.InnerException);
throw (s3Exception);
}
catch (Exception e) {
throw (e);
}
return retVal;
}
And my routine to check if the file exists or not:
public static bool wfFileExists(String pBucketName, String pKeyName) {
bool retVal = false;
try {
using (var lS3Client = new AmazonS3Client()) {
if (new Amazon.S3.IO.S3FileInfo(lS3Client, pBucketName, pKeyName).Exists) {
retVal = true;
}
}
}
catch (AmazonS3Exception s3Exception) {
Console.WriteLine(s3Exception.Message,
s3Exception.InnerException);
throw (s3Exception);
}
catch (Exception e) {
throw (e);
}
return retVal;
}
Well, I think the class/method I was using is one of the high level APIs that doesn't support encryption. I changed my code to do a meta-data query to see if anything comes back. If it can't find the file it throws a "NotFound" ErrorCode in the s3Exception that I check for. Hopefully this helps someone else. If someone else suggests a better approach, I'd love to learn it too.
public static bool wfFileExists(String pBucketName, String pKeyName, String pCryptoKey) {
bool retVal = false;
try {
using (var lS3Client = new AmazonS3Client()) {
GetObjectMetadataRequest request = new GetObjectMetadataRequest {
BucketName = pBucketName,
Key = pKeyName,
ServerSideEncryptionCustomerMethod = ServerSideEncryptionCustomerMethod.AES256,
ServerSideEncryptionCustomerProvidedKey = pCryptoKey,
};
GetObjectMetadataResponse lMetaData = lS3Client.GetObjectMetadata(request);
// If an error is not thrown, we found the metadata.
retVal = true;
}
}
catch (AmazonS3Exception s3Exception) {
Console.WriteLine(s3Exception.Message,
s3Exception.InnerException);
if (s3Exception.ErrorCode != "NotFound") {
throw (s3Exception);
}
}
catch (Exception e) {
throw (e);
}
return retVal;
}
I'm looking to parse the WebCacheV01.dat file using C# to find the last file location for upload in an Internet browser.
%LocalAppData%\Microsoft\Windows\WebCache\WebCacheV01.dat
I using the Managed Esent nuget package.
Esent.Isam
Esent.Interop
When I try and run the below code it fails at:
Api.JetGetDatabaseFileInfo(filePath, out pageSize, JET_DbInfo.PageSize);
Or if I use
Api.JetSetSystemParameter(instance, JET_SESID.Nil, JET_param.CircularLog, 1, null);
at
Api.JetAttachDatabase(sesid, filePath, AttachDatabaseGrbit.ReadOnly);
I get the following error:
An unhandled exception of type
'Microsoft.Isam.Esent.Interop.EsentFileAccessDeniedException' occurred
in Esent.Interop.dll
Additional information: Cannot access file, the file is locked or in use
string localAppDataPath = Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData);
string filePathExtra = #"\Microsoft\Windows\WebCache\WebCacheV01.dat";
string filePath = string.Format("{0}{1}", localAppDataPath, filePathExtra);
JET_INSTANCE instance;
JET_SESID sesid;
JET_DBID dbid;
JET_TABLEID tableid;
String connect = "";
JET_SNP snp;
JET_SNT snt;
object data;
int numInstance = 0;
JET_INSTANCE_INFO [] instances;
int pageSize;
JET_COLUMNDEF columndef = new JET_COLUMNDEF();
JET_COLUMNID columnid;
Api.JetCreateInstance(out instance, "instance");
Api.JetGetDatabaseFileInfo(filePath, out pageSize, JET_DbInfo.PageSize);
Api.JetSetSystemParameter(JET_INSTANCE.Nil, JET_SESID.Nil, JET_param.DatabasePageSize, pageSize, null);
//Api.JetSetSystemParameter(instance, JET_SESID.Nil, JET_param.CircularLog, 1, null);
Api.JetInit(ref instance);
Api.JetBeginSession(instance, out sesid, null, null);
//Do stuff in db
Api.JetEndSession(sesid, EndSessionGrbit.None);
Api.JetTerm(instance);
Is it not possible to read this without making modifications?
Viewer
http://www.nirsoft.net/utils/ese_database_view.html
Python
https://jon.glass/attempts-to-parse-webcachev01-dat/
libesedb
impacket
Issue:
The file is probably in use.
Solution:
in order to free the locked file, please stop the Schedule Task -\Microsoft\Windows\Wininet\CacheTask.
The Code
public override IEnumerable<string> GetBrowsingHistoryUrls(FileInfo fileInfo)
{
var fileName = fileInfo.FullName;
var results = new List<string>();
try
{
int pageSize;
Api.JetGetDatabaseFileInfo(fileName, out pageSize, JET_DbInfo.PageSize);
SystemParameters.DatabasePageSize = pageSize;
using (var instance = new Instance("Browsing History"))
{
var param = new InstanceParameters(instance);
param.Recovery = false;
instance.Init();
using (var session = new Session(instance))
{
Api.JetAttachDatabase(session, fileName, AttachDatabaseGrbit.ReadOnly);
JET_DBID dbid;
Api.JetOpenDatabase(session, fileName, null, out dbid, OpenDatabaseGrbit.ReadOnly);
using (var tableContainers = new Table(session, dbid, "Containers", OpenTableGrbit.ReadOnly))
{
IDictionary<string, JET_COLUMNID> containerColumns = Api.GetColumnDictionary(session, tableContainers);
if (Api.TryMoveFirst(session, tableContainers))
{
do
{
var retrieveColumnAsInt32 = Api.RetrieveColumnAsInt32(session, tableContainers, columnIds["ContainerId"]);
if (retrieveColumnAsInt32 != null)
{
var containerId = (int)retrieveColumnAsInt32;
using (var table = new Table(session, dbid, "Container_" + containerId, OpenTableGrbit.ReadOnly))
{
var tableColumns = Api.GetColumnDictionary(session, table);
if (Api.TryMoveFirst(session, table))
{
do
{
var url = Api.RetrieveColumnAsString(
session,
table,
tableColumns["Url"],
Encoding.Unicode);
var downloadedFileName = Api.RetrieveColumnAsString(
session,
table,
columnIds2["Filename"]);
if(string.IsNullOrEmpty(downloadedFileName)) // check for download history only.
continue;
// Order by access Time to find the last uploaded file.
var accessedTime = Api.RetrieveColumnAsInt64(
session,
table,
columnIds2["AccessedTime"]);
var lastVisitTime = accessedTime.HasValue ? DateTime.FromFileTimeUtc(accessedTime.Value) : DateTime.MinValue;
results.Add(url);
}
while (Api.TryMoveNext(session, table.JetTableid));
}
}
}
} while (Api.TryMoveNext(session, tableContainers));
}
}
}
}
}
catch (Exception ex)
{
// log goes here....
}
return results;
}
Utils
Task Scheduler Wrapper
You can use Microsoft.Win32.TaskScheduler.TaskService Wrapper to stop it using c#, just add this Nuget package [nuget]:https://taskscheduler.codeplex.com/
Usage
public static FileInfo CopyLockedFileRtl(DirectoryInfo directory, FileInfo fileInfo, string remoteEndPoint)
{
FileInfo copiedFileInfo = null;
using (var ts = new TaskService(string.Format(#"\\{0}", remoteEndPoint)))
{
var task = ts.GetTask(#"\Microsoft\Windows\Wininet\CacheTask");
task.Stop();
task.Enabled = false;
var byteArray = FileHelper.ReadOnlyAllBytes(fileInfo);
var filePath = Path.Combine(directory.FullName, "unlockedfile.dat");
File.WriteAllBytes(filePath, byteArray);
copiedFileInfo = new FileInfo(filePath);
task.Enabled = true;
task.Run();
task.Dispose();
}
return copiedFileInfo;
}
I was not able to get Adam's answer to work. What worked for me was making a copy with AlphaVSS (a .NET class library that has a managed API for the Volume Shadow Copy Service). The file was in "Dirty Shutdown" state, so I additionally wrote this to handle the exception it threw when I opened it:
catch (EsentErrorException ex)
{ // Usually after the database is copied, it's in Dirty Shutdown state
// This can be verified by running "esentutl.exe /Mh WebCacheV01.dat"
logger.Info(ex.Message);
switch (ex.Error)
{
case JET_err.SecondaryIndexCorrupted:
logger.Info("Secondary Index Corrupted detected, exiting...");
Api.JetTerm2(instance, TermGrbit.Complete);
return false;
case JET_err.DatabaseDirtyShutdown:
logger.Info("Dirty shutdown detected, attempting to recover...");
try
{
Api.JetTerm2(instance, TermGrbit.Complete);
Process.Start("esentutl.exe", "/p /o " + newPath);
Thread.Sleep(5000);
Api.JetInit(ref instance);
Api.JetBeginSession(instance, out sessionId, null, null);
Api.JetAttachDatabase(sessionId, newPath, AttachDatabaseGrbit.None);
}
catch (Exception e2)
{
logger.Info("Could not recover database " + newPath + ", will try opening it one last time. If that doesn't work, try using other esentutl commands", e2);
}
break;
}
}
I'm thinking about using the 'Recent Items' folder as when you select a file to upload an entry is written here:
C:\Users\USER\AppData\Roaming\Microsoft\Windows\Recent
string recent = (Environment.GetFolderPath(Environment.SpecialFolder.Recent));
I am currently using Renci SSH.NET to upload files and folders to a Unix Server using SFTP, and creating directories using
sftp.CreateDirectory("//server/test/test2");
works perfectly, as long as the folder "test" already exists. If it doesn't, the CreateDirectory method fails, and this happens everytime when you try to create directories containing multiple levels.
Is there an elegant way to recursively generate all the directories in a string? I was assuming that the CreateDirectory method does that automatically.
There's no other way.
Just iterate directory levels, testing each level using SftpClient.GetAttributes and create the levels that do not exist.
static public void CreateDirectoryRecursively(this SftpClient client, string path)
{
string current = "";
if (path[0] == '/')
{
path = path.Substring(1);
}
while (!string.IsNullOrEmpty(path))
{
int p = path.IndexOf('/');
current += '/';
if (p >= 0)
{
current += path.Substring(0, p);
path = path.Substring(p + 1);
}
else
{
current += path;
path = "";
}
try
{
SftpFileAttributes attrs = client.GetAttributes(current);
if (!attrs.IsDirectory)
{
throw new Exception("not directory");
}
}
catch (SftpPathNotFoundException)
{
client.CreateDirectory(current);
}
}
}
A little improvement on the code provided by Martin Prikryl
Don't use Exceptions as a flow control mechanism. The better alternative here is to check if the current path exists first.
if (client.Exists(current))
{
SftpFileAttributes attrs = client.GetAttributes(current);
if (!attrs.IsDirectory)
{
throw new Exception("not directory");
}
}
else
{
client.CreateDirectory(current);
}
instead of the try catch construct
try
{
SftpFileAttributes attrs = client.GetAttributes(current);
if (!attrs.IsDirectory)
{
throw new Exception("not directory");
}
}
catch (SftpPathNotFoundException)
{
client.CreateDirectory(current);
}
Hi I found my answer quite straight forwared. Since I found this old post, I thought others might also stumble upon it. The accepted answer is not that good, so here is my take. It does not use any counting gimmicks, so I think it's a little more easy to understand.
public void CreateAllDirectories(SftpClient client, string path)
{
// Consistent forward slashes
path = path.Replace(#"\", "/");
foreach (string dir in path.Split('/'))
{
// Ignoring leading/ending/multiple slashes
if (!string.IsNullOrWhiteSpace(dir))
{
if(!client.Exists(dir))
{
client.CreateDirectory(dir);
}
client.ChangeDirectory(dir);
}
}
// Going back to default directory
client.ChangeDirectory("/");
}
FWIW, here's my rather simple take on it. The one requirement is that the server destination path is seperated by forward-slashes, as is the norm. I check for this before calling the function.
private void CreateServerDirectoryIfItDoesntExist(string serverDestinationPath, SftpClient sftpClient)
{
if (serverDestinationPath[0] == '/')
serverDestinationPath = serverDestinationPath.Substring(1);
string[] directories = serverDestinationPath.Split('/');
for (int i = 0; i < directories.Length; i++)
{
string dirName = string.Join("/", directories, 0, i + 1);
if (!sftpClient.Exists(dirName))
sftpClient.CreateDirectory(dirName);
}
}
HTH
A little modification on the accepted answer to use spans.
It's probably utterly pointless in this case, since the overhead of the sftp client is far greater than copying strings, but it can be useful in other similiar scenarios:
public static void EnsureDirectory(this SftpClient client, string path)
{
if (path.Length is 0)
return;
var curIndex = 0;
var todo = path.AsSpan();
if (todo[0] == '/' || todo[0] == '\\')
{
todo = todo.Slice(1);
curIndex++;
}
while (todo.Length > 0)
{
var endOfNextIndex = todo.IndexOf('/');
if (endOfNextIndex < 0)
endOfNextIndex = todo.IndexOf('\\');
string current;
if (endOfNextIndex >= 0)
{
curIndex += endOfNextIndex + 1;
current = path.Substring(0, curIndex);
todo = path.AsSpan().Slice(curIndex);
}
else
{
current = path;
todo = ReadOnlySpan<char>.Empty;
}
try
{
client.CreateDirectory(current);
}
catch (SshException ex) when (ex.Message == "Already exists.") { }
}
}
my approach is more sufficient and easier to read and maintain
public static void CreateDirectoryRecursively(this ISftpClient sftpClient, string path)
{
// Consistent forward slashes
var separators = new char[] { Path.AltDirectorySeparatorChar, Path.DirectorySeparatorChar };
string[] directories = path.Split(separators);
string currentDirectory = "";
for (int i = 1; i < directories.Length; i++)
{
currentDirectory = string.Join("/", currentDirectory, directories[i]);
if (!sftpClient.Exists(currentDirectory))
{
sftpClient.CreateDirectory(currentDirectory);
}
}
}
First, my disclaimer: I'm a parallel noob. I thought this would be an easy "embarrassingly parallel" problem to tackle, but it's thrown me for a loop.
I'm trying to download some photos in parallel from the web. The original photos are Hi-Res and take up quite a bit of space, so I'm going to compact them once they're downloaded.
Here's the code:
private static void DownloadPhotos(ISet<MyPhoto> photos)
{
List<MyPhoto> failed = new List<MyPhoto>();
DateTime now = DateTime.Now;
string folderDayOfYear = now.DayOfYear.ToString();
string folderYear = now.Year.ToString();
string imagesFolder = string.Format("{0}{1}\\{2}\\", ImagePath, folderYear, folderDayOfYear);
if (!Directory.Exists(imagesFolder))
{
Directory.CreateDirectory(imagesFolder);
}
Parallel.ForEach(photos, photo =>
{
if (!SavePhotoFile(photo.Url, photo.Duid + ".jpg", imagesFolder))
{
failed.Add(photo);
Console.WriteLine("adding to failed photos: {0} ", photo.Duid.ToString());
}
});
Console.WriteLine();
Console.WriteLine("failed photos count: {0}", failed.Count);
RemoveHiResPhotos(string.Format(#"{0}\{1}\{2}", ImagePath, folderYear, folderDayOfYear));
}
private static bool SavePhotoFile(string url, string fileName, string imagesFolder)
{
string fullFileName = imagesFolder + fileName;
string originalFileName = fileName.Replace(".jpg", "-original.jpg");
string fullOriginalFileName = imagesFolder + originalFileName;
if (!File.Exists(fullFileName))
{
using (WebClient webClient = new WebClient())
{
try
{
webClient.DownloadFile(url, fullOriginalFileName);
}
catch (Exception ex)
{
Console.WriteLine();
Console.WriteLine("failed to download photo: {0}", fileName);
return false;
}
}
CreateStandardResImage(fullOriginalFileName, fullOriginalFileName.Replace("-original.jpg", ".jpg"));
}
return true;
}
private static void CreateStandardResImage(string hiResFileName, string stdResFileName)
{
Image image = Image.FromFile(hiResFileName);
Image newImage = image.Resize(1024, 640);
newImage.SaveAs(hiResFileName, stdResFileName, 70, ImageFormat.Jpeg);
}
So here's where things confuse me: each of the photos hits the Catch{} block of the SavePhotoFile() method at the webClient.DownloadFile line. The error message is an exception occured during a WebClient request and the inner detail is "The process cannot access the file . . . -original.jpg because it is being used by another process."
If I wasn't confused enough by this error, I'm confused even more by what happens next. It turns out that if I just ignore the message and wait, the image will eventually download and be processed.
What's going on?
OK, so it appears in my focus on parallelism that I made a simple error: I assumed something about my data that wasn't true. Brianestey figured out the problem: Duid isn't unique. It's supposed to be unique, except for some missing code in the process to create the list.
The fix was to add this to the MyPhoto class
public override bool Equals(object obj)
{
if (obj is MyPhoto)
{
var objPhoto = obj as MyPhoto;
if (objPhoto.Duid == this.Duid)
return true;
}
return false;
}
public override int GetHashCode()
{
return this.Duid.GetHashCode();
}