I have some performance issues while trying to get specific files from Azure Storage.
We have the following folder tree inside Azure:
In the folder Week there is a file. I would like to get on one call precise file based on Location, Year and Week. Currently there is a working solution but it isn't efficient. It is looking recursively through folder tree.
Does anyone has a better solution ?
Related
I'm a little stuck, as for this moment i don't have any code only an idea.
I was thinking to make a program in asp.net to list files within a location.
The file will probably be releases for programs and mods for gaming and stuff.
But i have searched everywhere to get files in a location. I could use system.io.fileinfo to list files in a directory.
But how to I get a Directory not in the solution but somehwere on the hard drive or even external storage server? The location i prefer not to be hard coded but selected by the user? any toughts on how a user can select a 'location' ?
In a console application, you could be doing something like this :
Ask for the path from the user with a readline.
Check if the path is correct / exists. (if not you could create it or display an error, also it could be a path with a file, you need to check if the path is to a directory or to a file)
Do the display of the files inside of the directory as intended.
We need more precisions if you want a better answer.
I am developing a winform application that moves files between directories in a SAN. The app searches a list of files in directories with over 200,000 files (per directory) in the SAN's Directory, and then moves the found files to another directory.
For example I have this path:
\san\xxx\yyy\zzz
I perform the search in \zzz and then move the files to \yyy, but while I'm moving the files, another app is inserting more files into the xxx, yyy and zzz directories.
I don't want to impact the access or performance to other applications that use these directories.
The file's integrity is also a big concern, because when the application moves the files to \yyy directory another app uses those files.
All of these steps should run in parallel.
What methods could I use to achieve this?
How can I handle concurrency?
Thanks in advance.
Using the published .net sources, one can see how the move file operation works Here. Now if you follow the logic you'll find there's a call to KERNEL32. The MSDN documentation for this states
To perform this operation as a transacted operation, use the MoveFileTransacted function.
Based on that I'd say the move method is a safe/good way to move a file.
Searching through a large number of files is always going to take time. If you need to increase the speed of this operation, I'd suggest think out of side of the box to achieve this. For instance, I've seen cache buckets produced for both letters and dates. Using some sort of cache system, you could search for files in a quicker manner, say by asking the cache bucket for files starting with "a" for "these files like this starting with a" file, or files between these dates - etc.
Presumably, these files are used by something. For the purposes of this conversation, let's assume they are indexed in an SQL Database.
TABLE: Files
File ID Filename
-------------- ----------------------------
1 \\a\b\aaaaa.bin
2 \\a\b\bbbbb.bin
To move them to \\a\c\ without impacting access, you can execute the following:
Copy the file from \\a\b\aaaa.bin to \\a\c\aaaa.bin
Update the database to point to \\a\c\aaaa.bin
Once all open sessions have been closed for \\a\b\aaaa.bin, delete the old file
If the files are on the same volume, you could also use a hardlink instead of a copy.
So I'm PUTing a file onto one drive, my URI looks like
https://apis.live.net/v5.0/me/skydrive/files/?access_token=12345
however, if I try and change the folder to, say, Documents:
https://apis.live.net/v5.0/me/skydrive/files/Documents?access_token=12345
or
https://apis.live.net/v5.0/me/skydrive/Documents/files?access_token=12345
I'll get a HTTP 400 (bad request error).
One "workaround" could be to query the top level directory of the onedrive, done by reading the JSON response from the first URI about, and then suck out the upload_location for a desired folder, however I feel this is a bit long winded. From the Upload a File section in
https://msdn.microsoft.com/en-us/library/dn659726.aspx
I got the impression I should be able to specify a friendly folder path.
I'm positive it's a really simple answer, but I've wasted long enough on it and hopefully this will serve as a resource for those having the same issue (haven't been able to find elsewhere except for an unanswered question here http://answers.microsoft.com/en-us/onedrive/forum/sdfiles-sdupload/how-to-upload-a-file-to-a-specific-folder-created/0d57e43c-a3ec-42fc-b3b7-f15a17b2836f)
Thanks
Based on your examples I think you just need to change Documents -> my_documents and then the scenario will work like you'd want. Here is the relative excerpt from the documentation.
Use friendly names to access certain OneDrive folders
To access certain OneDrive folders, you can use friendly names instead of folder IDs. Use the following friendly names to access these corresponding folders in the OneDrive UI:
USER_ID/skydrive/camera_roll represents the OneDrive camera roll folder.
USER_ID/skydrive/my_documents represents the Documents folder.
USER_ID/skydrive/my_photos represents the Pictures folder.
USER_ID/skydrive/public_documents represents the Public folder.
In each case, replace USER_ID with either me for the signed-in user or a user ID for any other consenting user.
For example, to read the Documents folder's properties, use a REST API call to the folder's ID, like this.
GET https://apis.live.net/v5.0/folder.a6b2a7e8f2515e5e.A6B2A7E8F2515E5E!110?access_token=ACCESS_TOKEN
Try the API's listed in the following link:
https://dev.onedrive.com/README.htm#
These work fine for me.
I'm having some trouble trying to get a list of files stored within a folder that's stored within my Silverlight application client project. I'm trying to collect a list of file names of images to dynamically create URIs based on the files within that folder. All I need is a list of the file names.
I've tried:
App.GetResourceStream(new Uri(PathName));
and this works fine for getting files I know the name of. But I haven't found a way of returning a list of files yet.
My project is called TestWebsite
The the files would be stored within "TestWebsite\Images\ConceptArt\"
and all the files will be of the same image format e.i. '.jpg'
I have also tried this:
var result = Directory.EnumerateFiles(#"C:\Users\UserName\Pictures\ConceptArt\", "*.JPG");
but this needs access which seems more trouble than it's worth.
Your app is SL which is Browser based, so if you try to do things outside the SL sandbox, you are going to have to deal with Security. That is the nature of SL. If you could use local storage, then you wouldn't have the same issue. Or if your app is putting the files there, keep a list in local storage.
Greg
If I have a directory structure like so on a web service that sends me a list of files and folders in the directory I request. How would you go about iterating through the entire structure. There is no recursive search, so each directory has to be pulled one at a time. I can get the directories content I request but it doesnt list any sub-directories information. I was trying to think of a way to do this with a for loop or for each loop, but I havent been able to come up with anything. I didnt write the web service so I cant include recursive directory searching in it.
Pictures
photo1.png
photo2.png
TestFolder1
April.png
MyPictures
ProfilePic.png
TestFolder2
2012
August
Photos
photo3.png
photo4.png
Your probably looking for a Tree Traversal Algorithm:
Traversing a tree of objects in c#
Tree traversal algorithm for directory structures with a lot of files
How to: Iterate Through a Directory Tree