I'm making an XNA game, which uses a lot (currently ~2800) of small resource files. It has become a problem to move them around from place to place unarchived, so I thought maybe I could just zip them and make the game unzip them automatically, into memory, preferably. I don't need the writing capability yet, right now only reading.
Is there an easy way to unzip a folder into memory and access those files just like, or as simple as the regular files on disk?
I've been reading some similar questions and I see many people say that the OS (Windows in my case) can handle file caching better than a ram drive. I'm just going for unzipping and reading files for now, but in future I might need to modify or create new files, and I'd like it to be quick and seamless for the user. Maybe I should take a different approach at solving my current problem, taking in account my future goal?
I haven't personally tried this but if you want to be able to zip/unzip in memory, you could just use a MemoryStream and pass that into a library (eg https://github.com/icsharpcode/SharpZipLib). Things you'll probably need to remember, are you just moving a bottleneck to a different bottleneck?
You could also try something like the approach with Sprites in HTML. You combine all your Zip's into 1 with an Index to where in the file they are. Then you move your FileStream.Position to the location for the resource you are looking for, read (the amount you need) then do what you need with it. You'd need to make sure that if you rebuild any you make something rebuild all your combining indexes etc. Then you just would be copying 1 file around, it just so happens that inside that file you have ~2800 smaller segments of interest.
Related
I am trying to figure out how to store data that can be easily/heavily edited.
Reading data from a big single file isn't really a problem. The problem starts when I need to make changes to that file.
Let's say I have a bit log file which always appends a string to the file. The Filesystem needs to recreate the whole file since it has changed. And the bigger the File the heavier the performance cost.
What I could do is simply create a new file for each log. Creating, removing and editing would be more efficient. Until I would like to copy all these files lets say on a new SSD.
Reading directories and copying thousand of files, even if they are small, hits hard on performance.
So maybe bundle all files into a single file/archive?
But then AFAIK archive like .zip ... also needs to be recreated when something changed.
Is there a good or maybe even simple solution to this?
How does a single file database like SQlite handle this?
Mention: I am using C#
I am quite new in game development with Unity and was wondering how the "export game" function of unity works. I not yet used this function in unity, but I've read that it will generate some .exe file from your complete game. I also read that it will create a "data" folder or something like that.
My question is: What exactly is stored in this "data" folder? And how can I write logic to save my own files (e.g. files which contain save states, settings, configurations, etc.) in some file inside this directory (which is then shipped with the complete game / created in the local game directory after the user e.g. saved his game the first time? Can i e.g. save those files in a relative path (e.g. ./MyGame/data/savegames)?
And which types of files can I create? Text / Binary? Or can I even use some relational Database (some small one like HSQLDB)?
And how are things like models, sounds, animations and other assets treated? Are they all packaged within the .exe file which is my complete game? Or do i have some seperate folders with the shipped game for them?
Thank you!
The data folder (named the same as the exe file, but _Data on the end instead of .exe, which can be safely renamed to just Data) contains all of the dlls that actually run the game (even a blank Unity project will have them! The unity engine itself compiles to several dlls) as well as any Resources you might have (tip: stop using resources and use Asset Bundles instead).
Omitting this folder would be very bad indeed!
As for reading/writing other data from the hard disk (which is not possible on all platforms--looking at you web deployment) I would recommend using your own folder, eg. RuntimeData which could contain external audio, image, or video files as well as mutable data such as save games or screenshots. Pretty much anything you'd be ok with your users modifying without seriously breaking stuff (modding is "in" these days).
As for the types of file: well, that's up to you, really! Creating text files (of any extension: xml, html, dat, qqq...) is very easy. Images tend to be done through a 3rd party script (do you really want to write your own JPG converter? Video, same thing). You can also create binary files following a format of your own choosing. The only difficulty is writing the serializer and deserializer for the data, which would scale in difficulty as the complexity of the data scales.
You have full file system access* so you can realistically read or write anywhere. This is C# we're talking about. But with great power comes great responsibility.
*Note: Mobile devices heavily frown on that sort of thing and will deny access to folders outside the one explicitly given to that application.
I want to rename large files while using c#.
If I have a couple of large files and if I use .IO.move function
my files will be copied with the right name and the old ones will be destroyed.
that will take a veryyyy long time with large files
I couldnt find a good solution.
anyone an idea fitting large files?
Whatever solution you choose - if you move your file between different logical/physical disks - you cannot do anything with it. It takes some time to move the data.
Have you tested your assumption? I did, and I found that if you use System.IO.File.Move, and the target location is on the same physical logical disk as the source, the file is just renamed. It doesn't take a long time.
You can create a new hardlink, then delete the original. This will only affect filesystem metadata, and not copy the file around.
I implemented a RAMDisk into my C# application, and everything is going great, except I need to back up the contents regularly due to it being volatile. I have been battling with AlphaVSS for Shadow Copy backups for a week, then someone informed me that VSS does not work on a RAMDisk.
The contents that are located on the RAMDisk (world files for Minecraft) are very small, but there can be hundreds of them. The majority of them are .dat files only a few hundred bytes in size, and there is other files that are 2-8MB each.
I posted about this yesterday Here, and the solution that was suggested was to use a FileStream, and save the data out of it. I just read that this is a horrible idea for binary data on another Stack Overflow question, so I am looking for a better approach to backup all of these little files, some of which might be in use.
I suggest you first zip all the small files together, then back them up to a location.
ref:
zip library: http://www.icsharpcode.net/opensource/sharpziplib/
use System.IO.File.Copy to copy the zip packed.
Given the path of a string i want to wipe out the contents of a file. The natural way I thought (which maybe incorrect) was to open a FileStream to the file and write gibberish (random data perhaps taken from a RNGCryptoServiceProvider) to it. And then perhaps do this several times and then delete the file.
My problem is that while this may look logically correct, i read up on another blog that Windows might actually choose to write the file to a different place in the hard disk.
Is that the case in Windows Mobile? Will this actually be a problem? Does this writing to a different location in the hard disk apply to even flash based (SD etc) cards ?
I've not personally done this, but you will probably need to use the low-level FLASH driver IOCTLs to do this correctly.
http://msdn.microsoft.com/en-us/library/aa927166.aspx
I think IOCTL_FMD_RAW_WRITE_BLOCKS looks particularly useful.
-PaulH
Another possibility that may work would be to erase the file normally, then use the defragment APIs to wipe ALL of the freespace on your flash. Since you're wiping everything, it won't be necessary to know exactly where on the disk your file was. But, this will wear out your flash drive more quickly. The C# method is detailed in this blog post: http://blogs.msdn.com/b/jeffrey_wall/archive/2004/09/13/229137.aspx