Develop Reference

extracting a file without letting the user access it

I was always pretty impressed by those programs that you could install by executing one installer file, which would then extract all the other required files to run the actual program.
And even now im still wondering how you would code a program that extracts files that are literally still inside the program ( so not in some kind of zip) , i've seen tons of installers for games who have this. I need this cause I want to extract a file on the right moment without giving the person who uses the program the ability to delete the file before its extracted, this may seem vague, but I hope i've informed you enough.

I'm just going to say that building an installer is difficult.
I'd recommend using NSIS: http://nsis.sourceforge.net/Main_Page
As for creating a file the user can't access, create a temp file with the correct read/write permissions, extract the data to the temp file, then copy the file where it needs to go.
Extract happens without the user interfering, and copy protection is handled by the OS.

What about changing the build action for the file you want to hide to Embedded Resource, or something like that that compiles the file inside the dll/exe?

Executable program is just file, So you can append any data at you executables. (works for my c++ compiled program)
$ cat executables some_blob_data > new_executables
Since argv[0] of main() is name of your file, you can use this to acess data in this file itself (works for c or c++ and likley for other languages to)

A really simple way to do this is to use your archive tool or one of the dozens of already made installers. WinRar, WinZip and most others allow the creation of self extracting exe files. I know you've said that is not what you want but if you make sure to make it auto exec your installer app and remove all of the temporary files when you're done it really can be very fool proof and professional looking. Obviously, the various installer applications are able to do what you're wanting.
If you really want to do this yourself the easy solution is going to most likely be dependant on your IDE software and language. The generic answer is that you'll need a compression library which can accept a stream as input. Then you'll want to make your actual files into resources inside your installer app. At that point it's just a matter of telling the compression library to read from a stream which is pointed at the resource. How that is done varies greatly from language to language.

How to read an non-shared file

I am making a data backup program. I need to copy files from one folder to another, while some files are still being used by a running process. It's OK when the running process shares file access. But I will get exception if files are not shared. (FileShare.None)
I am wondering if there is any way to read a non-shared file in C#.
Thanks,

C# under the hood uses the file access operations your operating system provides, including its file sharing behavior. So I'm afraid C# cannot go beyond this.

There is a technique if you are using Vista or later. You can use the Volume Shadow Copy feature. However, getting to work in C# will be tedious, and all that I can provide for help are references. However, if you wanted to administrate a solution instead of developing one, you could use Windows Backup to copy the files on a schedule.
Links:
Shadow Copy Explained
Shadow Copy SDK
Video Presentation on Shadow Copy

Sync services like Dropbox, theory behind file indexing?

I have realised that by using the Amazon S3 service directly, I can save myself a lot of money. Instead of buying a client like GoodSync or Jungle Disk I thought it would be interesting to create my own Windows syncing application, which would sync my files to S3.
I have discovered that I can use FileSystemWatcher to monitor for changes to files and directories, but I am looking for the theory behind how other services like Dropbox index their files. Things like comparing the file size of a file with the size recorded in an index somewhere on the client PC, then using this information to determine whether to sync or not.
I am using C# and references to different libraries or code samples I could use would be helpful, but I am mainly looking for the best way to index files and for someone to point me in the right direction.
Thanks

I've went down this path myself. In fact, now that Mozy dropped their unlimited plan and Carbonite chooses to NOT backup certain files...like 3GP files and *.dat files unless you routinely go in and manually add them, I am very disgruntled with online backups.
But your question was on syncing. Dropbox does it the best. But it's expensive. But I'm not sure S3 would be any cheaper.
Anyway, you will have a lot of hurdles. In my experiences, the problems I ran into are:
1) Propagating deletes
2) FileSystemWatcher simply missing events such as rapidly adding files to a folder then deleting them
3) etc..
Now some ideas on how I would tackle this again:
1) Keep a small SQLite db for files names/path locally
2) Copy files to a tmp directory before sending to S3.
3) On file changes/updates/deletions/etc store that meta information in SQLite
Anyway just some ideas.

How To Store Files In An EXE

Alright, so I'm working on programming my own installer in C#, and what I'd like to do is something along the lines of put the files in the .exe, so I can do
File.Copy(file, filedir);
Or, if this isn't possible, is there another way of doing what I am attempting to do?

I wouldn't code my own installer, but if you truely want to embed files into your assembly you could use strongly typed resources. In the properties dialog of your project open up the "Resources" tab and then add your file. You'll then be able to get the file using:
ProjectNamespace.Properties.Resources.MyFile
Then you'll be able to write the embedded resource to disk using:
System.IO.File.WriteAllBytes(#"C:\MyFile.bin", ProjectNamespace.Properties.Resources.MyFile);

Honestly, I would suggest you NOT create your own installer. There are many many issues with creating installers. Even the big installer makers don't make their own actual installers anymore, they just create custom MSI packages.
Use Mirosoft Installer (MSI). It's the right thing to do. Make your own custom front-end for it, but don't recreate the already very complex wheel that exists.
UPDATE: If you're just doing this for learning, then I would shy away from thinking of it as "an installer". You might be tempted to take your "research" and use it someday, and frankly, that's how we end up with so many problems when new versions of Windows come out. People create their own wheels with assumptions that aren't valid.
What you're really trying to do is called "packaging", and you really have to become intimately familiar with the Executable PE format, because you're talking about changing the structure of the PE image on disk.
You can simulate it, to a point, with putting files in resources, but that's not really what installers, or self-extractors do.
Here's a link to Self-Extractor tutorial, but it's not in C#.
I don't know enough about the .NET PE requirements to know if you can do this in with a managed code executable or not.
UPDATE2: This is probably more of what you're looking for, it embeds files in the resource, but as I said, it's not really the way professional installers or self-extractors do it. I think there are various limitations on what you can embed as resources. But here's the like to a Self-Extractor Demo written in C#.

I'm guessing here, but if you are trying to store resources in your application before compilation, you can in the Project Explorer, right click a file you would like to add, chose properties and change the type to Embedded Resource.
You can then access the embedded resources later by using the instructions from this KB:
http://support.microsoft.com/kb/319292

in case you simply want to store multiple files in a single file storage (and extract files from there, interact etc.) you might also want to check out NFileStorage, a .net file storage. written in 100% .NET C# with all sources included. It also comes with a command line interpreter that allows interaction from the command line.

Searching directories for tons of files?

I'm using MSVE, and I have my own tiles I'm displaying in layers on top. Problem is, there's a ton of them, and they're on a network server. In certain directories, there are something on the order of 30,000+ files. Initially I called Directory.GetFiles, but once I started testing in a pseudo-real environment, it timed out.
What's the best way to programatically list, and iterate through, this many files?
Edit: My coworker suggested using the MS indexing service. Has anyone tried this approach, and (how) has it worked?

I've worked on a SAN system in the past with telephony audio recordings which had issues with numbers of files in a single folder - that system became unusable somewhere near 5,000 (on Windows 2000 Advanced Server with an application in C#.Net 1.1)- the only sensible solution that we came up with was to change the folder structure so that there were a more reasonable number of files. Interestingly Explorer would also time out!
The convention we came up with was a structure that broke the structure up in years, months and days - but that will depend upon your system and whether you can control the directory structure...

Definitely split them up. That said, stay as far away from the Indexing Service as you can.

None. .NET relies on underlying Windows API calls that really, really hate that amount of files themselves.
As Ronnie says: split them up.

You could use DOS?
DIR /s/b > Files.txt

You could also look at either indexing the files yourself, or getting a third part app like google desktop or copernic to do it and then interface with their index. I know copernic has an API that you can use to search for any file in their index and it also supports mapping network drives.