Need to create a service application that monitor all files being downloaded. I was thinking about using the file watcher class but some end users can change the download directory. Any idea would be great.
Monitoring file system is no indicative of download activity, as you already figured out. Browsers do not expose a cross-platform API to report download events. Browsers plugins do, but that requires the plugin to be loaded, enabled and you'll need separate plugins for each browser out there. So the only sensible approach is a filter driver. In fact the OS exposes a handy helper layer specifically designed for this, the Windows Filtering Platform. Of course, C++ and networking expertise is required. Detecting downloads from all traffic is left as an exercise to the reader.
If you want to monitor files downloaded on computers in an office environment, your best bet would be to install a proxy server.
Related
I am attempting to connect to my Google Drive using C# and the Google Drive API and then map that as a network or local drive. There are other programs I know that do this like NetDrive (which is extremely useful and robust), but I am looking to create something on my own. I have created a project in the developer console and have been able to connect to Drive using my application and do various read and upload operations, so I know that particular portion is ok. Access and permissions all seem to be set. I just have no idea where to start when it comes to mapping that storage as a usable drive in Windows. Any advice would be most helpful, thank you very much!
There are two basic components for implementing a NetDrive/WebDrive type of solution. What you are looking at is the creation of an Installable File System and Network Provider.
The Network Provider, or NP, is the user mode component that handles the Network layers, including the mapping and unmapping of the drive letter, along with lots of other fairly complicated UNC/Network stuff. To get an idea of what you are in for, check out the Win32 WNET*() API; you will need to implement all of the WNet() calls specifically for your IFS and 'network'.
When you are done, you'll probably have the ability to to do a "net use \MyWebDrive\" in DOS and Map Network Drive in Windows Explorer. You might also be able to use Windows Explorer to enum the contents of the remote file system.
However, now you need to make sure that all third party applications can access your network drive...to do that, you want to implement the Win32 File System API, such as CreateFile, Read(), Write(), CloseHandle(), FindFirst(), etc.
To do this, you can write an Installable File System Driver, FSD, to handle all I/O calls from User mode applications wanting to read/write to the files on that mapped network drive. This will most likely be a Kernel Mode application...a signed/certified file system device driver....probably written in old-school C and maybe even utilizing TDI depending on how you want to do your network IO.
Microsoft is becoming much more strict about installing 3rd party kernel mode drivers and network providers. The WebDrive file system driver is now securely signed using a Microsoft based TLS certificate and our Network Provider has been registered with the Microsoft Windows SDK team as a legitimate Network Provider for the Windows platform.
Once you get these pieces in place, you'll then want to think about Caching. Direct I/O through your NP/FSD over the wire to Google is not practical, so you'll need an intermediate caching system on your local drive. There are lots of ways to do that, too many to go into here. However, just keep in mind that you may have multiple user mode applications reading and writing to your network drive simultaneously (or one app like WinWord which opens multiple file handles), and you'll need to be able to handle all those requests with proper locking and ACLs, and then map those changes and access rules to the remote server.
Don't lose faith...what you are looking to do is possible as WebDrive and NetDrive have shown, but it's not really a project that can be knocked out in a few weekends. I'm not sure about the author of NetDrive, but we've been developing WebDrive full time since 1997. It seems that every Windows Patch changes something and every new version of Adobe/Office/XYZ does something quirky with IO calls that makes us pull our hair out.
Note: There's also another way to implement this beast which may get around the FSD, it's the DropBox strategy. Using a temporary folder on your local hard drive, leverage Directory Change Notifications in a User Mode application to monitor file changes in the folder and dynamically synchronize the changes to the remote end. GoogleDrive and a lot of the other online storage companies do it this way because it's quick-&-easy; however, if many changes occur in a short period, a Change Notification could get lost in Windows Messaging and data might get trashed.
I realize this is a lot to digest, but it's doable...it's cool stuff; good luck!
I suggest that before you start coding, you take time to thoroughly understand Google Drive and map its capabilities to/from Windows. Some sample points of impedance:-
folders in Drive aren't folders at all
A file in Drive = the metadata, content is optional
Drive has a lot of metadata that doesn't map to NTFS (eg. properties)
Will applicable files be converted to Google Docs, or stored as is
How will you map revisions
Permissions
There are almost certainly more, this is just off the top of my head. Your app needs to make decisions regarding all of these aspects. Generally, Drive offers more capabilities than NTFS, so provided you are simply using it as a backup repository, you should be OK.
I would like to create a image file upload tool. The tool when started up and would ask the user what directory they would like to monitor and upload files from. In the background the tool would then post securely to a web service of some sort.
Constraints solution should work on Mac and Windows computers. I would like to make the upload process pretty automatic with little user intervention.
Basically the question I have is what would you use to create application that either runs in the background or foreground to basically continuously uploads files to a web server?
I have a couple of thoughts:
HTML5 using web workers (concerns accessing local files not sure the best way to do that)
Silverlight (my guess is I would have to sign the application)
Java web start application (Signing application)
Flash (scared of this idea)
I can do the Java web start application easy enough. Both HTML5 and Silverlight are things that I would like to know. I really don't care about flash at this point.
In Java, you can use the JNotify library: http://jnotify.sf.net . It works on Windows, OS X, and Linux.
I don't know about C# or HTML5. HTML5 (last I checked circa July 2011) doesn't have a filesystem API. In reality you need an app that can be distributed so it can run in the background, so Silverlight is out of the question. If I am misinterpreting your problem, please tell me.
I would try to go with a signed Java applet if you can foresee the need for downloading as well. It is difficult to find a good download solution other then a signed Java applet. A signed Java applet let's you push things to the users filesystem allowing for upload as well as download.
Is it possible to write a filesystem for Windows in pure usermode, or more specifically purely in managed code? I am thinking of something very similar to GMAILFS. Excluding what it is doing under the covers (GMAIL, Amazon, etc..) the main goal would be to provide a drive letter and support all of the basic file operations, and possibly even adding my own structures for storing metadata, etc..
Windows provides several approaches to building a user-mode file system for different purposes, depending on your storage location and features that you need to support. Two of them, Projected File System API and Cloud Files API were recently provided as part of the Windows 10 updates.
Windows Projected File System API
Projected File System API is designed to represent some hierarchical data, such as for example Windows Registry, in the form of a file system.
Unlike Cloud Files (see below) it does not provide any information about file status and hides the fact that this is not the “real” file system. Example.
Windows Cloud Sync Engine API
Cloud Sync Engine API (Cloud Files API, Cloud Filter API) is used in OneDrive on Windows 10 under the hood. It provides folder content loading during the first request, on-demand files content loading in several different modes, and offline files support. It integrates directly into Windows File Manager and Windows Notification Center and provides file status (offline, in-sync, conflict, pinned) and file content transfer progress.
The Cloud Files API runs under regular user permissions and does not require admin privileges for file system mounting or any API calls. Example.
Windows Shell Namespace Extensions API
While Shell Namespace Extension is not a real file system, in many cases you will use it to extend the functionality of the Projected File System and Cloud Files API. For example, you will it to add custom commands to context menus in Windows File Manager as well as you can create nodes that look and behave like a real file system (again, applications would not be able to read or write to such nodes, this is just a user interface).
Cloud Files API is using a namespace extension to show your sync root at the top level in Windows File Manager.
It's difficult. I'd take a look at some projects which have done some of the hard work for you, e.g. Dokan.
Yes. It's possible and has been successfully done for the ext2 filesystem.
Note that you will need to write your own driver which will require Microsoft signing to be run on some OSes.
Sure, you can abstract the regular file operations and have them running in the cloud (see Google Apps, Amazon S3, Microsoft Azure etc.). But if you'd like to talk to local devices - including the local HD - you'll have to use system APIs and those use drivers (system/kernel mode).
As long as all you want is a storage service -no problem. If you want a real OS, you'll need to talk to real hardware and that means drivers.
Just as a reference - our Callback File System is a maintained and supported solution for creation of filesystems in user-mode.
I've got an app that publishes and updates from an http update location (I publish to the ftp site of the host, and update from the website).
The publish.htm page is very handy as I can install the app on any machine, anywhere without needing media. The problem is, so can anyone else. How can I secure the update location so that only authorized users can install the app without buggering the auto-update feature of clickonce?
Is this an internal application? If so you could just exclude the publish.htm page from your deployment. Then to install you would then just use the application manifest link http://yoursite/YourApplication.application which should kick the install off, this would not affect automatic updates. This may be just enough obfuscation to for your purposes.
Failing that you can dynamically generate the application manifest using a little bit of asp.net which would only produce the manifest for the users you want. The other benefit this has is that you can isolate a small group of users when rolling out a new version.
Just a thought.
If you're still transferring over HTTP, it's as easy as running a traffic sniffing program like Wireshark to see where the application is downloading from. To evade this you'll need to make sure to transfer over HTTPS, on top of whatever obfuscation you do to hide the update location.
Is there any way to create a virtual drive in "(My) Computer" and manipulate it, somewhat like JungleDisk does it?
It probably does something like:
override OnRead(object sender, Event e) {
ShowFilesFromAmazon();
}
Are there any API:s for this? Maybe to write to an XML-file or a database, instead of a real drive.
The Dokan Library seems to be the answer that mostly corresponds with my question, even though System.IO.IsolatedStorage seems to be the most standardized and most Microsoft-environment adapted.
Depending on what type of virtual drive you wish to build, here are some new OS API recently introduced in Windows, macOS and iOS.
Some of the below API is available as managed .NET code on Windows but many are a native Windows / macOS / iOS API. Even though, I was able to consume many of the below API in .NET and Xamarin applications and build entire Virtual Drive in C# for Windows, macOS and iOS.
For Remote Cloud Storage
On Windows. Windows 10 provides Cloud Sync Engine API for creating virtual drives that publish data from a remote location. It is also known under the “Cloud Filter API” name or “Windows Cloud Provider”. Here are its major features:
On-demand folders listing. Folder listing is made only when the first requested by the client application to the file system is made. File content is not downloaded, but all file properties including file size are available on the client via regular files API.
On-demand file content loading. File content can be downloaded in several modes (progressive, streaming mode, allow background download, etc) and made available to OS when application makes first file content reading request.
Offline files support. Files can be edited in the offline mode, pinned/unpinned and synched to/from the server.
Windows shell integration. Windows File Manager shows file status (modified, in-sync, conflict) and file download progress.
Metadata and properties support. Custom columns can be displayed in Windows File Manager as well as some binary metadata can be associated with each file and folder.
On macOS and iOS. MacOS Big Sur and iOS 11+ provides similar API called File Provider API. Its features are similar to what Windows API provides:
On-demand folders listing.
On-demand files content loading.
Offline files support.
File Manager Integration. In macOS Finder and iOS Files application you can can show file status (in the cloud, local).
I am not sure currently if files/folders and can show custom columns in macOS Finder and store any metadata.
For High-Speed Local Storage
On Windows. Windows provides ProjFS API. Its main difference from the Cloud Sync Engine API and macOS/iOS File Provider API is that it hides the fact that it is a remote storage. It does not provide any indication of the file status, download progress, ets. The documentation says it is intended for “projecting” hierarchical data in the form of file system.
You can use the Dokan library to create a virtual drive. There is a .Net wrapper for interfacing with C#.
Yes, use the classes in System.IO.IsolatedStorage
The contents of My Computer can include Shell Namespace Extensions. These COM objects run inside the main Explorer process, as do many other shell extensions. Using C# for such extensions is a bad idea, since your extension cannot control which CLR version Explorer.exe can use. And Microsoft allows only one CLR per process.