I am working on a project for visualizing large specialized datasets.
A "Project" consists of two large datasets (that are imported from XLS or CSV) and several other settings and smaller tables. We were planning to store those tables in SQLite (for ease of access).
We would like to allow the users to save a "Project" so they do not need to load the CSV and XLS files again. We would also like to prevent users who get the project file later from tampering with the data. We want managers to be able to create a "Project" file and pass it to other employees (so they can use the visualized data in presentations, but they can not modify it). The program already has the access control built in.
What suggestions do people have about creating this type of "Project" file, preferably based around a SQLite database.
Note: I am asking for advice here because it is impossible to Google for C# and Project. You get Visual Studio how-to results.
Clarification: Imagine this a if you opened Microsoft Word (or OpenOffice Write if your prefer). You can create open and save a "project". You can send a "project" to someone else who has the program. I would like the "project" file not to be usable in a different SQLite client (for data security).
I would not say there is a any kind of template or common structure for those kind of things. Your structure is already good by itself. You may have not only data information, but also meta-information inside database file, specifying (say) user name, data last changed, comments, whatever to make your project/doc file more comprehencive and full of information, keeping constant attention on memory footprint, cause it would be very nice to have a possibility to pass that file to someone by key-drive or send by e-mail.
Good luck.
Could you not create a table that has a series of BLOB data types (for documents) and a read_only boolean field. If you don't know how many docs there will be you need to enter another table called document that links to the project table!
Related
One of the many things that SharePoint does extremely well is that when you have versioning enabled for files uploaded to a Document Library, every time you save changes to a file it only saves the difference from the previous version of the file to the Content Database but NOT the whole file again.
I am trying to duplicate that same behavior with standard C# code on either a File System folder in Windows or a SQL Database blob field. Does anyone have any idea or pointers on how SharePoint accomplishes this and how it can be done outside of SharePoint?
SharePoint uses a technique called data "shredding" to contain each change to a given file. Unfortunately, I don't think you will find enough technical details to truly reproduce what they are doing, but you might be able to devise a reasonable approximation using your own design.
When shredded, the data associated with a file such as Document.docx is distributed across a set of BLOBs associated with the file. The independent BLOBS are each assigned a unique ID (offset) to enable reconstruction in the correct order when requested by a user.
Each document "shred" is stored in a SQL database table named DocStreams. Each BLOB contains a numerical Id representative of the source BLOB when coalesced. When a client updates a file, only the shredded BLOB that corresponds to the change is updated with the update occurring on the database server as opposed to the Web server.
For more details on Shredding see
http://download.microsoft.com/download/9/6/6/9661DAC2-393D-445A-BDC1-E60743B1231E/Shredded%20Storage%20in%20SharePoint%202013.pdf
https://jeremythake.com/the-truth-behind-shredded-storage-in-sharepoint-2013-a84ec047f28e
https://www.c-sharpcorner.com/UploadFile/91b369/shredded-storage-in-sharepoint-2013/
I have to load huge amount of data, pre-process it, share it among few users and finally gather updates back from users.
This is what I did in my previous project -
Created an excel add-in using C++. Loaded the data in memory using the add-in code and processed it. For each type of data I have sent the processed data to a sheet and saved a new excel file. That way, if I have three types of data, I have created three new excel workbooks. My users then opened those new workbooks, made their changes and dropped a text file that contains their changes (through a button). The main excel keeps polling for those updates (text files) and loads them as soon as they are found. That's the way I get the updates back from my users.
I am not a fan of what I did in my previous project, it produces too many temporary files (of course I can delete those). In my current project I want to use C# VSTO Workbook so I can have more control over excel. I was hoping once I load the data, I will ask my users to open the same excel in Read-Only mode and they will make changes. While testing this, I realized user's excel (opened in read-only) mode does not see the loaded data. And their changes do not update the data held in memory. This probably means I have no idea what I am doing.
Do you guys have any idea how to achieve this? I will really appreciate any help/hint.
Excel supports so-called "co-authoring" mode, when many people can edit the same document at the same time. But there is might be a catch: afaik, you need a Share Point/Office Online server/OneDrive Business to support this scenario (you need a non-free office document server product).
Using VSTO, you can do just the same you have done with C++ add-in, but in C# (means, the set of capabilities is 1:1 - it basically just wraps C++ COM Excel API for .NET)
But for online version of Excel, there may be yet another alternative - javascript addins (now that's called "Office Addins", afaik). But I doubt you'd want to process your "huge amounts of data" with javascript.
So I would say, there is a good rule: Don't fix something that isn't broken :)
If the problem is the number of temporary files, these files is not the only option to transfer data between applications. You know, you can connect two applications directly (so that they can exchange data with messages/updates). Use network, Luke :)
Of course if your 3 users live on 3 deserted islands, totally disconnected from anything, exchanging with text files on USB stick may still be the only viable option...
I think the "web" solution could be: store your file in some "co-authoring"-capable service (sharepoint, google shees, onedrive, officeonline, whatever). Make some web job to update that file in that storage automatically. Just like a "fourth" user would do.
I have an old application made with Visual Basic that can upload and download documents as OLE objects. That is retrieved to a SQL Server database, which stores them in a varbinary(max) field. However, those bins don't have the same format as regular files, as OLE structured them in its own fashion.
I want to mass download all those docs with a .NET C# app that uses SQL, but I can't find a way to do it. I have tried to copy the binary data in a new file using SqlDataReader, MemoryStream and FileStream, but they interpreted the information straightforward and not in the way OLE structured them. So, the resulting files were corrupted.
Is there a class that can interpret properly this OLE binaries? The old app used an OLE Container component, but they don't exist since a few years.
This is an old post that never got answered. I am in a similar situation myself in that I need to take images and convert them into OLE Files so that Microsoft Access OLE Bound Frames can read them. I have been researching this for days and have just now discovered the Binary Format that I need i.e. Compound File... or Structured File. They have this Nuget Package available which should accomplish what we both need available here: OpenMCDF
I am still in the process of trying to figure out how to create a compound file from Raw Binary... but feel I am close and will post the resolution when I get it.
UPDATE: The project that I was working on was not supposed to be so involved. Thus I didn't want to waste anymore time with it after already trying different recommendations for a couple days. Thus I ended up creating a complete hack that I never hope to have to repeat. Here are the steps that I used:
I set up a form with a maximized window to display each of the images one by one via a timer interval.
Between each interval I took programmatic screen shots (Print Screen) of the images, saving them in a separate folder using an identifiable naming convention.
That portion was done all in VBA.
Once I had full screen shots in a folder, I then used PhotoShop to then crop all the images in a batch to the exact image sizes.
A hack yes, but it accomplished what I needed... what a pain!
Following requirements we have users, their passwords and their profile picture stored in a .txt file. A user can be redirected to a profile modification page where they can change their password and/or password picture.
I need to replace the password and/or profile picture name in the .txt file when the user clicks OK.
The users.txt file is laid out like:
mike mike Avatar1
jessica123 123jessica Avatar4
mohd MoHd Avatar3
xiao AxiaoA Avatar2
anna abcANNAabc Avatar1
After doing some research i've tried using:
userFilePath = userFilePath.Replace(Session["password"].ToString(),
passwordTextBox.Text);
The original password is stored on log-in, through the session variable which I was using to find the original text in the file.
But this doesn't work. I'm pretty new to c#. Any help would be appreciated.
Be careful you have to change only the line concerned, many users could have the same passeword, so you have to read the file by line , identify the login concerned and of course not replace anything because also the login or avatar could has the same value as the password, but recreate the line with the new info, and after save the file with the new whole text value.
And just an advice for such needs it's better to use relational database.
Other than storing information in plain text files and saving passwords in clear text which others already mentioned as a bad idea, I think it is best to void direct string modification on the file level. It is best to abstract this away in a (list of) object and the modify the object and save it back to disk. You can easily do this using a library named FileHelpers, if you still want to use plain text files, without reinventing the wheel.
Also thing to consider, if you insist keeping this on a text file, is to switch to a better structured text file like JSON or XML format. You wouldn't need to use the above mentioned library anymore as .NET already comes with de/serializers for both.
If at all possible you should take the advice of the comments on your question and not do this in this manner.
If for some reason that is not possible, then here is a place to get started. You want to load the entire file from the file system into memory during a load phase, generally when your application starts, or first has need of the data.
Once the data is in some structure in memory you can alter like you would alter any object in C#.
Once the changes to the memory are done, you would write the entire thing back out to the file, overwriting the old file.
So your question is a very large one, there is the question of how do you read data from a file? How do you create a data structure in memory to hold this particular set of data? And how do you write the data back out to a file? I'd suggest working on one at a time in that order.
Oh and if you can't have a database, but you can change the file format, switch it to XML or JSON and the process of taking it from a file to memory, and back to a file will be much easier.
I need to be able to extract BLOBs from both Access 2003 and Access 2007. Access 2003 stores BLOBs as "OLE Objects", and Access 2007 gives you another option, "Attachment". The major difference is that multiple attachments can added to a single row, whereas there can be only one BLOB per "OLE Object" data type.
I have to be able to do this without using interop as I cannot force the dependency of having Office installed. That leaves me with either DAO or ADO. So I have written the code to use both technologies to pull the BLOBs out of a test database with loads of different types of file types embedded.
The problem I have is that it seems Access wraps the embedded files in some type of meta data. The net result is that the file, once extracted, is no longer the same and can not be opened by the associated application because its "corrupted". Access stores things like the original file name and so forth in this meta data. I need to be able to strip that meta data off of the files to have the file in its original state.
Is there some dark voodoo magic which can do this? There is very little by way of documentation on this subject. Any help would be appreciated.
Thanks in advance.
This occours because OLE objects are stored like "images" on Access. This leads to performance issues, also problems like yours. To deal with these limitations, Microsoft introduced Attachment fields on Access 2007/2010, wich doesn't need OLE servers to run content. Attachments, wich should be more than one file per record, are automaticaly managed in backstage by MS-Access. Maybe you should use Attachments or change your database for SQL Server, MY-SQL, Firebird.