I have a aspx webpage where there's an option to upload multiple files.These files have to be stored in the database in BLOB format.
What will be the most efficient manner to store these files? There's no constraint on the size or number of file to be uploaded.
Should I upload the files one by one whenever the user clicks the file upload button or
upload them once simultaneously when the whole form gets submitted on save button. Please keep in mind, this is to used by multiple users about ~1000 at a time
Considering that it is a BLOB data, I would consider to use some No-SQL database (MongoDB, RavenDB), where you save just "document" with data, so it's easier to manage in these kind of situations. But you will need more disk space in this case.
What about upload: I would go one after one, as in case if connection drops, at least some of the files are delivered.
On the server side, would look on Redis like in memory cache that always ready to accept user "session" (a sequence of declared quantity of files), and one time all of them delivered, or connection failure the content of that session is saved on the disk.
Just general overview what can be done to give you some hints.
set file upload control multiple attribute to 'multiple' and in the code behind get request user submitted files, then loop through their memory stream and store them in your db as byte arrays
Given the limitations of your scenario (where the images must be stored in (I assume) a single SQL Database) I would look into uploading the images one by one, and I would investigate if the SQL Server Transient Fault Handling with the Exponential back-off strategy could help to try and handle the 'queued' uploads.
Without more details I can't really say.
Related
One of the many things that SharePoint does extremely well is that when you have versioning enabled for files uploaded to a Document Library, every time you save changes to a file it only saves the difference from the previous version of the file to the Content Database but NOT the whole file again.
I am trying to duplicate that same behavior with standard C# code on either a File System folder in Windows or a SQL Database blob field. Does anyone have any idea or pointers on how SharePoint accomplishes this and how it can be done outside of SharePoint?
SharePoint uses a technique called data "shredding" to contain each change to a given file. Unfortunately, I don't think you will find enough technical details to truly reproduce what they are doing, but you might be able to devise a reasonable approximation using your own design.
When shredded, the data associated with a file such as Document.docx is distributed across a set of BLOBs associated with the file. The independent BLOBS are each assigned a unique ID (offset) to enable reconstruction in the correct order when requested by a user.
Each document "shred" is stored in a SQL database table named DocStreams. Each BLOB contains a numerical Id representative of the source BLOB when coalesced. When a client updates a file, only the shredded BLOB that corresponds to the change is updated with the update occurring on the database server as opposed to the Web server.
For more details on Shredding see
http://download.microsoft.com/download/9/6/6/9661DAC2-393D-445A-BDC1-E60743B1231E/Shredded%20Storage%20in%20SharePoint%202013.pdf
https://jeremythake.com/the-truth-behind-shredded-storage-in-sharepoint-2013-a84ec047f28e
https://www.c-sharpcorner.com/UploadFile/91b369/shredded-storage-in-sharepoint-2013/
for the past 3 days I've been trying to create an upload system for multiple files, possibly large, with progress bars.
I've been roaming the web relentlessly for the past few days, and I can say, I am now familiar with most difficulties.
sadly, all the solutions I've found online are not written c# or vbscript, in fact most of them are written in php.
I wouldn't mind switching to another language but the entire website is written in vb.net and for the sake of coherence I thought it might be best to keep with it.
File uploads:
Problem 1 - progress bar:
I understand file uploads will not work with ajax, since the ajax response will only occur after the file had completed its upload.
I understand there is a solution using iFrames but I cannot seem to find any online examples (preferably using vb.net or c#).
I understand there is another alternative using flash. how???
I also understand people are mostly against using iframes but I can't find what the reason might be.
Problem 2 - Multiple Files:
I can have multiple file support with HTML5. great, but IE doesn't support it? well... IE users will just have to upload one file at a time.
Problem 3 - Large files:
how?
I heard something about chunking, and blobs, but these are still just random gibberish words for me. can somebody explain, the meaning and the implementation?
references to reading material are much appreciated even though, if it's on the web, I've probably already read it in my search for my solution.
#DevlshOne has a decent thread with some good information.
Here are the three basic requirements for what I did:
Create Silverlight app for clientside access and upload control. (use app of your choice)
Create an HttpHandler to receive the data in chunks and manage requests.
Create the database backend to handle the files.
Silverlight worked well because I was already in VB (ASP.NET). When used in-browser, as opposed to out-of-browser, the ASP.NET session was shared with Silverlight, so there was no need to have additional security/login measures. Silverlight also allowed me to limit what file types could be selected and allow the user to select multiple files from the same folder.
The Silverlight app grabs the files selected by the user, displays them for editing of certain properties, and then begins the upload when the user clicks the 'upload' button. This sets off a number of threads that each upload chunks of data to the httphandler. The HttpHandler and Silverlight app send and receive in chunks, with the HttpHandler always sending an OK or ERROR message when the request has been processed for the uploaded chunk.
Our specific implementation of file uploading also required some database properties (fields) to be filled out by the user, so we also had inputs for those properties and uploaded them to the server with the file data.
An in-browser Silverlight app can also have parameters passed into it through the html, so I do this with settings like 'max chunk size' or 'max thread count'. I can change the setting in the database and have it apply to all users.
The database backend is basically a few stored procedures (insert your data management preference here) that control the flow of the logic. One table holds completed files (no file data), and a second holds the temp files that are in progress of being uploaded. One stored procedure initiates a new file record in the temp table and processes additional chunk uploads, and another controls the migration of the completely uploaded file from the temp table to the completed table. (A piece of VB code in the HttpHandler migrates the actual binary file data from the temp table to a physical file.)
This seems pretty complex, but the most difficult part would be the interaction with the handler and passing the chunks around (response/requests, uploading successive chunks, etc.). I left out a lot of information, but this is the basic implementation.
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Storing Images in DB - Yea or Nay?
Pretty straight forward, I am hosting a site where users can upload pictures, and I have a .net File upload control working appropriately.. I'm just wondering what methodology I should use to store them on the server..
I can use the SaveAs() method off the FileUpload control which saves it as an actual file..
I can break the image down to a Byte[] and store it in the database for access.
I believe the real question here is.. Do I want the load on IIS or Sql Server 2008 R2?
What's the general consensus on which methodology I should use and why? Thanks!
There is no general consensus, because neither method is ideal.
Storing the images as files means that it becomes harder to scale the application to run on multiple servers. Then you would need a SAN/NAS to store the files instead of a local disk.
Storing the images in the database means that you are using a lot of the cache space for image data, slowing down other queries, and it also increases the size of the database, and the backups. It also means that you have to serve the images through a page, you can't just request the file directly.
So, it comes down to how much images you will have, and how scalable you need the application to be.
Avoid getting them in the database. That will make your DB a lot larger in size just because of a few files. That also affects the backup size.
I do not see any real gain on having the actual file bytes in the database. If you have the physical path to the file in the file system, that would suffice.
That also allows you to have your own backup strategy for the files.
In my web application I am working with files. Some files are very large. I use Response.Write() to write the file to the browser. This goes well for the smaller files, but for large files this can take a while and the bandwidth is fully used.
Is it possible to split large documents and send it piece by piece to the browser? Are there other ways to send the document quicker to the browser?
I hold the document as a property of an object.
Why don't you compress the file and store it in the DB and decompress it will extracting it?
You can do a lot of things depending on this questions:
How often does the file change?
Do I really need the files in the DB?
Why not store the File path in the
DB and the File on disk?
Anyhow, since your files are extremely high bandwidth and you would want your app to respond appropriately you might want to use AJAX load the files Asynchronously. You can have a WebHandler .ashx for this.
Here's a few examples:
http://www.dotnetcurry.com/ShowArticle.aspx?ID=193&AspxAutoDetectCookieSupport=1
http://www.viawindowslive.com/Articles/VirtualEarth/InvokingserversidecodeusingAJAX.aspx
My question is, is it possible to
split large documents and send it
piece by piece to the browser?
It depends on the file type, but in general no. If you are sending something like an excel file or a word doc etc. the receiving application will need all of the information (bytes) to fully form the document. You could physically separate the document into multiple ones, and that would allow you to do so.
If the bandwidth is fully used, then there is nothing you can do to "speed it up" short of compressing the document prior to send. In other words, zip it up.
Depending on the document (I know you said .mht, but we're talking content here) you will see the size go down by some amount. Maybe it's enough, maybe not.
Either way, this is entirely a function of the amount of content you want to send versus the size of the pipe available to send it. One of those is more difficult to change than the other.
Try setting IIS's dynamic compression. By default, it's set fairly low, but you can try setting it for a higher compression level and see how much that helps.
I'm not up to speed with ASP.NET but you might be able to buffer from a FileStream to some sort of output stream.
You can use the Flush method to send the currently buffered data to the client (the browser).
Note that this has some implications, as is described aptly here.
I've considered using it myself, a project sent documents that became fairly large and I was cautious about storing the whole data in memory. In the end I decided the data was not large enough to be a problem though.
Sadly the MSDN documentation is very, very vague on what Flush implies and you will probably have to use Google to troubleshoot.
I want to retrieve an image from SQL Server to show it in an ASP.NET app. I think that I can write down the image retrieved from SQL Server to a server's folder and show it throw its path. Is there any other better way?
I have the same problem with audio and video files (can I use silverlight to play these audio and video files?)
My worry is that I don't want to store these files (images, audios and videos) on server to show it.
You can write a custom HTTP handler that will take ID of the item you are trying to display on the query string parameter.
This handler will then retrieve the data from SQL Server, and return it just like downloading a file. This post has the steps.
I am not sure about the capability of Silverlight streaming from a "file", you may need to use Silverlight Streaming Service.
Is there a better way? IMHO, yes there is. Store your image file as a file. That's what the filesystem is for, and you don't get involved in fudging around with the database.
I have, in the past, used MySQL to store images and retrieved them with PHP, but that solution got really old really quickly from a maintenance point of view. It comes down to the fact that you are always going to have to store the data in some way, and the most efficient way is also the simplest.
If you must store it in the database, I'd use a generic accessor page (e.g. myImage.aspx) which queries the database and streams the returned blob directly to the browser. Your img tag would probably then need to look like
<img src="/img/myImage.aspx?imgId=123456" <other tag data> />
If you use SQL Server 2008, then I believe you can store these objects as a FILESTREAM, and you can then retrieve the path to the files on the file system. You should be able to arrange for that path to be mapped via IIS, and thus it should be practical to turn it into a URL.
Your image source can be
<img src="blah.aspx?id=123" />
In blah.aspx, retrieve the image data from the db and use Response.BinaryWrite to send to the browser, with the appropriate content type set.
HI all, I guess the solution for this is present in below links:
http://www.codeproject.com/KB/aspnet/asp_net_and_mysql.aspx
http://www.codeproject.com/KB/aspnet/image_asp.aspx
This solution in for mysql database and also could be suitably modified to work with mssql database as well.