Develop Reference

Streaming a large file into a database BLOB field

I have a large file on disk whose contents I need to store in a SqlServer database as a VARBINARY(MAX) field. By "contents", I mean using something like File.ReadAllBytes(). (I understand the pitfalls of this, but it's the current mode of operations, so I have to deal with it for now.)
I found this answer, which provides a way to stream a large Byte[] by using UPDATE.WRITE:
How to I serialize a large graph of .NET object into a SQL Server BLOB without creating a large buffer?
However, simply reading the contents of a large file into a Byte[] in memory leads to this problem:
OutOfMemoryException when I read 500MB FileStream
I might be overlooking something obvious, but I just can't work out how should I go about getting from a large file on disk, to the resulting storage into the database.

Using some hints I got from these two pages, I have an answer that works:
http://www.syntaxwarriors.com/2013/stream-varbinary-data-to-and-from-mssql-using-csharp/ (this looks a lot like the serialize SO answer, but there's more here...not sure who copied who!).
How do I copy the contents of one stream to another?
Basically, it uses the same methodology as the answer about serializing Blobs, but instead of using BinaryFormatter (a class I'm not fond of anyhow), it creates a FileStream that takes the path to the file, and an extension method to copy that stream into the target stream, or BlobStream, as the example named it.
Here's the extension:
public static class StreamEx
{
public static void CopyTo(this Stream Input, Stream Output)
{
var buffer = new Byte[32768];
Int32 bytesRead;
while ((bytesRead = Input.Read(buffer, 0, buffer.Length)) > 0)
Output.Write(buffer, 0, bytesRead);
}
}
So the trick was to link two streams, copying the data from one to another in chunked fashion, as noted in the comments.

creating a file and allocating storage before writing to it

I have a file that gets GBs of data written to it over time (looping once it reaches the end). I would like to create the file ahead of time and preset the storage so that the required storage is never taken up by other downloads during the writing to the file. This is done using visual studio 2012 in C#.
I have tried:
if (fileSizeRequirement or fileName is changed) //if filePath or file size is changed
{
//Open or create the file, set the file to size requirement, close the filestream
fileStream = new FileStream(fileName, FileMode.OpenOrCreate, FileAccess.ReadWrite);
fileStream.SetLength((long)fileSizeRequirement);
fileStream.Close();
}
1) Is this an appropriate way to "preallocate" space for a folder?
2) Will the SetLength require a seek to the beginning after setting the length or does the position in the folder stay at the beginning?
3) What is the correct way to achieve file preallocation of storage space?
Thanks ahead of time and I appreciate any suggestions.

Using SetLength is a common approach although I'd generally use a using statement here.
using(var fileStream = new FileStream(fileName, FileMode.OpenOrCreate, FileAccess.ReadWrite))
{
fileStream.SetLength((long)fileSizeRequirement);
}
Calling fileStream.Position straight after SetLength yields 0 so you shouldn't need to seek to the beginning.

how to get a picture from a postgresql database

i want to select a picture that save as a large object in a postgresql database.
i know i use lo_export to do this.
but there is a problem: i want to save this picture directly to my computer because i cant access the files that save on Server using lo_export
(i think that is best for me if picture transfer to my computer by a select query)

I don't exactly know my way around C# but the Npgsql Manual has an example sort of like this of writing a bytea column to a file:
command = new NpgsqlCommand("select blob from t where id = 1);", conn);
Byte[] result = (Byte[])command.ExecuteScalar();
FileStream fs = new FileStream(args[0] + "database", FileMode.Create, FileAccess.Write);
BinaryWriter bw = new BinaryWriter(new BufferedStream(fs));
bw.Write(result);
bw.Flush();
fs.Close();
bw.Close();
So you just read it out of the database pretty much like any other column and write it to a local file. The example is about half way down the page I linked to, just search for "bytea" and you'll find it.
UPDATE: For large objects, the process appears to be similar but less SQL-ish. The manual (as linked to above) includes a few large object examples:
NpgsqlTransaction t = Polacz.BeginTransaction();
LargeObjectManager lbm = new LargeObjectManager(Polacz);
LargeObject lo = lbm.Open(takeOID(idtowaru),LargeObjectManager.READWRITE); //take picture oid from metod takeOID
byte[] buf = new byte[lo.Size()];
buf = lo.Read(lo.Size());
MemoryStream ms = new MemoryStream();
ms.Write(buf,0,lo.Size());
// ...
Image zdjecie = Image.FromStream(ms);
Search the manual for "large object" and you'll find it.

Not familiar with C# but if you've contrib/dblink around and better access to a separate postgresql server, this might work:
select large object from bad db server.
copy large object into good db server using dblink.
lo_export on good db server.

If your pictures don't exceed 1GB (or if you don't access only parts of the bytes) then using bytea is the better choice to store them.
A lot of SQL GUI tools allow to directly download (even view) the content of bytea columns directly

How to save file in SQL Server database if have file path?

I am building some C# desktop application and I need to save file into database. I have come up with some file chooser which give me correct path of the file. Now I have question how to save that file into database by using its path.

It really depends on the type and size of the file. If it's a text file, then you could use File.ReadAllText() to get a string that you can save in your database.
If it's not a text file, then you could use File.ReadAllBytes() to get the file's binary data, and then save that to your database.
Be careful though, databases are not a great way to store heavy files (you'll run into some performance issues).

FileStream fs = new FileStream(fileName, FileMode.Open, FileAccess.Read);
BinaryReader br = new BinaryReader(fs);
int numBytes = new FileInfo(fileName).Length;
byte[] buff = br.ReadBytes(numBytes);
Then you upload it to the DB like anything else, I'm assume you are using a varbinary column (BLOB)

So filestream would be it but since you're using SQL 2K5 you will have to do it the read into memory way; which consumes alot of resources.
First of the column type varchar(max) is your friend this give you ~2Gb of data to play with, which is pretty big for most uses.
Next read the data into a byte array and convert it to a Base64String
FileInfo _fileInfo = new FileInfo(openFileDialog1.FileName);
if (_fileInfo.Length < 2147483647) //2147483647 - is the max size of the data 1.9gb
{
byte[] _fileData = new byte[_fileInfo.Length];
_fileInfo.OpenRead().Read(_fileData, 0, (int)_fileInfo.Length);
string _data = Convert.ToBase64String(_fileData);
}
else
{
MessageBox.Show("File is too large for database.");
}
And reverse the process to recover
byte[] _fileData = Convert.FromBase64String(_data);
You'll want to dispose of those strings as quickly as possible by setting them to string.empty as soon as you have finished using them!
But if you can, just upgrade to 2008 and use FILESTREAM.

If you're using SQL Server 2008, you could use FILESTREAM (getting started guide here). An example of using this functionality from C# is here.

You would need the file into a byte array then store this as a blob field in the database possible with the name you wanted to give the file and the file type.
You could just reverse the process for putting the file out again.

c# - reading/writing blob from firebird

I'm trying to read image from firebird with c#, and save it to file system.
I can read from table and save. but image can not be viewed. I tried many piece of
code on net.but result is same :(
Can any one help me please?

How are you retrieving the code form Firebird? If you are getting the data and casting it to a byte[] try:
byte [] blob = row["image"];
// create a file FileStream to write the data
FileStream fs = new FileStream("image.jpg", FileMode.Create);
fs.Write(blob,0,blob.Length);
fs.Close();
The above code should do the trick.
PS: I have made lots of assumption in the code, but you can get the idea.