I'm using the code
cmd.Parameters.Add("#Array", SqlDbType.VarBinary).Value = Array;
The SqlDbType.VarBinary description states that it can only handle array's upto 8 K bytes. I have a byte array that represents an image and can go upto 10k bytes.
How do I store that in a varbinary(max) column using C#?
I have no trouble creating the array. I'm stuck at this 8k limit when trying to execute the query.
Edit: Let me clarify, on my machine even pictures upto 15k bytes get stored on the database in the varbinary(MAX) column when I run the asp.net application locally but once I deployed it the pictures would not get stored. I then resorted to drastically resizing the images to ensure their size was less that 8K and now the images get stored without any problem.
Perhaps you could look at the Sql Server FILESTREAM feature since its meant for storing files. It basically stores a pointer to your file and the file is stored directly in the filesystem (in the databases data directory).
I like FILESTREAM since you it means you continue to use the interface to the database (SQLClient for example) rather then breaking out to an adhoc method to read/write files to the harddrive. This means security is managed for you in that your app doesn't need special permissions to access the filesystem.
Quick google gave this acticle on using filestream in c# but I'm sure there are many others.
UPDATE following OP EDIT
So once deployed to other server the upload fails? Perhaps the problem is not the sql insert but that there is a http request content length limit imposed - for example in your web.config the httpRuntime element has the maxRequestLength attribute. If this is set to a low value perhaps this is the problem. So you could set to something like this (sets max to 6MB well over the 10kb problem):
<system.web>
<httpRuntime maxRequestLength="6144" />
The only thing here is the limit it 4MB buy default :|
No, this is what the description actually says:
Array of type Byte. A variable-length stream of binary data ranging
between 1 and 8,000 bytes. Implicit conversion fails if the byte array
is greater than 8,000 bytes. Explicitly set the object when working
with byte arrays larger than 8,000 bytes.
I would assume that what that actually means is that you cannot use AddWithValue to have a parameter infer the type as VarBinary if the byte array is over 8000 elements. You would have to use Add, specify the type of the parameter yourself and then set the Value property, i.e. use this:
command.Parameters.Add("#MyColumn", SqlDbType.VarBinary).Value = myByteArray;
rather than this:
command.Parameters.AddWithValue("#MyColumn", myByteArray);
Adding the length of data seems to be the fix
var dataParam = cmd.Parameters.AddWithValue("#Data", (object)data.Data ?? DBNull.Value);
if (data.Data != null)
{
dataParam.SqlDbType = SqlDbType.VarBinary;
dataParam.Size = data.Data.Length;
}
Related
I'm currently using SSIS to do an improvement on a project. need to insert single documents in a MongoDB collection of type Time Series. At some point I want to retrieve rows of data after going through a C# transformation script. I did this:
foreach (BsonDocument bson in listBson)
{
OutputBuffer.AddRow();
OutputBuffer.DatalineX = (string) bson.GetValue("data");
}
But this piece of code that works great with small file does not work with a 6 million line file. That is, there are no lines in the output. The other following tasks validate but react as if they had received nothing as input.
Where could the problem come from?
Your OuputBuffer has DatalineX defined as a string, either DT_STR or DT_WSTR and a specific length. When you exceed that value, things go bad. In normal strings, you'd have a maximum length of 8k or 4k respectively.
Neither of which are useful for your use case of at least 6M characters. To handle that, you'll need to change your data type to DT_TEXT/DT_NTEXT Those data types do not require a length as they are "max" types. There are lots of things to be aware of when using the LOB types.
Performance can suck depending on whether SSIS can keep the data in memory (good) or has to write intermediate values to disk (bad)
You can't readily manipulate them in a data flow
You'll use a different syntax in a Script Component to work with them
e.g.
// TODO: convert to bytes
Output0Buffer.DatalineX.AddBlobData(bytes);
Longer example of questionable accuracy with regard to encoding the bytes that you get to solve at https://stackoverflow.com/a/74902194/181965
I'm passing messages between Windows C# client and Linux C++ server via TCP sockets. C# code uses protobuf-net v2, Linux Google's version of protobuf. Small test object that I pass has 6 fields ( Enum, Int, String). I need help with two problems:
C# portion unable to deserialize data sent from Linux, unless the Memory stream used as the storage for the data is initialized with the binary array in the constructor. Array can not be larger than the data sent from Linux ( 9 bytes in my case ). Code sample - byte[] data = new byte[9], copy data from the socket into the array.
MemoryStream myStream = new MemoryStream(data), pass myStream to Serializer.Deserialize...
If I initialize MemoryStream without bynary buffer or with array of 1024 bytes, Deserialize will create empty object, without processing data.
When I try to serialize the same object with the same values as Linux same in C# the size of the data is 11 bytes vs 9 on Linux. I checked the byte array in the debugger, C# version has the same 9 fields as Linux data in the indexes 2-11 of the array. Index 0 is 8 and index 1 is 9. I can try to get around the problem, by modifying Linux deserialization code, just need to know, if I always have to deal with two extra fields at the beginning of the message. Also, I can add two extra fields to messages generated on Linux if it going to fix my deserialization in C#, just need to know how to generate values for these two fields.
Thanks.
Alex.
Protobuf data is not self-terminating, simply. You can, however, create either a MemoryStream or ProtoReader that takes a larger payload, but is limited to a virtual length. If you are sending multiple messages, you will need to know the length of the payload - that is inescapable. This is often achieved via a length prefix. I would expect this to throw random errors - most likely "invalid field header 0" - and I wonder if you are swallowing that exception
impossible to comment without the specific example; most likely something to do with the default value of field number 1 (header 8 === "field 1 varint")
I have a situation where I need to return the MAX of 2MB sized string from a method.
I have seen some web services returning big XML contents as strings. What is the optimum size with which we should transact in value parameter passing and returning?
There is no technically correct answer to this because it's perfectly normal to pass any amount of characters as long as it's needed in your program. You need to evaluate if passing such a large string is valueable to your program. If it is, great. If it's not, find a way to shorten it or pass a file path or database identifier instead so the service can get the text from there itself.
Is it OK to return a string with the size around 2 MB from a C#
if it is local (intranet,network) then it is mostly ok (unless the network is v v old). But, if you are passing it over the internet then you should consider the bandwidth of the client on which the data will be sent/received. Will it slow down your page or not etc. In most cases though it will be fine.
What is the optimum size with which we should transact in value parameter passing and returning
Any size that will not cause your application to slow down is fine.
It is completely okay, the only thing you should keep in mind is:
If a user is using 56kbps connection 2024/56=36.14 sec if this is not important for your case then no need to worry.
I know this question has been asked before, but I can't seem to get it working with the answers I've read. I've got a CSV file ~ 1.2GB , If I'm running the process like a 32bit i get outOfMemoryException, it works if i run it as a 64bit process, but it still takes 3,4gb in memory, i do know that I'm storing a lot of data in my customData class, but still 3,4gb of ram?, Am I doing something wrong when reading the file?
dict is a dictionary in which i just have a mapping to which property to save something in, depending on the column it's in. Am i doing the reading the right way?
StreamReader reader = new StreamReader(File.OpenRead(path));
while(!reader.EndOfStream) {
String line = reader.ReadLine();
String[] values = line.Split(';');
CustomData data = new CustomData();
string value;
for (int i = 0; i < values.Length; i++) {
dict.TryGetValue(i, out value);
Type targetType = data.GetType();
PropertyInfo prop = targetType.GetProperty(value);
if(values[i]==null)
{
prop.SetValue(data, "NULL",null);
}
else
{
prop.SetValue(data, values[i], null);
}
}
dataList.Add(data);
}
There doesn't seem to be anything wrong in your usage of the stream reader, you read a line in memory, then forget it.
However, in C# a string is encoded in memory as UTF-16 so on the average a character consumes 2 bytes in memory.
If your CSV contains also a lot of empty fields that you convert to "NULL" you add up to 7 bytes for each empty field.
So on the whole, since you basically store all the data from your file in memory, it's not really surprising that you require almost 3 times the size of the file in memory.
The actual solution is to parse your data by chucks of N lines, treat them, and free them from memory.
Note: Consider using a CSV parser, there is more to CSV than just comas or semi-colons, what if one of your field conatins a semi-colon, a newline, a quote... ?
Edit
Actually each string take up to 20+(N/2)*4 bytes in memory see C# in Depth
Ok a couple of points here.
As pointed out in the comments, .NET under x86 can only consume 1.5GBytes per process, so consider that your maximum memory in 32 bit
The StreamReader itself will have an overhead. I don't know if it caches the entire file in memory, or not (maybe someone can clarify?). If so, reading and processing the file in chunks might be a better solution
The CustomData class, how many fields does it have, and how many instances are created? Note you will need 32bits for each reference in x86 and 64 bits for each reference in x64. So if you have CustomData class, which has 10 fields of type System.Object, each CustomData class before storing any data requires 88 bytes.
The dataList.Add at the end. I assume you are adding to a generic List? If so, note that List employes a doubling algorithm to resize. If you have 1GByte in a List and it requires 1 more byte in size, it will create a 2GByte array and copy the 1GByte to the 2GByte array on resize. So all of a sudden the 1GByte + 1 byte actually requires 3GBytes to manipulate. Another alternative is to use a pre-sized array
Hey Everyone, I am writing some code that makes use of SQL Server CE 3.5 and I am having a very strange problem. I have a string field in one of the tables that needs to store a full file path.
Over the course of trying to fix this problem I have that field set as nvarchar with a max size of 4000, but it is still cutting longer strings that are much shorter than the limit off
for example:
D:\iTunes\iTunes Media\Music\Abigail Williams\In The Absence Of Light\02 Final Destiny Of The Gods.m
This is clearly smaller than 4000 characters, yet it is missing the p3 at the end of the string.
I am using a table adapter to enter the data into the database with the following query:
INSERT INTO [Track] ([Artist_ID], [Album_ID], [FilePath], [LastUpdate])
VALUES (#Art, #Al, #Fp, #LU)
I know that the strings are fully formed on insert because I am using the following code to check:
if(!temp.Filepath.EndsWith(".mp3"))
MessageBox.Show("File Error");
this.trackTableAdapter1.InsertQuery(ArtID, AlID, temp.Filepath, File.GetLastWriteTime(temp.Filepath));
The message box does not get shown, so the string must end correctly on insert.
the query that extracts the data is:
SELECT
*
FROM Track
WHERE Artist_ID=#Artist_ID AND Album_ID=#Album_ID
The involved code is:
foreach (Database.MusicDBDataSet.TrackRow TR in this.trackTableAdapter1.GetAlbumTracks(AR.Artist_ID, AlR.Album_ID).Rows)
{
//if (!TR.FilePath.EndsWith(".mp3"))
//MessageBox.Show("File Path Error");
this.ArtistList[AR.Name].AlbumList[this.ArtistList[AR.Name].AlbumList.Count - 1].TrackList.Add(new Track(TR.FilePath, AlR.Name, AR.Name));
}
Has anyone ever run into this problem before?
Check the XSD file. Specifically, check the FilePath column of your table and look for the max length.
Maybe take a look at the SQLServerCE Parameter Size limitation.
What is the specific maximum length? Is it around 100 chars? (Guessing based on your provided input example).
The 100 unicode chars also matches with D.K. Mulligan's answer. Looking at SQL ServerCE Paramater Size Property
For variable-length data types, the Size property describes the maximum amount of data to send to the server. For example, the Size property can be used to limit the amount of data sent to the server for a string value to the first 100 bytes.
For Unicode string data, the Size property refers to the number of characters. The count for strings does not include the terminating character.
Try bumping the size to see if this is the magic number that is truncating your strings.