I am executing a query in Entity Framework to select LONG RAW data into a byte array.
var result = db.Database.SqlQuery<byte[]>("SELECT MESSAGE FROM FOCUS.ENTRIES");
var list = await result.ToListAsync();
When I execute this code, I get a list of byte arrays, but all of them are empty. In the database they are not empty.
The MESSAGE table looks like this:
CREATE TABLE "FOCUS"."ENTRY"
( "PRIMKEY" NUMBER,
"TITLE" VARCHAR2,
"MESSAGE" LONG RAW
);
I am using ODP.NET, Managed Driver as DB provider.
I guess it's some mapping problem, but i can't figure it out.
Any help would be welcome.
Thanks!
SqlQuery expects a class with member name equivalent to SQL column.
public class MessageInfo{
public byte[] Message;
}
var result = await db.Database
.SqlQuery<MessageInfo>("SELECT MESSAGE FROM FOCUS.ENTRIES")
.ToListAsync();
var list = result.Select( x => x.Message );
Few notes from Oracle Docs , https://docs.oracle.com/html/A96160_01/features.htm
When an OracleDataReader is created containing LONG or LONG RAW types,
OracleDataReader defers the fetch of the LONG or LONG RAW column data.
The initial number of characters for LONG or bytes for LONG RAW
fetched on the client side depends on the InitialLONGFetchSize
property of the OracleCommand. By default, InitialLONGFetchSize is 0.
ODP.NET does not support CommandBehavior.SequentialAccess. Therefore,
LONG and LONG RAW data can be fetched in a random fashion.
To obtain data beyond InitialLONGFetchSize bytes or characters, a
primary key column must be provided in the list of selected columns.
The requested data is fetched from the database when the appropriate
typed accessor method (GetOracleString for LONG or GetOracleBinary for
LONG RAW) is called on the OracleDataReader object.
So try adding primary key and see if you can retrieve data. Otherwise you will have to skip SqlQuery and use ODP.NET directly to fetch if you cannot change configuration. Or you will have to create instance of OracleConnection and pass it as parameter in your DbContext's constructor.
Related
I have search this problem on internet, but my issue is different.
I'm using Fluent NHibernate and try insert data with sql query:
var query = "INSERT INTO TABLE_NAME('ID','CONTENT') VALUES(:ID, :CONTENT)";
var executedQuery = Session.CreateSQLQuery(query);
executedQuery.SetParameter("ID", data.Id);
executedQuery.SetParameter("CONTENT", data.Content);
executedQuery.ExecuteUpdate();
Here data passing to method. In database(Oracle 11g) datatype of CONTENT is NCLOB. When try to insert data, I get this error:
ORA-01461: can bind a LONG value only for insert into a LONG column
What is problem in here?
This error is not very helpful and goggling it will most likely result in topics regarding oracle patches and the like. In reality this is a bug with the microsoft oracle client driver. The driver mistakenly infers the column type of the string being saved, and tries forcing the server to update a LONG value into a CLOB/NCLOB column type. The reason for the incorrect behavior is even more obscure and only happens when all the following conditions are met:
when we set the IDbDataParameter.Value = (string whose length is : 4000 > length > 2000 )
when we set the IDbDataParameter.DbType = DbType.String
when DB Column is of type NCLOB/CLOB
In this situation you must set database column type in set parameter method overload, so:
executedQuery.SetParameter("CONTENT", data.Content, NHibernateUtil.StringClob);
So I have a program that processes a lot of data, it pulls the data from the source then saves it into the database. The only conditions is that there cannot be duplicates based on just a single name field, there is no way to only pull new data, I have to pull it all.
The way I have it working right now is that it pulls all the data from the source, then all of the data from the DB (~1.5m records) then compares and only sends the new entries, however this is not very good in terms of RAM since it can use up around 700mb.
I looked into a way to let the DB handle more of the processing and found the below extension method on another question, but my concerns with using it is that it might be even more inefficient due to having to check 200 000 singularly against all records in the db
public static T AddIfNotExists<T>(this DbSet<T> dbSet, T entity, Expression<Func<T, bool>> predicate = null) where T : class, new()
{
var exists = predicate != null ? dbSet.Any(predicate) : dbSet.Any();
return !exists ? dbSet.Add(entity) : null;
}
So, any ideas as to how I might be able to handle this problem in an efficient manner?
1) Just a suggestion: because this task seems to be "data intensive" you could use a dedicated tool: SSIS. Create one package having one Data Flow Task thus:
OLE DB source (to read data from source)
Lookup transformation (to check if every source value exists or not within target table; This needs indexing)
No match data flow (source value doesn't exist) then
OLE DB destination (it inserts the source value into target table).
I would use this approach if I have to compare/search on small set of columns (this columns should be indexed).
2) Another suggestion: is to simply insert source data into #TempTable and then
INSERT dbo.TargetTable (...)
SELECT ...
FROM #TempTable
EXCEPT
SELECT ...
FROM dbo.TargetTable
I would use this approach if I have to compare/search on all columns.
You have to do your tests to see which is the right/fastest approach.
If you stick to EF, there's a few ways of improving your algorithm:
Download only Name field from your source table (create a new DTO object that maps to the same table but contains only the name field against which you compare). That will minimize the memory usage.
Create and store the hash of the name field in the database. Then download on the client only hashes. Check the input data - if there's no hash, just add it; if there's such hash - check with dbSet.Any(..). If you use the MD5 hash it will take only 16 (bytes) *1,5M records ~ 23Mb. If you use CRC32, 4*1,5 records ~ 6Mb.
You have the high memory usage because all the data you added is stored in the DB context. If you do in chunks (let's say 4000 records) and then close the context, the memory will be released.
using (var context = new MyDBContext())
{
context.DBSet.Add(...); // and no more than 4000 records here.
context.SaveChanges();
}
The better solution is to use SQL statements that can be executed via
model.Database.ExecuteSqlCommand("INSERT...",
new SqlParameter("Name",..),
new SqlParameter("Value",..));
(Mind to set the parameter type for SqlParameter)
INSERT target(name, value)
SELECT #Name, #Value
WHERE NOT EXISTS (SELECT name FROM target WHERE name = #Name);
If the expected number of inserted values is high, you might get benefits form using Table type (System.Data.SqlDbType.Structured) (assuming you use MS SQL server) and executing the similar query in some chunks but it is a little bit more complex.
Depending on the SQL server you may download the data into a temporary table (e.g. you have a CSV file or BulkInsert for MS SQL) and then execute the similar query.
I am currently developing an application in c# where I have a string array NameArray and recorded the number of values in the array as count. I need to pass this to my sql server database where I will store each of the value of the array as individual records(names) into a Table. How do I write a stored procedure for doing such a operation.
Furthermore the name of the entity connection string is MyEntities and I have created its object as Entityobj.
The code I am trying to execute in c# is something like this
public void Method1(string[] NameArray, int count)
{
Entityobj.CallSproc(NameArray,count);// Here I am passing the values to stored procedure
}
How do I make the stored procedure receive the values and store them as individual records?
Anyways I worked out the problem myself with the help of using this link
http://www.c-sharpcorner.com/UploadFile/78607b/using-table-valued-parameters-in-entity-framework/
The link shows how to pass simple values to Table value parameters using Entity object. I edited the code to pass an by adding each value of the array into a single row(through iteration). I don't know whether this is a efficient way to do it but it serves my purpose well and code is working pretty fine.
Also there was a warning in my code as Entity Framework does not support table as datatype(table valued parameters) but the code worked fine anyway.
I'm trying to store a binary blob in a database table and get an output back from it. The data is stored in a MemoryStream object. I'm trying to save this to a sql server 2012 table using query async. The call succeeds but no data is inserted into the column. (e.g. I get a 0x0 entry when I query it back).
Sure enough, actually checking a trace I see dapper sending a 0x0. The memorystream has a length so am I doing something wrong or does dapper not support this scenario?
My query string is just a simple insert and gets the id and insertion time back.
I'm using the following call
using(var conn=new SqlConnection(_connstr)){
var dynParams = new DynamicParameters();
dynParams.Add("BinaryBlob",
_memoryStream,DbType.Binary,ParameterDirection.Input,-1);
var res = await conn.QueryAsync<MyResultType>(_queryStr, dynParams);
}
The query inserts a row and gets a timestamp back, but no data is actually inserted. What am I doing wrong?
Make sure you seek to the beginning of the Memory stream. Streams have a positional state. Another approach would be to convert the memory stream to a Byte[] before trying to persist it.
e.g.
_memorystream.Seek(0, SeekOrigin.Begin);
Overall Description
I have a web service that:
Retrieves a dataset from our
database
Uses JsonConvert.SerializeObject() to serialize the
object and returns that data to a user's local application as a string
then my client's local application:
Uses JsonConvert.DeserializeObject<DataSet>() to convert the object back into a System.Data.DataSet
My problem
I have a Data Column that holds Varbinary data (Byte[]). This serialize/deserialize process ends up causing that column data type to be a String (I'm guessing base64).
Is there a way that when I deserialize the JSON object that it treats this column as a varbinary?
EDIT: In response to comments
My issue is that I generate a CREATE TABLE scripts based on the columns of my DataSet. When I iterate through the columns I need to specify what datatype each column will be in this new table I generate with SQL. For the columns that should be varbinary my DataColumn data type is String and is indistinguishable from actual string columns like 'Username'
As #eulerfx mentioned you can leave it that way. But if you absolutely need an array use a int[] instead of byte[]. I needed to present the array of numbers as an array of numbers in other contexts (UI) and this worked for me.