I'm hoping you guys can help me figure out why this is happening. I've been tearing my hair out trying to figure this out.
Here's an example directly from my code (with the boring bits cut out)
...(Set up the connection and command, initialize a datatable "dataTable")...
using (SqlDataReader reader = cmd.ExecuteReader())
{
//Query storage object
object[] buffer = new object[reader.FieldCount];
//Set the datatable schema to the schema from the query
dataTable = reader.GetSchemaTable();
//Read the query
while (reader.Read())
{
reader.GetValues(buffer);
dataTable.Rows.Add(buffer);
}
}
The error is
Input string was not in a correct format.Couldn't store in NumericScale Column. Expected type is Int16.
The specific column data types as returned by the schema are (ordered by column)
System.Data.SqlTypes.SqlInt32
System.Data.SqlTypes.SqlInt32
System.Data.SqlTypes.SqlByte
System.Data.SqlTypes.SqlMoney
System.Data.SqlTypes.SqlString
System.Data.SqlTypes.SqlGuid
System.Data.SqlTypes.SqlDateTime
It would appear that the data that should be in column #5 is actually appearing in column #3. But that is pure speculation.
What I know is that in order to use a dataTable "dynamically" with a query that can continue any number of different types of data the best route is to use GetSchemaTable() to retrieve it.
What I Saw In The Debugger
When I dropped into the debugger I took a look at dataTable's types built from the schema vs. the types returned to the object from reader.GetValues(). They are exactly the same.
It seems like dataTable.Rows.Add(buffer) is adding the columns a few columns off from where it should be. But this shouldn't be possible. Especially with the schema being directly built from the reader. I've played with options such as "CommandBehavior.KeyInfo" within ExecuteReader() and still had the same error occur.
Note: I need to run the query this way to enable the end-user to halt the query mid-read. Please do not suggest I scrap this and use an SqlDataAdapter or DataTable.Load() solution.
I'd really appreciate any help. Thank you!
Method DbDataReader.GetSchemaTable() return metadata table containing column names and types. It is not an empty table that one could expect. For more details see MSDN
I'm sorry but GetSchemaTable retrieves schema for a given table and GetValues retrieves the actual row data. E.g. your dataTable will contain columns like column name, column type, etc. (see MSDN reference) while your buffer will contain the actual data whose representation will differ from what the dataTable contains.
Why don't you just use:
dataTable.Fill();
Why are you manually loading it one row at a time?
Please check what you are inserting into the column row which gas GUID data type. I got the same error and found that while inserting records after reading records from csv file, there were couple of empty spaces in csv file.
Related
TL;DR In a SSRS 2008 report which uses a custom assembly to do some extra calculations can I pass an entire report dataset as a method parameter?
Full story
I have an SSRS report with 3 datasets, each returned from an SQL query.
(In case it makes a difference to my question they're currently shared datasets although I'm sure local would work too)
The largest and primary dataset is a list of tasks which may or may not have been completed. I have information in here such as the ID, status, create date/time, target resolution hours etc of each task.
This dataset is displayed in a tablix and is the focus of the report.
The remaining two datasets are not displayed and are for reference. One is a simple one column query which returns a list of holiday dates for the UK. The other is a small table which contains our exact business hours.
At the moment I'm able to loop through the rows in the tablix of tasks and pass multiple values from the current row to a method. This is useful if I want to do some calculations based on data found only in the current row. For example I could take the create date/time and the response target hours and the assembly would return a target date/time for the current task. Cool so far.
I want to do a more complicated version of this where I not only pass in the row data but the 2 other datasets to get my return value. This is because in reality the due date calculation is much more complex and must take into account changing business hours and holidays from the other 2 datasets.
Can I pass a dataset as a method parameter to an assembly? Something like:
=Code.MyClass.MyMethod(val1, val2, dataset1, dataset2);.
I've been unable to find much definitive information on this. Nearly all tutorials demonstrate what I'm already doing by processing single rows. I'm sure I had an MSDN article that hinted this was not possible but I've lost it (helpful I know). There's a post on the Microsoft forums where a moderator says it's not possible. The general lack of information and tutorials suggests it's not possible or I'm doing this in the wrong way.
Any suggestions?
(I have alternate solutions such as having the assembly fetch the other datasets or just writing something outside SSRS but I'm not pursuing those until I knnow whether it can be done this way).
An older topic on the MSDN forums Iterate through rows of dataset in report's custom code offers a more definitive answer as well as a potential solution to this problem.
Passing the DataSet as an object or collection is not a possibility because:
A dataset in Reporting Services is not the same type of object as an ADO.Net dataset. A report dataset is an internal object managed by the SSRS runtime (it's actually derived from a DataReader object) and not an XML structure containing datatables, etc. and cannot be passed into the report's custom code.
The only way to effectively loop through the rows of a report dataset is to call a custom function or referenced method in a report data region expression. Using this technique, it may be possible to pass all of the the row and field information into a code structure, array or collection.
The hint given in the above statement suggests passing row and field information into a code structure. A contributor to the linked MSDN topic, Migeul Catalao developed a workaround using such an approach.
A real-world scenario of it's usage with example code demonstrating Migeul Catalao's solution can be found here.
Granted, it is still more of a row-by-row approach, so I would strongly suggest moving outside of SSRS and pursue alternative solutions.
Although I've accepted the other answer due to it being clear and helpful I didn't use that solution in the end (I was too stupid to understand it) and went for something else that works.
Disclaimer: This is a horrible hack. It works absolutely great in my scenario so I though I'd share in case it was useful to somebody else. There are many pitfalls here which could most likely be worked around given time.
I ended up following the advice in the comment given by Steven White and looking into LookupSet. This function allows you to query a dataset to return matching rows and a single column of data.
It looks like this:
LookupSet(Fields!ComparisonField.Value, // The value to search for, e.g '001'.
Fields!MatchField.Value, // The column to match on in the target dataset.
Fields!MyColumn.Value, // The column that will be returned.
"MyDataSet") // The dataset to search.
This returns a string array representing the returned values.
So far so good, but I needed ALL columns and rows. This is where the dirty hack appears in the form of string concatenation:
LookupSet(0, // Dummy ID 0.
0, // Matches the dummy ID 0 so all rows are returned.
Fields!Column1.Value + "[^]" // I concatenate all of the values into
+ Fields!Column2.Value + "[^]" // one string with the separator [^]
+ Fields!.Column3.Value, // so I can split them later.
"MyDataSet") // The dataset to query
I can now pass this to my custom assembly:
=MyAssemblyNamespace.Class.Method(LookupSet(0,0,Fields!Column1.Value..., "MyDataSet"), other, parameters, here)
Now in my C# method I have a generic object which after some reflection is actually an array of strings.
Cast to something useful:
var stringList = ((IEnumerable)MyDataSetObject).Cast<string>().ToList();
Split it:
foreach (var item in stringList)
{
var columns = item.Split(new[] { "[^]" }, StringSplitOptions.None);
// columns is a string[] which holds each column value for the current row
// So columns[0] is the value for column 1 in this row
// In my case I pushed the values to a DataTable row each time and built a datatable
// which when finished represented my dataset in full with all rows and columns.
}
I hope this makes sense to anyone trying to achieve a similar result.
I'm trying to populate a DataTable, to build a LocalReport, using the following:
MySqlCommand cmd = new MySqlCommand();
cmd.Connection = new MySqlConnection(Properties.Settings.Default.dbConnectionString);
cmd.CommandType = CommandType.Text;
cmd.CommandText = "SELECT ... LEFT JOIN ... WHERE ..."; /* query snipped */
// prepare data
dataTable.Clear();
cn.Open();
// fill datatable
dt.Load(cmd.ExecuteReader());
// fill report
rds = new ReportDataSource("InvoicesDataSet_InvoiceTable",dt);
reportViewerLocal.LocalReport.DataSources.Clear();
reportViewerLocal.LocalReport.DataSources.Add(rds);
At one point I noticed that the report was incomplete and it was missing one record. I've changed a few conditions so that the query would return exactly two rows and... surprise: The report shows only one row instead of two. I've tried to debug it to find where the problem is and I got stuck at
dt.Load(cmd.ExecuteReader());
When I've noticed that the DataReader contains two records but the DataTable contains only one. By accident, I've added an ORDER BY clause to the query and noticed that this time the report showed correctly.
Apparently, the DataReader contains two rows but the DataTable only reads both of them if the SQL query string contains an ORDER BY (otherwise it only reads the last one). Can anyone explain why this is happening and how it can be fixed?
Edit:
When I first posted the question, I said it was skipping the first row; later I realized that it actually only read the last row and I've edited the text accordingly (at that time all the records were grouped in two rows and it appeared to skip the first when it actually only showed the last). This may be caused by the fact that it didn't have a unique identifier by which to distinguish between the rows returned by MySQL so adding the ORDER BY statement caused it to create a unique identifier for each row.
This is just a theory and I have nothing to support it, but all my tests seem to lead to the same result.
After fiddling around quite a bit I found that the DataTable.Load method expects a primary key column in the underlying data. If you read the documentation carefully, this becomes obvious, although it is not stated very explicitly.
If you have a column named "id" it seems to use that (which fixed it for me). Otherwise, it just seems to use the first column, whether it is unique or not, and overwrites rows with the same value in that column as they are being read. If you don't have a column named "id" and your first column isn't unique, I'd suggest trying to explicitly set the primary key column(s) of the datatable before loading the datareader.
Just in case anyone is having a similar problem as canceriens, I was using If DataReader.Read ... instead of If DataReader.HasRows to check existence before calling dt.load(DataReader) Doh!
I had same issue. I took hint from your blog and put up the ORDER BY clause in the query so that they could form together the unique key for all the records returned by query. It solved the problem. Kinda weird.
Don't use
dr.Read()
Because It moves the pointer to the next row.
Remove this line hope it will work.
Had the same issue. It is because the primary key on all the rows is the same. It's probably what's being used to key the results, and therefore it's just overwriting the same row over and over again.
Datatables.Load points to the fill method to understand how it works. This page states that it is primary key aware. Since primary keys can only occur once and are used as the keys for the row ...
"The Fill operation then adds the rows to destination DataTable objects in the DataSet, creating the DataTable objects if they do not already exist. When creating DataTable objects, the Fill operation normally creates only column name metadata. However, if the MissingSchemaAction property is set to AddWithKey, appropriate primary keys and constraints are also created." (http://msdn.microsoft.com/en-us/library/zxkb3c3d.aspx)
Came across this problem today.
Nothing in this thread fixed it unfortunately, but then I wrapped my SQL query in another SELECT statement and it work!
Eg:
SELECT * FROM (
SELECT ..... < YOUR NORMAL SQL STATEMENT HERE />
) allrecords
Strange....
Can you grab the actual query that is running from SQL profiler and try running it? It may not be what you expected.
Do you get the same result when using a SqlDataAdapter.Fill(dataTable)?
Have you tried different command behaviors on the reader? MSDN Docs
I know this is an old question, but for me the think that worked whilst querying an access database and noticing it was missing 1 row from query, was to change the following:-
if(dataset.read()) - Misses a row.
if(dataset.hasrows) - Missing row appears.
For anyone else that comes across this thread as I have, the answer regarding the DataTable being populated by a unique ID from MySql is correct.
However, if a table contains multiple unique IDs but only a single ID is returned from a MySql command (instead of receiving all Columns by using '*') then that DataTable will only organize by the single ID that was given and act as if a 'GROUP BY' was used in your query.
So in short, the DataReader will pull all records while the DataTable.Load() will only see the unique ID retrieved and use that to populate the DataTable thus skipping rows of information
Not sure why you're missing the row in the datatable, is it possible you need to close the reader? In any case, here is how I normally load reports and it works every time...
Dim deals As New DealsProvider()
Dim adapter As New ReportingDataTableAdapters.ReportDealsAdapter
Dim report As ReportingData.ReportDealsDataTable = deals.GetActiveDealsReport()
rptReports.LocalReport.DataSources.Add(New ReportDataSource("ActiveDeals_Data", report))
Curious to see if it still happens.
In my case neither ORDER BY, nor dt.AcceptChanges() is working. I dont know why is that problem for. I am having 50 records in database but it only shows 49 in the datatable. skipping first row, and if there is only one record in datareader it shows nothing at all.
what a bizzareeee.....
Have you tried calling dt.AcceptChanges() after the dt.Load(cmd.ExecuteReader()) call to see if that helps?
I know this is an old question, but I was experiencing the same problem and none of the workarounds mentioned here did help.
In my case, using an alias on the colum that is used as the PrimaryKey solved the issue.
So, instead of
SELECT a
, b
FROM table
I used
SELECT a as gurgleurp
, b
FROM table
and it worked.
I had the same problem.. do not used dataReader.Read() at all.. it will takes the pointer to the next row. Instead use directly datatable.load(dataReader).
Encountered the same problem, I have also tried selecting unique first column but the datatable still missing a row.
But selecting the first column(which is also unique) in group by solved the problem.
i.e
select uniqueData,.....
from mytable
group by uniqueData;
This solves the problem.
I believe in JAVA we can set the data reader to point to the 1st row of it's result set. I have a SQLDataReader that has over 20 columns and I need to pre-process some columns and calculate some values before I use the entire result set. Is it possible to re-read the data reader after reading it for the first time to get just selected columns? Or will I need to store the results in a data table and read from it later?
From the MSDN docs at http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqldatareader.aspx
Provides a way of reading a forward-only stream of rows from a SQL Server database.
You can read one row at a time, but you cannot go back to a row once you've read the next row.
You can read columns in a row in any order, any number of times, and skip fields if you want.
while (reader.Read())
{
var a = reader[2]; // you can skip fields
var b = reader[0]; // you don't have to read the fields in order
var c = reader[2]; // you can re-read fields
}
No it is not possible.
From SqlDataReader on MSDN:
Provides a way of reading a forward-only stream of rows from a SQL Server database.
(emphasis mine)
You can read all the needed data into a collection and iterate over the copy as often as you want.
Perhaps you could run 2 queries as part of an sproc. 1st query is limited to the fields for precalculated fields, the 2nd for the rest of the fields. Then in your C# make use of the reader.ReadNext method.
Alternatively SQL does support calculated fields, which might make things simpler.
But personally I'd read the whole data into a List<T> for processing...
I am stucked up at this point. I searched alot on gooogle but didnot find anything.
My problem is:
I have an Excel file which i want to export to datatable and from datatable i want to save it to oracle DB.
Excel file contains multiple columns and each column consists of large data(approx 20000characters/numbers).
using oledbconnection,excel columns with such large data are not copied to datatble.(Small data columns gets copied).
Can anyone suggest workaround to my problem???
Thanks in advance.
Check the dataypes and their length i.e nvarchar(3000)
If that doesnt work, test with a small set of data maybe 5 rows, you should be able to see a trend there.
Also check the data type of your application, sometimes for large number you may use long, or if they are large strings maybe use a stringbuilder to pass the data instead of just a string....
First, I want to say that I'm out on deep water here, since I'm just doing some changes to code that is written by someone else in the company, using OleDbDataAdapter to "talk" to Excel and I'm not familiar with that. There is one bug there I just can't follow.
I'm trying to use a OleDbDataAdapter to read in a excel file with around 450 lines.
In the code it's done like this:
connection = new OleDbConnection("Provider=Microsoft.Jet.OLEDB.4.0;" + "Data Source='" + path + "';" + "Extended Properties=\"Excel 8.0;HDR=Yes;IMEX=1;\"");
connection.Open();
OleDbDataAdapter objAdapter = new OleDbDataAdapter(objCommand.CommandText, connection);
objAdapter.Fill(objDataSet, "Excel");
foreach (DataColumn dataColumn in objTable.Columns) {
if (dataColumn.Ordinal > objDataSet.Tables[0].Columns.Count - 1) {
objDataSet.Tables[0].Columns.Add();
}
objDataSet.Tables[0].Columns[dataColumn.Ordinal].ColumnName = dataColumn.ColumnName;
objImport.Columns.Add(dataColumn.ColumnName);
}
foreach (DataRow dataRow in objDataSet.Tables[0].Rows) {
...
}
Everything seems to be working fine except for one thing. The second column is filled with mostly four digit numbers like 6739, 3920 and so one, but fice rows have alphanumeric values like 8201NO and 8205NO. Those five cells are reported as having blank contents instead of their alphanumeric content. I have checked in excel, and all the cells in this columns are marked as Text.
This is an xls file by the way, and not xlsx.
Do anyone have any clue as why these cells are shown as blank in the DataRow, but the numeric ones are shown fine? There are other columns with alphanumeric content that are shown just fine.
What's happening is that excel is trying to assign a data type to the spreadsheet column based on the first several values in that column. I suspect that if you look at the properties in that column it will say it is a numerical column.
The problem comes when you start trying to query that spreadsheet using jet. When it thinks it's dealing with a numerical column and it finds a varchar value it quietly returns nothing. Not even a cryptic error message to go off of.
As a possible work around can you move one of the alpha numeric values to the first row of data and then try parsing. I suspect you will start getting values for the alpha numeric rows then...
Take a look at this article. It goes into more detail on this issue. it also talks about a possible work around which is:
However, as per JET documentation, we
can override the registry setting thru
the Connection String, if we set
IMEX=1( as part of Extended
Properties), the JET will set the all
column type as UNICODE VARCHAR or
ADVARWCHAR irrespective of
‘ImportMixedTypes’ key value.hey
IMEX=1 means "Read mixed data as text."
There are some gotchas, however. Jet will only use several rows to determine whether the data is mixed, and if so happens these rows are all numeric, you'll get this behaviour.
See connectionstrings.com for details:
Check out the [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Jet\4.0\Engines\Excel] located registry REG_DWORD "TypeGuessRows". That's the key to not letting Excel use only the first 8 rows to guess the columns data type. Set this value to 0 to scan all rows. This might hurt performance. Please also note that adding the IMEX=1 option might cause the IMEX feature to set in after just 8 rows. Use IMEX=0 instead to be sure to force the registry TypeGuessRows=0 (scan all rows) to work.
I would advise against using the OleDb data provider stuff to access Excel if you can help it. I've had nothing but problems, for exactly the reasons that others have pointed out. The performance tends to be atrocious as well when you are dealing with large spreadsheets.
You might try this open source solution:
http://exceldatareader.codeplex.com/