Develop Reference

How do I get ExecuteReader() to efficiently add items to a List?

I have a button in the UI that reveals all remaining records in a table.
To handle this, in my controller, I have a simple SQL SELECT * statement with a LIMIT and OFFSET. My ExecuteReader is currently returning my data from the SQL command, which I am adding to a List. The list contains instances of my custom Run class.
In SSMS, the SQL query executes without exception no matter how large of a LIMIT I request. If my limit is > the number of rows in the table, it just returns all rows.
In webAPI, though, when my limit is > 200, it returns an exception. Otherwise, when less than 200, it returns a List of Runs without exception. I'm trying to debug the exception that occurs when I try to return all the data, but when it passes to the catch block, the exception is null. Which is weird.
So, I think there is a step I'm missing. Maybe I shouldn't be transforming the data into the Run class while the Reader is streaming. If I verified that the SQL command is accurate, then this seems to be the step that is causing the bug. Maybe transforming the data is making the Reader sorta time out? I don't understand ExecuteReader well enough to be able to figure out how I can pass all the data to List and then transform the data in that list into Runs after closing the connection. And don't even know if that would solve problem anyway.
All misgivings about potential SQL injections and lack of dbContext, etc. aside, how can I return all my records from the database utilizing ExecuteReader()?
Thanks.
Edit to add:
My exception value in the catch block is {"Data is Null. This method or property cannot be called on Null values."}.
In the debugger output, I my exception Exception thrown: 'System.Data.SqlTypes.SqlNullValueException' in Microsoft.Data.SqlClient.dll.
Edit to comment on the solution.
Ann L. figured this out. I had null values coming from the database. I learned from her and PSGuy that I can check for null values by using DbNull. Thank you!
Note - an easy place to get tripped up is that your class has to allow for nulls or else VS won't allow you to check for nulls in the method in the controller.

Here's one approach to the syntax you'll need to use (although there are lots of other approaches: see here for a bunch of alternatives!)
shoeAge = reader.IsDBNull(13) ? null : reader.GetInt64(13)
This assumes shoeAge is a nullable Int64. If it isn't, you'll get another error since you won't be able to assign null to it.

TSQL BULK INSERT row errors cause C# exceptions

I am processing extremely large delimited files. These files have been pre-processed to ensure that field and row delimiters are valid. Occasionally a row is processed that fails TSQL constraints (usually a datatype issue). 'Fixing' the input data is not an option in this case.
We use MAXERRORS to set an acceptable number of input errors and ERRORFILE to log failed rows.
The bulk insert completes in SSMS with severity level 16 error messages logged to the messages window for each failed row. Attempting to execute this code via the C# SqlCommand class causes an exception to be thrown when the first severity level 16 error message is generated, causing the batch to fail.
Is there a way to complete the operation and ignore SQL error messages via C# and something like SqlCommand?
Example Command:
BULK INSERT #some-table FROM 'filename'
WITH(FIELDTERMINATOR ='\0',ROWTERMINATOR ='\n',FIRSTROW = 2, MAXERRORS = 100, ERRORFILE = 'some-file')

Why not use SqlBulkCopy, and then capture the rows copied using the SqlRowsCopied event. This will more closely mimic the BULK INSERT T-SQL command.
EDIT: It looks like the error handling isn't that robust with SqlBulkCopy. However, here is an example that seems to do what you're looking for:
http://www.codeproject.com/Articles/387465/Retrieving-failed-records-after-an-SqlBulkCopy-exc

Since .NET support all the datatype as SQL you should be able to TryParse in .NET to catch any conversion errors. On date you also need to test for within the SQL data range. On text need to test length. I do exactly this on some very large inserts were I parse down some CSV. TryParse is pretty fast. Better than a Try Catch as it does not have the overhead of throwing and error.
And why not insert in .NET C#. There is a class for bulkcopy. I use TVP asynch insert while parsing and do 10,000 at a time.

It appears that an exception is thrown for each row that has an error but is just counted by SQL. However, it is also passed back to C# (SSIS in my case). I found that wrapping the bulk insert with TRY/CATCH logic and using THROW (to re-throw the exceptions) when more than MAXERRORS occur, works for me.
BEGIN TRY
BULK INSERT #some-table FROM 'filename'
WITH(FIELDTERMINATOR ='\0',ROWTERMINATOR ='\n',FIRSTROW = 2, MAXERRORS = 100,
ERRORFILE = 'some-file')
END TRY
BEGIN CATCH
THROW;
END CATCH

Input array is longer than the number of columns in this table

I've recently started to use SQLite and began to integrate it into a C# project I'm working on.
However, randomly my project will throw the exception:
Input array is longer than the number of columns in this table
I'm having a hard time trying the trace the problem because it seems to be thrown on a random basis.
DataTable table = new DataTable();
//exception is thrown here
table = Global.db.ExecuteQuery("SELECT * FROM vm_manager");
Some of the data that gets returned from this query is as follows:
http://i.stack.imgur.com/9rlLN.png
If anyone has any advice, I'd be grateful.
EDIT: I'm unable to show the execute query function as it resides inside a dll from the following sql lite wrapper http://www.codeproject.com/KB/database/cs_sqlitewrapper.aspx
EDIT 2 Problem stems from the new record array function inside this particular sql lite wrapper.

Based on your SQLite wrapper's implementation, it is adding columns to its own internal DataTable before returning to yours. I suspect your defect is in the wrapper, and not in your code. I dug into the source of your SQLiteWrapper from CodeProject; here it is at PasteBin: http://pastebin.com/AjGaX0kL
I suspect the error is occurring in that helper method ExecuteQuery(), or its helpers: ReadFirstRow() or ReadFirstRow(), and not your code. Wrap your code in a try catch. Inspect the Exception, and the properties will tell you which method this exception is actually being created in.
Likely you're encountering a defect in this wrapper class.
try
{
DataTable table = Global.db.ExecuteQuery("SELECT * FROM vm_manager");
}
catch (Exception ex){
//who threw this from which method and line?
}
If this SQLite provider isn't working, suggest evaluating:
System.Data.SQLite - An open source ADO.NET provider for the SQLite database engine
ADO.NET 2.0 Provider for SQLite at SourceForge -- and here's a nice tutorial by Mike Duncan on this SQLite provider.

PL/SQL Exceptions and Errors Handling

I'm doing a project in Asp .Net and C#. I'm also using ODP .NET to connect to my Oracle DB.
I am using a stored procedure to insert a value into the database. Everything is fine, it also raises all the exceptions I have.
I have 3 exceptions:
when the value I try to insert already exists in the database
when i try to insert a null value
Default "Others"
When the first two exceptions are raised I insert the error into a table Errors, everything goes well.
What I really want to know is, how can I show a message to the user when a Exception occurs?
Something like dbms_output.put_line("Error");...but I want it to show in my webpage. Is this possible?
Any tips are welcome, thanks in advance.

Since your .NET program is your client, you should let any unhandled exceptions propagate from your PL/SQL program and back to the client, where you can deal with it like any other exception.
In other words, you should remove the "when others" exception from your PL/SQL code, and instead wrap your database call (using ODP.NET) with a C# exception block. There you can catch the exception and get the Oracle error number and text, and display it to the user if you want.
(Using this approach you can also use RAISE_APPLICATION_ERROR in your PL/SQL code to signal errors back to the C# client.)

You can fix the stored procedure to return an integer, 0 means ok, 1 means exception of type 1, 2 exception of type 2 and so on... in this way the UI can notify the user that saving method did not go well as expected.
Normally I would not have handled the exception in the stored procedure but I understand that you want to log the error from SQL so the above way should allow you to do what you want.

In our sql server stored procedures we have something like this:
IF ##ERROR <> 0
return (1)
ELSE
return (0)
but it depends on what the stored does and what else it returns, if the stored returns a table, you can then use output parameter to send back the integer discussed above.

why does entity framework+mysql provider enumeration returns partial results with no exceptions

I'm trying to make sense of a situation I have using entity framework on .net 3.5 sp1 + MySQL 6.1.2.0 as the provider. It involves the following code:
Response.Write("Products: " + plist.Count() + "<br />");
var total = 0;
foreach (var p in plist)
{
//... some actions
total++;
//... other actions
}
Response.Write("Total Products Checked: " + total + "<br />");
Basically the total products is varying on each run, and it isn't matching the full total in plist. Its varies widely, from ~ 1/5th to half.
There isn't any control flow code inside the foreach i.e. no break, continue, try/catch, conditions around total++, anything that could affect the count. As confirmation, there are other totals captured inside the loop related to the actions, and those match the lower and higher total runs.
I don't find any reason to the above, other than something in entity framework or the mysql provider that causes it to end the foreach when retrieving an item.
The body of the foreach can have some good variation in time, as the actions involve file & network access, my best shot at the time is that when the .net code takes beyond certain threshold there is some type of timeout in the underlying framework/provider and instead of causing an exception it is silently reporting no more items for enumeration.
Can anyone give some light in the above scenario and/or confirm if the entity framework/mysql provider has the above behavior?
Update 1: I can't reproduce the behavior by using Thread.Sleep in a simple foreach in a test project, not sure where else to look for this weird behavior :(.
Update 2: in the example above the .Count() always returns the same + correct amount of items. Using ToList or ToArray as suggested gets around of the issue as expected (no flow control statements in the foreach body) and both counts match + don't vary on each run.
What I'm interested in is what causes this behavior in entity framework + mysql. Would really prefer not having to change the code in all the projects that use entity framework + mysql to do .ToArray before enumerating the results because I don't know when it'll swallow some results. Or if I do it, at least know what/why it happened.

If the problem is related to the provider or whatever, then you can solve/identify that by realising the enumerable before you iterate over it:
var realisedList = plist.ToArray();
foreach(var p in realisedList)
{
//as per your example
}
If, after doing this, the problem still persists then
a) One of the actions in the enumerator is causing an exception that is getting swallowed somewhere
b) The underlying data really is different every time.
UPDATE: (as per your comment)
[deleted - multiple enumerations stuff as per your comment]
At the end of the day - I'd be putting the ToArray() call in to have the problem fixed in this case (if the Count() method is required to get a total, then just change it to .Length on the array that's constructed).
Perhaps MySql is killing the connection while you're enumerating, and doesn't throw an error to EF when the next MoveNext() is called. EF then just dutifully responds by saying that the enumerable is simply finished. If so, until such a bug in the provider is fixed, the ToArray() is the way forward.

I think actually that you hit on the answer in your question, but it may be the data that is causing the problem not the timeout. Here is the theory:
One (or several) row(s) in the result set has some data that causes an exception / problem, when it hits that row the system thinks that it has reached the last row.
To test this you could try:
Ordering the data and see if the number returned in the for each statement is the same each time.
Select only the id column and see if the problem goes away
Remove all rows from the table, add them back a few at a time to see if a specific row is causing the problem
If it is a timeout problem, have you tried changing the timeout in the connection string.

I believe it has to do with the way the EF handles lazy loading. You might have to use either Load() or Include() and also check using IsLoaded property within your processing loop. Check out these two links for more information:
http://www.singingeels.com/Articles/Entity_Framework_and_Lazy_Loading.aspx
http://blogs.msdn.com/jkowalski/archive/2008/05/12/transparent-lazy-loading-for-entity-framework-part-1.aspx
I apologize I don't know more about EF to be more specific. Hopefully the links will provide enough info to get you started and others can chime in with any questions you might have.

The issue, cause and workaround is described exactly in this mysql bug.
As suspected it Is a timeout related error in the provider, but its not the regular timeout i.e. net_write_timeout. That's why the simple reproduction in a test project didn't work, since the timeout relates to All the cycles of the foreach and not just a particularly long body between the read of 2 rows.
As of now, the issue is present in the latest version of the MySql provider and under normal conditions would only affect scenarios where rows are being read with a connection maintained for a long time (which might or not involve a slow query). This is great, because it doesn't affect all of the previous projects where I have used MySql / applying the workaround to the sources also means it doesn't fail silently.
Ps. couple of what seem to be related mysql bugs: 1, 2