I am trying to get the data from a MySQL table and storing it into a dictionary. I know I can do that with a loop but DB table contains more than a million tuples and will slow the progress. Is there any way to do that without having to loop all the entries?
DB table has two columns, 1st(key) float type and 2nd(value) varchar.
My Current approach:
public static Dictionary<Single, string> GetData()
{
Dictionary<Single, string> dic= new Dictionary<Single, string>();
string query = "select * from table;";
if (OpenConnection() == true)
{
try
{
MySqlCommand cmd = new MySqlCommand(query, connection);
MySqlDataReader reader = cmd.ExecuteReader();
if (reader.Read())
{
//Something to store DB columns into dictionary without having to loop.
}
reader.Close();
}
catch (Exception e)
{
Console.WriteLine(e.Message);
}
}
return dic;
}
Next is what you want:
try
{
MySqlCommand cmd = new MySqlCommand(query, connection);
MySqlDataAdapter msda = new MySqlDataAdapter(cmd);
DataSet ds = new DataSet();
msda.Fill(ds);
your_control.DataSource = ds;
your_control.DataBind();
//your control means some repeater, or gridview, or anything else, which may show all your elements together.
}
But you also should consider that if you have millions of records, it would be better to make them selected partially. For example, 100 records per page, using row_number or similar methods
Related
Using a Microsoft Access database for a Web App Quiz Manager, I have table with a ID column that has a list of IDs which looks something like this:
ID Answer QuesDescription QuesAnswer QuestionNum
1 1 Example Example 1
3 3 Example Example 2
4 4 Example Example 3
6 1 Example Example 4
Using the query SELECT ID FROM (QuizName) with OleDbCommand I managed to get the ID values from the database and stored into OleDbDataReader reader. But i don't know how to get the ID values from the reader and store them as a String List. Does anyone know how to do this?
I've tried using stuff like
public List<string> GetIDValueFromQuestionNumber(string quizNumber)
{
try
{
string strSQL = string.Concat("SELECT count(ID) as RowCount FROM ", quizNumber);
List<string> resourceNames = new List<string>();
using (OleDbConnection connection = new OleDbConnection(connectionString))
{
OleDbCommand command = new OleDbCommand(strSQL, connection);
connection.Open();
OleDbDataReader reader = command.ExecuteReader();
reader.Read();
int rowCount = (int)reader["RowCount"];
strSQL = string.Concat("SELECT ID FROM ", quizNumber);
command = new OleDbCommand(strSQL, connection);
using (reader = command.ExecuteReader())
{
while (reader.Read())
{
resourceNames.Add(" " + reader.GetString(0));
}
}
connection.Close();
for (int count = 0; count < rowCount; count++)
{
int value = (int)reader.GetValue(count);
resourceNames.Add(value.ToString());
}
}
return resourceNames;
}
catch (Exception e)
{
return null;
}
}
But to no luck.
I should note that these tables can vary in depth.
I suggest this approach.
Say a form - DataGridView to display our data.
And say a listbox to display the list of id that you build up into that List
So, this form:
And the button click code:
private void button1_Click(object sender, EventArgs e)
{
// load up our data list with Hotels
string strSQL =
#"SELECT ID, FirstName, LastName, City, HotelName
FROM tblHotelsA ORDER BY HotelName";
DataTable rstData = MyRst(strSQL);
dataGridView1.DataSource = rstData;
// now build up a list of id in to string colleciton
List<string> MyIDList = new List<string>();
foreach (DataRow MyOneRow in rstData.Rows)
{
MyIDList.Add(MyOneRow["ID"].ToString());
}
// Lets set the id list to a listbox
listBox1.DataSource = MyIDList;
}
DataTable MyRst(string strSQL)
{
DataTable rstData = new DataTable();
using (OleDbConnection conn = new OleDbConnection(Properties.Settings.Default.AccessDB))
{
using (OleDbCommand cmdSQL = new OleDbCommand(strSQL, conn))
{
conn.Open();
rstData.Load(cmdSQL.ExecuteReader());
}
}
return rstData;
}
And now we get/see this:
So, pull the table. Display it, do whatever.
Then use the SAME table, and simple loop each row, grab the ID and add to your list.
And of course, one would probably hide the "id" in the above list (just add the columns using edit columns - only add the ones you want). You can still get/grab/use ANY column from the data source - it not a requirement to display such columns.
I have this DataSet which calls my Stored Procedure and returns a list of integers. How can I extract the list of integers which I could store in a variable be it a collection which grows in size such as a List<T> or a primitive data type like an array of integers.
Below is my code:
private DataSet getSubGroupsBelongingToUser()
{
DataTable variable;
DataSet DS;
myConnectionString = ConfigurationManager.ConnectionStrings["FSK_ServiceMonitor_Users_Management.Properties.Settings.FSK_ServiceMonitorConnectionString"].ConnectionString;
using (mySQLConnection = new SqlConnection(myConnectionString))
{
SqlParameter param = new SqlParameter("#UserId", getUserID(cbxSelectUser.Text));
DS = GetData("Test", param);
variable = DS.Tables[0];
}
return DS;
}
When I hover over the DS magnifier (refer to image):
I want to retrieve and store that list of integers somewhere. How can I go about doing this? All the examples I came across online make use of linq and that is not applicable here since I am getting the results from my stored procedure which requires one input parameter. Here is the definition of the stored procedure below:
create proc [dbo].[Test]
#UserId smallint
as
begin
select DepartmentSubGroupId from DepartmentSubGroupUser
where UserId= #UserId
end
GO
So essentially when you pass in a UserId, you should get those values. I am using SQL Server as my DBMS.
The simplest and most efficient approach would be to not use a DataSet/DataTable at all:
private List<int> GetSubGroupsBelongingToUser()
{
List<int> list = new List<int>();
using (var con = new SqlConnection(ConfigurationManager.ConnectionStrings["FSK_ServiceMonitor_Users_Management.Properties.Settings.FSK_ServiceMonitorConnectionString"].ConnectionString))
using (var cmd = new SqlCommand("Test", con))
{
cmd.CommandType = CommandType.StoredProcedure;
var param = new SqlParameter("#UserId", SqlDbType.Int).Value = int.Parse(cbxSelectUser.Text);
cmd.Parameters.Add(param);
con.Open();
using (var rd = cmd.ExecuteReader())
{
while (rd.Read()) list.Add(rd.GetInt32(0));
}
} // no need to close the connection with the using
return list;
}
If you insist on the DataTable, at least it's more conscise:
return DS.Tables[0].AsEnumerable().Select(r => r.Field<int>(0)).ToList();
As #David pointed out the simplest option would be to use a SqlDataReader and loop through all the records.
However, if your heart is set on DataTables then all you need to do is to iterate all rows from the result table, grab the value from column DepartmentSubGroupId and add it to a list. You can do that with Linq also like this:
return DS.Tables[0].Rows
.Cast<DataRow>() // Rows is an ICollection and you need to cast each item
.Select(r => (int)r["DepartmentSubGroupId"]) // For each row get the value from column DepartmentSubGroupId
.ToList();
Before I saw #Tim's solution, I had already prepared this (which works just as well as most of the above solutions):
public List<int> getSubGroupsBelongingToUser()
{
List<int> DepartmentSubGroupIds = new List<int>();
myConnectionString = ConfigurationManager.ConnectionStrings["FSK_ServiceMonitor_Users_Management.Properties.Settings.FSK_ServiceMonitorConnectionString"].ConnectionString;
using (mySQLConnection = new SqlConnection(myConnectionString))
{
SqlParameter parameter = new SqlParameter("#UserId", getUserID(cbxSelectUser.Text));
mySQLCommand = new SqlCommand("Test", mySQLConnection);
mySQLCommand.CommandType = CommandType.StoredProcedure;
mySQLCommand.Parameters.Add(parameter);
mySQLConnection.ConnectionString = myConnectionString;
mySQLConnection.Open();
SqlDataReader sqlDataReader = mySQLCommand.ExecuteReader();
while (sqlDataReader.Read())
{
DepartmentSubGroupIds.Add(Convert.ToInt32(sqlDataReader["DepartmentSubGroupId"]));
}
}
return DepartmentSubGroupIds;
}
Thanks everybody, much appreciated.
I have a SQL Server stored procedure that I run two select statements. I can easily return one select statement and store it in a DataTable but how do I use two?
How can I set my variable countfromfirstselectstatement equal to the count returned from my first select statement and how can I set my variable countfromsecondselectstatement equal to the count returned from the second select.
private void ReturnTwoSelectStatements()
{
DataSet dt112 = new DataSet();
using (var con = new SqlConnection(connectionString))
{
using (var cmd = new SqlCommand("Return2SelectStatements", con))
{
using (var da = new SqlDataAdapter(cmd))
{
cmd.CommandType = CommandType.StoredProcedure;
da.Fill(dt112);
}
}
}
DataTable table1 = dt112.Tables[0];
DataTable table2 = dt112.Tables[1];
string countfromFirstSelectStatement = table1.Rows.Count().ToString();
string countfromSecondSelectStatement = table2.Rows.Count().ToString();
//I only want to iterate the data from the 1st select statement
foreach (DataRow row in dt112.Rows)
{
}
}
You could also directly use the DbDataReader (cmd.ExecuteReader()), which gives you NextResult to advance to the next result set. DataTable.Load allows you to load rows from the data reader to the table.
DbDataAdapter is really a bit of an overkill for just reading data into a data table. It's designed to allow the whole CRUD-breadth of operation for controls that are abstracted away from the real source of data, which isn't really your case.
Sample:
using (var reader = cmd.ExecuteReader())
{
dataTable1.Load(reader);
if (!reader.NextResult()) throw SomethingWhenTheresNoSecondResultSet();
dataTable2.Load(reader);
}
Fill a Dataset with both select statements and access it:
....
DataSet dataSet = new DataSet();
da.Fill(dataSet);
....
DataTable table1 = dataSet.Tables[0];
DataTable table2 = dataSet.Tables[1];
string countfromFirstSelectStatement = table1.Rows.Count.ToString();
string countfromSecondSelectStatement = table2.Rows.Count.ToString();
I'm having some performance issues when I return a record set with more than 1,000 records.
Sometimes the records are in upwards of 2,100 but can be as low as 10.
I have some bulk actions that I take on all the records by selecting them.
However, when the number is low, the Gridview is fine. When the record count is greater than 500 I see performance issues on the page.
What I want to happen is: if there are more than 500 records, DO NOT DISPLAY THE GRID, instead show a download button that exports to CSV or do other control things on the page.
My issue:
Even if i tell it not to display the grid and instead display a message and a button, the performance is still slow.
Below is my C# code for populating the GridView. Some stuff has been removed that are unimportant and to help with readability.
How can I adjust my C# code for better performance?
SqlConnection conn = new SqlConnection(ConfigurationManager.AppSettings["ConnectString"].ToString());
SqlCommand cmd = conn.CreateCommand();
cmd.CommandType = CommandType.StoredProcedure;
cmd.CommandText = "SomeProcedure";
cmd.Parameters.Add(SearchParam);
try {
DataTable GridData = new DataTable();
conn.Open();
using(SqlDataAdapter Sqlda = new SqlDataAdapter(cmd)) {
Sqlda.Fill(GridData);
}
if (GridData.Rows.Count == 0) {
lblSearchMsg.Text = "No Fee Records are in the Queue at this time.";
} else {
if (GridData.Rows.Count > 500) {
lblSearchMsg.Text = "More than " + gridLimit.ToString() + " records returned.";
//Show the download button
} else {
//Persist the table in the Session object. (for sorting)
Session["GridData"] = GridData;
lblRowCount.Text = "Count: " + GridData.Rows.Count.ToString();
myGridView.DataSource = GridData;
myGridView.DataBind();
myGridView.Visible = true;
}
}
} catch (Exception ex) {
//Do the error stuff
} finally {
if (conn != null) {
conn.Close();
}
}
Create a separate procedure that returns only the row count.
Check that value not the row count of a fully retrieved data set then retrieve the full data set as needed.
Keep in mind you can use the same connection to do both retrievals, no need to close the connection between calls.
if you determine you need to fill a gridview and there is no need to edit the data you can read into the DataTable without the use of an adapter. Here is the basic idea modify with using statements or try/catch as you prefer:
conn = new SqlConnection(connString);
string query = "SELECT * FROM ....";
SqlCommand cmd = new SqlCommand(query, conn);
conn.Open();
SqlDataReader dr = cmd.ExecuteReader();
DataTable dt = new DataTable();
dt.Load(dr);
GridView1.DataSource = dt;
GridView1.DataBind();
Can anybody help me with the issue I'm seeing? For some reason when I run my page, I get my drop down lists to populate the data, however the first item in my database, per each SQL query, doesn't get populated.
For example, my database table is:
Category
1 Books
2 Clothing
3 Toys
4 Household Items
my first query -
SELECT Category FROM ProductCategories
my drop down list gets populated with
Clothing
Toys
Household Items
I have 2 other drop down lists I'm populating and those are doing the same thing. Once I get this figured out, I'll try to figure out the other problem I'm having with inserting the data in the database.
Thank you!
public partial class InsertItems : System.Web.UI.Page
{
protected void Page_Load(object sender, EventArgs e)
{
SqlConnection connection;
SqlCommand populateList;
SqlDataReader reader;
string connectionString = ConfigurationManager.ConnectionStrings["LakerBids"].ConnectionString;
connection = new SqlConnection(connectionString);
populateList = new SqlCommand("USE LakerBids SELECT Category FROM ProductCategories;" +
"USE LakerBids SELECT SubCategory FROM ProductSubCategories;" +
"USE LakerBids SELECT LName FROM Users", connection);
if (!IsPostBack)
{
try
{
connection.Open();
reader = populateList.ExecuteReader();
while (reader.Read())
{
pcategory.DataSource = reader;
pcategory.DataValueField = "Category";
pcategory.DataBind();
}
reader.NextResult();
while (reader.Read())
{
psubcategory.DataSource = reader;
psubcategory.DataValueField = "SubCategory";
psubcategory.DataBind();
}
reader.NextResult();
while (reader.Read())
{
user.DataSource = reader;
user.DataValueField = "LName";
user.DataBind();
}
reader.Close();
}
finally
{
connection.Close();
}
}
}
protected void AddItem(object sender, EventArgs e)
{
if (Page.IsValid)
{
SqlConnection connection;
SqlCommand insertData;
string connectionString = ConfigurationManager.ConnectionStrings["LakerBids"].ConnectionString;
connection = new SqlConnection(connectionString);
insertData = new SqlCommand("INSERT INTO Products (ProductName, ProductDesc, CategoryID, SubCatID, StatusID, UserID, ReservePrice, AuctionLength, BidID)" +
"VALUES (#ProductName, #ProductDesc, #CategoryID, #SubCatID, 1, #UserID, #ReservePrice, #AuctionLength, NULL)", connection);
insertData.Parameters.Add("#ProductName", System.Data.SqlDbType.NVarChar, 50);
insertData.Parameters["#ProductName"].Value = pname.Text;
insertData.Parameters.Add("#ProductDesc", System.Data.SqlDbType.NVarChar, 200);
insertData.Parameters["#ProductDesc"].Value = pdesc.Text;
insertData.Parameters.Add("#CategoryID", System.Data.SqlDbType.Int);
insertData.Parameters["#CategoryID"].Value = pcategory.SelectedIndex;
insertData.Parameters.Add("#SubCatID", System.Data.SqlDbType.Int);
insertData.Parameters["#SubCatID"].Value = psubcategory.SelectedIndex;
insertData.Parameters.Add("#UserID", System.Data.SqlDbType.Int);
insertData.Parameters["#UserID"].Value = user.SelectedIndex + 2;
insertData.Parameters.Add("#ReservePrice", System.Data.SqlDbType.Money);
insertData.Parameters["#ReservePrice"].Value = Convert.ToDecimal(reserveprice.Text);
insertData.Parameters.Add("#AuctionLength", System.Data.SqlDbType.Int);
insertData.Parameters["#AuctionLength"].Value = Convert.ToInt32(auctionlength.Text);
try
{
connection.Open();
insertData.ExecuteNonQuery();
Response.Redirect("Categories.aspx");
}
catch (Exception error)
{
dberror.Text = error.ToString();
}
finally
{
connection.Close();
}
}
}
}
You need to either use a DataSet or populate business entities within a collection and then bind to the collection.
List<Category> cats = new List<Category>();
while (reader.Read())
{
Category cat = new Category();
// fill properties from DataReader
cats.Add(cat);
}
pcategory.DataSource = cats;
pcategory.DataValueField = "Category";
pcategory.DataBind();
Sab0tr0n, I suspect one of two things are happening.
1) If you are saying the first item does not appear AFTER you do some kind of "Add Category" action, then it might be that the dropdown is populated BEFORE the insert completes. Meaning, you need to requery after allowing the insert to be committed to the database.
OR
2) Put a breakpoint on this line:
string connectionString = ConfigurationManager.ConnectionStrings["LakerBids"].ConnectionString;
Then confirm the connectionString is to the correct database. I've seen old config files that point to test or staging databases cause this kind of confusion.
Good luck. If these aren't the answer, maybe simplify your example to us or elaborate on exactly what you do with your application and when you see the problem.
The "reader.read()" statement in each block is actually reading the first row of data, so when you set the DataSource, the first row has already been read. Try taking it out
You should use "while (reader.Read())" if you want to iterate over each result, not bind the resulset in one step.
That being said, the comments about using a Dataset, seperating logic, etc., are valid