Fastest way to Read huge volume of data from DB - c#

I have a table which contains around 500 Million records. I am reading the data from table and storing those in a Dictionary.
EDIT: I am loading the data into dictionary because these data needs to be compared with another volume of data coming from an indexing server.
My code is as below:
public static void GetDetailsFromDB()
{
string sqlStr = "SELECT ID, Name ,Age, email ,DOB ,Address ,Affiliation ,Interest ,Homepage FROM Author WITH (NOLOCK) ORDER BY ID";
SqlCommand cmd = new SqlCommand(sqlStr, _con);
cmd.CommandTimeout = 0;
using (SqlDataReader reader = cmd.ExecuteReader())
{
while (reader.Read())
{
//Author Class
Author author = new Author();
author.id = Convert.ToInt32(reader["ID"].ToString());
author.Name = reader["Name"].ToString().Trim();
author.age = Convert.ToInt32(reader["Age"].ToString());
author.email = reader["email"].ToString().Trim();
author.DOB = reader["DOB"].ToString().Trim();
author.Address = reader["Address"].ToString().Trim();
author.Affiliation = reader["Affiliation"].ToString().Trim();
author.Homepage = reader["Homepage"].ToString().Trim();
string interests = reader["Interest"].ToString().Trim();
author.interest = interests.Split(new char[] { ',' }, StringSplitOptions.RemoveEmptyEntries).Select(p => p.Trim()).ToList();
if (!AuthorDict.ContainsKey(author.id))
{
AuthorDict.Add(author.id, author);
}
if (AuthorDict.Count % 1000000 == 0)
{
Console.WriteLine("{0}M author loaded.", AuthorDict.Count / 1000000);
}
}
}
}
This process is taking long time to read and store all 500 Million records from DB. Also, the RAM usage is very high.
Can this be optimized ? also, can the running time be decreased ? any help is appreciated.

If I hold my nose I can come up with the following optimisations:
Store the ordinal positions of your fields in local variables and reference the fields in your reader using these ordinal variables.
Do not call ToString on the reader and convert - just get the value out in the correct type.
Check for the existence of the author id in the AuthorDict as soon as you have the id. Don't even create the Author instance if you don't need it.
using (SqlDataReader reader = cmd.ExecuteReader())
{
var idOrdinal = reader.GetOrdinal("ID");
//extract other ordinal positions and store here
while (reader.Read())
{
var id = reader.GetInt32(idOrdinal);
if (!AuthorDict.ContainsKey(id))
{
Author author = new Author();
author.id = reader.GetInt32(idOrdinal);
...
}
}
}

Related

How do I remove a line from the list based on the ID of that line?

I've been trying to figure this out for the past few days, but I just can't seem to get it work.
So I have a txt file which has this format:
id;könyvcím;szerző;kiadó;kiadási év;
I am using a structs and a list such as this:
public static List<Books> BooksList = new List<Books>();
public struct Books
{
public int id;
public string title;
public string writer;
public string publisher;
public int published_year;
}
And I'm also putting all these into a List based on the struct like this:
StreamReader booksRead = new StreamReader("konyvek.txt", Encoding.UTF8);
booksRead.ReadLine();
while (!booksRead.EndOfStream)
{
string[] split = booksRead.ReadLine().Split(';');
Books inRead = new Books();
inRead.id = Convert.ToInt32(split[0]);
inRead.title = split[1];
inRead.writer = split[2];
inRead.publisher = split[3];
inRead.published_year = Convert.ToInt32(split[4]);
BooksList.Add(inRead);
}
booksRead.Close();
All I want is, for example, to find where the line with ID 2 is, and remove that line from my textfile. I've tried to get the index of the line I want, and remove it like that from my textfile, but it even fails to get the index, I tried using IndexOf, FindIndex and trying to go on a loop. I'm pretty sure my struct is not happy with me for using it like that because I get errors such as this when I run my code:
System.InvalidCastException: 'Unable to cast object of type 'Books' to
type 'System.IConvertible'.'
Here is the way I'm trying to get the index of the line I want to remove
Books item = new Books();
for (int i = 0; i < BooksList.Count; i++)
{
if (Convert.ToInt32(textBox_id_delete.Text) == item.id)
{
RemoveAt = item.id;
}
}
int index = BooksList.FindIndex(x => Convert.ToInt32(x) == RemoveAt);
MessageBox.Show(Convert.ToString(index));
I'm pretty sure I'm approaching this extremely wrong, and I'd accept any kind of help.
You are doing it completely wrong for a number of reasons.
First, how would you do that the way you are doing:
void Main()
{
var filename = #"c:\myFolder\mybooklist.txt";
// read into an enumerable
var books = File.ReadAllLines(filename)
.Select(x => x.Split(';'))
.Select(x => new Book {
Id = int.TryParse(x[0], out int bookId)?bookId:0,
Title = x[1],
Writer = x[2],
Publisher = x[3],
Published_year=int.TryParse(x[4], out int year)?year:0
});
// remove the one with id 2
// and save back
var otherBooks = books.Where(b => b.Id != 2);
File.WriteAllLines(filename, otherBooks.Select(b => $"{b.Id};{b.Title};{b.Writer};{b.Publisher};{b.Published_year}"));
}
public struct Book
{
public int Id;
public string Title;
public string Writer;
public string Publisher;
public int Published_year;
}
And now what is wrong with this.
A text file is not a database but you are trying to use a text file as a database.
With a text file, you are not actually doing any control here, if the ID is unique or not (there might be N books with the ID 2).
(Side matter) You are using C#, but looks like you are coming from another language and not using the naming conventions at all.
IMHO, instead you should simply use a database, an embedded one for example like LiteDb or Sqlite. If you care to see a sample with LiteDb or Sqlite, let me know.
EDIT: I am adding SQLite and LiteDb samples. In either case, you would need to add Sqlite.Data.Sqlite and LiteDB respectively from Nuget and add using statements.
In case of SQLite, please note that you could use Linq adding some drivers. I directly used the ADO.Net commands and didn't use a Book class for mapping.
LiteDB, being a NoSQL database written in C# for C#, can directly use objects and support Linq out of the box.
Samples show only the surface for both.
SQLite sample:
private static readonly string dataFile = #"d:\temp\books.s3db";
void Main()
{
CreateDb(dataFile);
SeedSampleData(dataFile);
// List the current data
Console.WriteLine("Current Data");
Console.WriteLine("".PadRight(100, '='));
ListData(dataFile);
Console.WriteLine("".PadRight(100, '='));
DeleteSampleRow(dataFile);
// List the current data
Console.WriteLine("After deleting");
Console.WriteLine("".PadRight(100, '='));
ListData(dataFile);
Console.WriteLine("".PadRight(100, '='));
}
void DeleteSampleRow(string dbName)
{
string deleteById = "delete from books where id = #id";
string deleteByTitle = "delete from books where Title = #title";
string deleteByWriter = "delete from books where Writer = #writer";
using (SQLiteConnection cn = new SQLiteConnection($"Data Source={dbName}"))
using (SQLiteCommand cmdById = new SQLiteCommand(deleteById, cn))
using (SQLiteCommand cmdByTitle = new SQLiteCommand(deleteByTitle, cn))
using (SQLiteCommand cmdByWriter = new SQLiteCommand(deleteByWriter, cn))
{
cmdById.Parameters.Add("#id", DbType.Int32).Value = 2; // delete the book with id = 2
cmdByTitle.Parameters.Add("#title", DbType.String).Value = $"Sample Title #5"; // delete all books having title "Sample Title #5"
cmdByWriter.Parameters.Add("#writer", DbType.String).Value = $"Sample Writer #3"; // delete all books written by "Sample Writer #3"
cn.Open();
cmdById.ExecuteNonQuery();
cmdByTitle.ExecuteNonQuery();
cmdByWriter.ExecuteNonQuery();
cn.Close();
}
}
void ListData(string dbName)
{
string selectCommand = "select * from books";
using (SQLiteConnection cn = new SQLiteConnection($"Data Source={dbName}"))
using (SQLiteCommand cmd = new SQLiteCommand(selectCommand, cn))
{
cn.Open();
var r = cmd.ExecuteReader();
while (r.Read())
{
Console.WriteLine($"{r["id"]},{r["title"]},{r["writer"]},{r["publisher"]},{r["published_year"]}");
}
cn.Close();
}
}
private void CreateDb(string dbName)
{
if (File.Exists(dbName)) // if it exists, delete and create afresh, just for sampling
{
File.Delete(dbName);
}
string createTable = #"Create Table books (
id int primary key not null,
title varchar(500) not null,
writer varchar(100) not null,
publisher varchar(100) not null,
published_year int not null
)";
using (SQLiteConnection cn = new SQLiteConnection($"Data Source={dbName}"))
using (SQLiteCommand cmd = new SQLiteCommand(createTable, cn))
{
cn.Open();
cmd.ExecuteNonQuery();
cn.Close();
}
}
private void SeedSampleData(string dbName)
{
string insertCommand = #"insert into books
(id, title, writer, publisher, published_year)
values
(#id, #title, #writer, #publisher, #year);";
using (SQLiteConnection cn = new SQLiteConnection($"Data Source={dbName}"))
using (SQLiteCommand cmd = new SQLiteCommand(insertCommand, cn))
{
cmd.Parameters.Add("#id", DbType.Int32);
cmd.Parameters.Add("#title", DbType.String);
cmd.Parameters.Add("#writer", DbType.String);
cmd.Parameters.Add("#publisher", DbType.String);
cmd.Parameters.Add("#year", DbType.Int32);
Random r = new Random();
cn.Open();
int id = 1;
using (SQLiteTransaction transaction = cn.BeginTransaction())
{
cmd.Parameters["#id"].Value = id++;
cmd.Parameters["#title"].Value = $"Around the World in Eighty Days";
cmd.Parameters["#writer"].Value = $"Jules Verne";
cmd.Parameters["#publisher"].Value = $"Le Temps, Pierre-Jules Hetzel";
cmd.Parameters["#year"].Value = 1873;
cmd.ExecuteNonQuery();
cmd.Parameters["#id"].Value = id++;
cmd.Parameters["#title"].Value = $"A Tale of Two Cities";
cmd.Parameters["#writer"].Value = $"Charles Dickens";
cmd.Parameters["#publisher"].Value = $"Chapman & Hall";
cmd.Parameters["#year"].Value = 1859;
cmd.ExecuteNonQuery();
// add dummy 10 more rows
for (int i = 0; i < 10; i++)
{
cmd.Parameters["#id"].Value = id++;
cmd.Parameters["#title"].Value = $"Sample Title #{i}";
cmd.Parameters["#writer"].Value = $"Sample Writer #{r.Next(1, 5)}";
cmd.Parameters["#publisher"].Value = $"Sample Publisher #{i}";
cmd.Parameters["#year"].Value = r.Next(1980, 2022);
cmd.ExecuteNonQuery();
}
transaction.Commit();
}
// databases generally use some indexes
new SQLiteCommand(#"Create Index if not exists ixId on books (id);", cn).ExecuteNonQuery();
new SQLiteCommand(#"Create Index if not exists ixTitle on books (title);", cn).ExecuteNonQuery();
new SQLiteCommand(#"Create Index if not exists ixWriter on books (writer);", cn).ExecuteNonQuery();
new SQLiteCommand(#"Create Index if not exists ixPublisher on books (publisher);", cn).ExecuteNonQuery();
cn.Close();
}
}
LiteDb sample:
private static readonly string dataFile = #"d:\temp\books.litedb";
void Main()
{
//CreateDb(dataFile); // this step is not needed with LiteDB
// instead we just simply delete the datafile if it exists
// for starting afresh
// if it exists, delete and create afresh, just for sampling
// so you can run this same sample over and over if you wish
if (File.Exists(dataFile))
{
File.Delete(dataFile);
}
SeedSampleData(dataFile);
// List the current data
Console.WriteLine("Current Data");
Console.WriteLine("".PadRight(100, '='));
ListData(dataFile);
Console.WriteLine("".PadRight(100, '='));
DeleteSampleRow(dataFile);
// List the current data
Console.WriteLine("After deleting");
Console.WriteLine("".PadRight(100, '='));
ListData(dataFile);
Console.WriteLine("".PadRight(100, '='));
}
void DeleteSampleRow(string dbName)
{
using (var db = new LiteDatabase(dbName))
{
var bookCollection = db.GetCollection<Book>("Books");
// by ID
bookCollection.Delete(2);
// by Title
bookCollection.DeleteMany(c => c.Title == "Sample Title #5");
// by Writer
bookCollection.DeleteMany(c => c.Writer == "Sample Writer #3");
}
}
void ListData(string dbName)
{
using (var db = new LiteDatabase(dbName))
{
var bookCollection = db.GetCollection<Book>("Books");
foreach (var book in bookCollection.FindAll())
{
Console.WriteLine($"{book.Id},{book.Title},{book.Writer},{book.Publisher},{book.Published_year}");
}
}
}
private void SeedSampleData(string dbName)
{
Random r = new Random();
var books = new List<Book> {
new Book {Title="Around the World in Eighty Days",Writer = "Jules Verne",Publisher = "Le Temps, Pierre-Jules Hetzel",Published_year= 1873},
new Book {Title="A Tale of Two Cities",Writer = "Charles Dickens",Publisher = "Chapman & Hall",Published_year= 1859},
};
// add dummy 10 more rows
books.AddRange(Enumerable.Range(0, 10).Select(i => new Book
{
Title = $"Sample Title #{i}",
Writer = $"Sample Writer #{r.Next(1, 5)}",
Publisher = $"Sample Publisher #{i}",
Published_year = r.Next(1980, 2022)
}));
using (var db = new LiteDatabase(dbName))
{
var bookCollection = db.GetCollection<Book>("Books");
bookCollection.InsertBulk(books);
// databases generally use some indexes
// create the same indexes that we created in SQLite sample
bookCollection.EnsureIndex(c => c.Id);
bookCollection.EnsureIndex(c => c.Title);
bookCollection.EnsureIndex(c => c.Writer);
bookCollection.EnsureIndex(c => c.Publisher);
}
}
public class Book
{
public int Id {get;set;}
public string Title {get;set;}
public string Writer {get;set;}
public string Publisher {get;set;}
public int Published_year {get;set;}
}
welcome to SO. I'm going to assume you've got a reason for keeping the data in a text file. As several answers have suggested if you need it in a text file the easiest thing to do is to simply create a new file with the lines you want.
One way to do that is to make use of a interator function to filter the lines. This lets you easily use the .NET File class to do the rest - creating the new file and removing the old if you want to. Often keeping the old file and archiving it can be useful too but anyway, here's a way to filter the lines.
static void Main(string[] _)
{
var filteredLines = FilterOnID(File.ReadAllLines("datafile.txt"), "2");
File.WriteAllLines("updated.datafile.txt", filteredLines);
// rename if necessary
File.Delete("datafile.txt");
File.Move("updated.datafile.txt", "datafile.txt");
}
static IEnumerable<string> FilterOnID(IEnumerable<string> lines, string id)
{
foreach (var line in lines)
{
var fields = line.Split(';');
if (fields.Length != 0 || !string.IsNullOrEmpty(fields[0]))
{
if (id == fields[0])
continue;
}
yield return line;
}
}
To test I added simple file like so:
1;field1;field2;field3
2;field1;field2;field3
3;field1;field2;field3
4;field1;field2;field3
5;field1;field2;field3
6;field1;field2;field3
And after running you get this:
1;field1;field2;field3
3;field1;field2;field3
4;field1;field2;field3
5;field1;field2;field3
6;field1;field2;field3
When you put books into a list from a file, you can search the book for remove from BooksList.
Delete it and save BooksList into a file.
var removeBook = BookList.FirstOrDefault(book => book.id == removeId);
if (removeBook != null)
{
BookList.Remove(removeBook);
}
var booksAsString = BookList.Select(book => $"{book.id};{book.title};{book.writer};{book.publisher};{book.published_year}");
File.WriteAllLines("konyvek.txt", booksAsString, Encoding.UTF8);

Query taking longer than expected odbc sage

I am doing a simple select with a date filter on it with a months range where only 32 records are present however its taking 15 seconds to query and return the data I am using sage 50 as you can probally tell and c#. I am using odbc to create the query the seem speeds can be found if i use the odbc query tool.
This is for a stright forward select and it should not be taking that long to return the data through odbc.
String SQL = string.Format("SELECT 'ORDER_NUMBER', 'ORDER_OR_QUOTE',
'ANALYSIS_1','ACCOUNT_REF','ORDER_DATE','NAME',
'COURIER_NUMBER','COURIER_NAME','CUST_TEL_NUMBER'
,'DESPATCH_DATE','ACCOUNT_REF', 'DEL_NAME', 'DEL_ADDRESS_1',
'DEL_ADDRESS_2', 'DEL_ADDRESS_3', 'DEL_ADDRESS_4', 'DEL_ADDRESS_5',
'INVOICE_NUMBER','ORDER_DATE','INVOICE_NUMBER_NUMERIC',
'CONTACT_NAME','CONSIGNMENT', 'NOTES_1', 'ITEMS_NET'
,'ITEMS_GROSS','QUOTE_STATUS' FROM SALES_ORDER WHERE ORDER_DATE
='{0}' and ORDER_DATE <='{1}'", fromD, toD);
public List<SalesOrders> GetSalesOrders()
{
List<SalesOrders> _salesOrdersList = new List<SalesOrders>();
try
{
string sageDsn = ConfigurationManager.AppSettings["SageDSN"];
string sageUsername = ConfigurationManager.AppSettings["SageUsername"];
string sagePassword = ConfigurationManager.AppSettings["SagePassword"];
//int totalRecords = GetSalesOrdersount();
int counter = 0;
//using (var connection = new OdbcConnection("DSN=SageLine50v24;Uid=Manager;Pwd=;"))
using (var connection = new OdbcConnection(String.Format("DSN={0};Uid={1};Pwd={2};", sageDsn, sageUsername, sagePassword)))
{
connection.Open();
//string sql = string.Format(getInvoiceSql, customerCode, DateTime.Today.AddMonths(-1).ToString("yyyy-MM-dd"));
string fromD = dtpFrom.Value.ToString("yyyy-MM-dd");
string toD = dtpTo.Value.ToString("yyyy-MM-dd");
String SQL = string.Format("SELECT 'ORDER_NUMBER', 'ORDER_OR_QUOTE', 'ANALYSIS_1','ACCOUNT_REF','ORDER_DATE','NAME', 'COURIER_NUMBER','COURIER_NAME','CUST_TEL_NUMBER' ,'DESPATCH_DATE','ACCOUNT_REF', 'DEL_NAME', 'DEL_ADDRESS_1', 'DEL_ADDRESS_2', 'DEL_ADDRESS_3', 'DEL_ADDRESS_4', 'DEL_ADDRESS_5', 'INVOICE_NUMBER','ORDER_DATE','INVOICE_NUMBER_NUMERIC', 'CONTACT_NAME','CONSIGNMENT', 'NOTES_1', 'ITEMS_NET' ,'ITEMS_GROSS','QUOTE_STATUS' FROM SALES_ORDER WHERE ORDER_DATE >='{0}' and ORDER_DATE <='{1}'", fromD, toD);
using (var command = new OdbcCommand(SQL, connection))
{
using (var reader = command.ExecuteReader())
{
while (reader.Read())
{
counter++;
backgroundWorker1.ReportProgress(counter);
var salesOrders = new SalesOrders();
salesOrders.ACCOUNT_REF = Convert.ToString(reader["ACCOUNT_REF"]);
salesOrders.RecordIdentifier = "";
salesOrders.ShipmmentId = Convert.ToString(reader["ORDER_NUMBER"]);
salesOrders.OrderDate = Convert.ToDateTime(reader["ORDER_DATE"]);
salesOrders.OrderNumber = Convert.ToString(reader["ORDER_NUMBER"]);
salesOrders.Company = "";
salesOrders.Carrier = Convert.ToString(reader["COURIER_NUMBER"]);
salesOrders.CarrierService = Convert.ToString(reader["COURIER_NAME"]);
salesOrders.CustomerName = Convert.ToString(reader["NAME"]);
salesOrders.ShipToAddress1 = Convert.ToString(reader["DEL_ADDRESS_1"]);
salesOrders.ShipToAddress2 = Convert.ToString(reader["DEL_ADDRESS_2"]);
salesOrders.ShipToAddress3 = Convert.ToString(reader["DEL_ADDRESS_3"]);
salesOrders.ShipToAddress4 = Convert.ToString(reader["DEL_ADDRESS_4"]);
salesOrders.ShipToAddress5 = Convert.ToString(reader["DEL_ADDRESS_5"]);
salesOrders.ShiptoAttention = Convert.ToString(reader["DEL_NAME"]);
salesOrders.ShiptoPhoneNo = Convert.ToString(reader["CUST_TEL_NUMBER"]);
salesOrders.Country = Convert.ToString(reader["ANALYSIS_1"]);
salesOrders.ShiptoEmail = "";
salesOrders.MakeAddressDefault = "Y";
bool isProcessed = _sqlManager.hasbeenProcessed(salesOrders.OrderNumber);
if (isProcessed == true)
salesOrders.Exported = true;
_salesOrdersList.Add(salesOrders);
}
}
}
}
return _salesOrdersList.OrderByDescending(o => o.OrderDate).ToList();
}
don't use {0}, {1} for embedding values in strings... ADD via Parameters
String SQL =
#"SELECT
ORDER_NUMBER,
ORDER_OR_QUOTE,
ANALYSIS_1,
ACCOUNT_REF,
ORDER_DATE,
`NAME`,
COURIER_NUMBER,'
OURIER_NAME,
CUST_TEL_NUMBER,
DESPATCH_DATE,
ACCOUNT_REF,
DEL_NAME,
DEL_ADDRESS_1,
DEL_ADDRESS_2,
DEL_ADDRESS_3,
DEL_ADDRESS_4,
DEL_ADDRESS_5,
INVOICE_NUMBER,
ORDER_DATE,
INVOICE_NUMBER_NUMERIC,
CONTACT_NAME,
CONSIGNMENT,
NOTES_1,
ITEMS_NET,
ITEMS_GROSS,
QUOTE_STATUS
FROM
SALES_ORDER
WHERE
ORDER_DATE >= ?
and ORDER_DATE <= ?
ORDER BY
ORDER_DATE DESC";
using (var command = new OdbcCommand(SQL, connection))
{
// assuming fields are actually date data types fields
command.Parameters.Add( "parmFromDate", fromD );
command.Parameters.Add( "parmToDate", toD );
The "?" in the query are place-holders for the parameter values which are handled by the ODBC process. The Parameters being added within the using() portion are added in the same ordinal position as their respective place-holder parts. I just assigned the parameter name to give context to whoever is looking at it after.
The query itself SHOULD be very quick depending on the date range you are pulling. Even added the SQL Order by descending order so it is pre-pulled down in the order you intended it too.

Find matching contacts from contact list in an SQL table

I'm currently writing a server application for a chat app which receives the phone contact list from an users adressbook as a List<string> where the string is a standardized phone number. I now want to compare it to my SQL Database where all registered users are saved.
Right now I iterate trough every item on the list and query
SELECT * FROM Users WHERE Number = #0 and open on every query a new connection with SqlConnection(connectionString). I do so because I know that .NET will handle/pool the open connections.
var phoneNumbers = new List<string> { "0049123456789", "001123456789" };
var registeredUsers = new List<string>();
foreach (var phoneNumber in phoneNumbers)
{
try
{
var userName = GetFrom(phoneNumber);
registeredUsers.Add(userName);
}catch{}
}
//...
public static string GetFrom(string telephoneNumber)
{
string retVal = null;
using (var connection = new SqlConnection(connectionString))
{
using (var cmd = SetSQLCommand(connection, "SELECT * FROM Users WHERE Number = #0", telephoneNumber))
{
connection.Open();
using (var reader = cmd.ExecuteReader())
{
if (reader.HasRows)
{
while (reader.Read())//Get last
{
retVal = (string)reader["Name"];
}
}
else
{
throw (new KeyNotFoundException());
}
}
}
}
return retVal;
}
This method feels quite stupid as an user contact list could contain 1000 or more contacts.
Is there a more elegant way to find all matching telephone numbers from a List of telephone numbers I have as an C# object in an SQL table?
As performance is quite important to me I use the .NET 4.5 implementation of SQL System.Data.SqlClient, the Entity Framework is too slow for my purposes.
Here's my take when I face similar case like this :
I will concatenate all the phone numbers into one single formatted string
I will pass that string in a dynamic SQL query by utilizing the "IN" T-SQL
It could be as simple as this :
public List<string> ReturnContactNames(string telephoneNumbers)
{
var retVal = new List<string>();
using (var connection = new SqlConnection(connectionString))
{
using (var cmd = new SqlCommand("SELECT * FROM Customers WHERE Phone IN (" + telephoneNumbers + ")", connection))
{
connection.Open();
using (var reader = cmd.ExecuteReader())
{
while (reader.Read())
{
retVal.Add((string)reader["ContactName"]);
}
}
}
}
return retVal;
}
And then this is how you may use this method :
var phoneNumbers = "'030-0074321','0621-08460'";
var list = ReturnContactNames(phoneNumbers);
Warning : use parameterized query whenever possible. This is just a quick sample only to give you an idea.
Hope it helps.

Export data to text file with fixed length records

I am using MYSQL, WinForms and c#
I have a requirement when I next to get data from my database and export to a text file but each entry needs to be a specific length regardless of the information the database brings in -where data is less than total amount required, empty spaces are to follow it.
E.g First Name could bring in Sam or Samatha but I need to export to be 8 characters in length
"Sam " (8 total characters)
"Samatha "(8 total characters)
this is the function that I am using - any help would be greatly appreciated
public MySqlDataReader RunQueryTextFile(string query, string s1,string pathName)
{
MySqlConnection conDataBase = new MySqlConnection(connString);
MySqlCommand cmdDatabase = new MySqlCommand(query, conDataBase);
MySqlDataReader myReader;
StreamWriter sw = new StreamWriter(#pathName);
conDataBase.Open();
myReader = cmdDatabase.ExecuteReader();
while (myReader.Read())
{
sw.WriteLine (myReader[s1].ToString());
}
sw.Close();
conDataBase.Close();
return myReader;
}
use PadRight
int totalLength = 8;
myString.PadRight(totalLength, " ").
You trying to write procedural code. Think objects. I would do something like this
// create enum - how to pad
public enum PadStyle
{
Right
Left
}
// create attribute class
public class PaddingAttribute : Attribute
{
public int TotalLen {get;set;}
public PadStyle Style {get;set;}
public char PadChar {get;set;}
public int Order {get;set;}
}
// Create class record base - this one knows how to get properties, read attributes and pad values into one string
public class RecordBase
{
protected string CreateRecordProtected()
{
// here you
// 1 - use reflection to get properties
// 2 - use reflection to read PaddingAttribute from properties
// 3 - pad property values using data in PaddingAttribute
// 4 - concatenate record
// something like
var result = o.GetPropeties().Where(...).Select(...padding logic...);
return string.Join("", result);
// padding logic = string.PadRight or string.PadLeft
}
}
Read here how to deal with attribute classes
// Create class record - this one corresponds to your DB
public class Record : RecordBase
{
private const string _fn_FiedName = "firstName";
private const string _ln_FiedName = "lastName";
public Record(IDataReader r)
{
FirstName = r[_fn_FiedName];
LastName = r[_ln_FiedName];
}
[Padding(TotalLen = 15, Style = PadStyle.Right, PadChar = " ", Order = 1)]
public string FirstName{get;set;}
[Padding(TotalLen = 20, Style = PadStyle.Right, PadChar = " ", Order = 2)]
public string LastName {get;set;}
public override string ToString()
{
return base.CreateRecordProtected();
}
And now, your code with some mods
var recList = new List<Record>();
using (var conn = new MySqlConnection(connString))
{
using (var cmd = new MySqlCommand(query, conn))
{
conn.Open();
using (var reader = cmdDatabase.ExecuteReader())
{
while (reader.Read())
{
recList.Add(new Record(reader));
}
}
}
conn.Close();
}
if (!recList.Any()) // System.Linq
return;
IEnumerable<string> textFileRecords = recList.Select(x => x.ToString());
File.WriteAllLines(pathName, textFileRecords); // System.IO
Now, you have some decent reusable code.
NOTE: if data amount is huge, you can append right in while
var rec = new Record(reader);
rec.ToString() // append this

how to separate the "collection" from creating and adding a new instance of "lookaheadRunInfo"

I am trying to "collect" the GetString(2) until GetString(0) changes,so am trying to find out how to separate the "collection" from creating and adding a new instance of "lookaheadRunInfo"?I have tried as below which throws an exception
System.NullReferenceException was unhandled by user code at line
lookaheadRunInfo.gerrits.Add(rdr.GetString(1)); ,can anyone provide guidance on how to fix this issue?
try
{
Console.WriteLine("Connecting to MySQL...");
conn.Open();
string sql = #"select lr.ec_job_link, cl.change_list ,lr.submitted_by, lr.submission_time,lr.lookahead_run_status
from lookahead_run as lr, lookahead_run_change_list as lrcl, change_list_details as cld,change_lists as cl
where lr.lookahead_run_status is null
and lr.submission_time is not null
and lrcl.lookahead_run_id = lr.lookahead_run_id
and cl.change_list_id = lrcl.change_list_id
and cl.change_list_id not in (select clcl.change_list_id from component_labels_change_lists as clcl)
and cld.change_list_id = lrcl.change_list_id
group by lr.lookahead_run_id, cl.change_list
order by lr.submission_time desc
limit 1000
";
MySqlCommand cmd = new MySqlCommand(sql, conn);
MySqlDataReader rdr = cmd.ExecuteReader();
var ECJoblink_previous ="";
var gerritList = new List<String>();
while (rdr.Read())
{
//Console.WriteLine(rdr[0] + " -- " + rdr[1]);
//Console.ReadLine();
var lookaheadRunInfo = new LookaheadRunInfo();
lookaheadRunInfo.ECJobLink = rdr.GetString(0);
if (ECJoblink_previous == lookaheadRunInfo.ECJobLink)
{
//Keep appending the list of gerrits until we get a new lookaheadRunInfo.ECJobLink
lookaheadRunInfo.gerrits.Add(rdr.GetString(1));
}
else
{
lookaheadRunInfo.gerrits = new List<string> { rdr.GetString(1) };
}
ECJoblink_previous = lookaheadRunInfo.ECJobLink;
lookaheadRunInfo.UserSubmitted = rdr.GetString(2);
lookaheadRunInfo.SubmittedTime = rdr.GetString(3).ToString();
lookaheadRunInfo.RunStatus = "null";
lookaheadRunInfo.ElapsedTime = (DateTime.UtcNow-rdr.GetDateTime(3)).ToString();
lookaheadRunsInfo.Add(lookaheadRunInfo);
}
rdr.Close();
}
catch
{
throw;
}
If I understand your requirements correctly, you wish to keep a single lookaheadRunInfo for several rows of the resultset, until GetString(0) changes. Is that right?
In that case you have some significant logic problems. The way it is written, even if we fix the null reference, you will get a new lookaheadRunInfo with each and every row.
Try this:
string ECJoblink_previous = null;
LookAheadRunInfo lookaheadRunInfo = null;
while (rdr.Read())
{
if (ECJoblink_previous != rdr.GetString(0)) //A new set of rows is starting
{
if (lookaheadRunInfo != null)
{
lookaheadRunsInfo.Add(lookaheadRunInfo); //Save the old group, if it exists
}
lookaheadRunInfo = new LookAheadRunInfo //Start a new group and initialize it
{
ECJobLink = rdr.GetString(0),
gerrits = new List<string>(),
UserSubmitted = rdr.GetString(2),
SubmittedTime = rdr.GetString(3).ToString(),
RunStatus = "null",
ElapsedTime = (DateTime.UtcNow-rdr.GetDateTime(3)).ToString()
}
}
lookahead.gerrits.Add(rdr.GetString(1)); //Add current row
ECJoblink_previous = rdr.GetString(0); //Keep track of column 0 for next iteration
}
if (lookaheadRunInfo != null)
{
lookaheadRunsInfo.Add(lookaheadRunInfo); //Save the last group, if there is one
}
The idea here is:
Start with a blank slate, nothing initialized
Monitor column 0. When it changes (as it will on the first row), save any old list and start a new one
Add to current list with each and every iteration
When done, save any remaining items in its own list. A null check is required in case the reader returned 0 rows.

Categories

Resources