WMI is being too slow - c#

Is there a way to limit the number of entries WMI retrieves with a WQL statement?
I say this because running a query to retrieve all Win32_NTLogEvent instances is taking forever! All I really need are the most recent events (for about a week, or 2000 entries)
Here's a snippet of the code I'm using to get the log data. Other queries such as Win32_Processor are nice and quick.
if (Configuration.OnlyErrorLogs)
{
// If Information logs should be suppressed, only get events where event type is not 3
WMIDataTemp1 = DataRetriever.GetWMIData("Win32_NTLogEvent", "EventType<>3");
}
else
{
WMIDataTemp1 = DataRetriever.GetWMIData("Win32_NTLogEvent");
}
foreach (ManagementObject Object in WMIDataTemp1)
{
this.Log.Add(new Log(Object));
}
And the functions to get WMI data are as follows:
public static ManagementObject[] GetWMIData(string wmiClass) { return GetWMIData(wmiClass, "", "CIMV2"); }
public static ManagementObject[] GetWMIData(string wmiClass, string whereClause) { return GetWMIData(wmiClass, whereClause, "CIMV2"); }
public static ManagementObject[] GetWMIData(string wmiClass, string whereClause, string nameSpace)
{
try
{
// If a where clause has been set, prepare the clause to add to the query string
if (whereClause != "")
{
whereClause = " WHERE " + whereClause;
}
// Create a search query
string query = "SELECT * FROM " + wmiClass + whereClause;
ManagementObjectSearcher wmiSearcher = new ManagementObjectSearcher("root\\" + nameSpace, query);
ManagementObjectCollection matches = wmiSearcher.Get();
// Create an array to hold the matches
ManagementObject[] matchArray = new ManagementObject[matches.Count];
// If matches found, copy to output
if(matches.Count > 0)
{
// Copy the search matches into this array
matches.CopyTo(matchArray, 0);
}
// Return array
return matchArray;
}
catch (Exception e)
{
ErrorDialogue errorReporter = new ErrorDialogue(e);
return null;
}
}
Where each Log gets stored:
public class Log
{
public string Category = "N/A";
public string DateTime = "N/A";
public UInt16 ID = 0;
public string Level = "N/A";
public string Message = "N/A";
public string Source = "N/A";
public Log() { }
public Log(ManagementObject wmiLogEvent)
{
this.GetInfo(wmiLogEvent);
}
public void GetInfo(ManagementObject wmiLogEvent)
{
try
{
this.Category = DataRetriever.GetValue(wmiLogEvent, "CategoryString");
this.DateTime = DataRetriever.GetValue(wmiLogEvent, "TimeGenerated");
this.ID = DataRetriever.GetValueUInt16(wmiLogEvent, "EventIdentifier");
this.Level = DataRetriever.ConvertEventType(DataRetriever.GetValueUInt16(wmiLogEvent, "CategoryString"));
this.Message = DataRetriever.GetValue(wmiLogEvent, "Message");
this.Source = DataRetriever.GetValue(wmiLogEvent, "SourceName");
}
catch (Exception e)
{
ErrorDialogue errorReporter = new ErrorDialogue(e);
}
}
}

One option is to use a WHERE clause to specify the range of the entries you want...
For example you could use TimeGenerated in the WHERE clause to specify a time-based range...
Another option is to set BlockSize accordingly when creating ManagementObjectSearcher.
You could use that to specify that you want 2000 entries per call for example - together with an ORDER BY TimeGenerated DESC this should give a nice result.

Speed is not a strong suit for WMI. It tends to be quite memory intensive. However, the question has been addressed and there are a few things you can do. Check out Why are my queries taking such a long time to complete? from Microsoft TechNet.

Now using the System.Diagnostics.EventLog class as a faster alternative. Much more beneficial to the program compared to WMI.
http://msdn.microsoft.com/en-us/library/system.diagnostics.eventlog.aspx

Related

Reading the file only once for every method call

I am new to object-oriented programming and I am working on a small personal project with some SQL scripts.
I have a scenario where a SQL script calls a static method with a file path as input.
queries = Select Query from Table where Utils.ContainsKeyword(Query, #Path1) AND NOT Utils.ContainsKeyword(Query, #Path2);
I had initially created a static class that does the following:
public static class Utils
{
public static bool ContainsKeyword(string query, string path)
{
var isQueryInFile = false;
var stringFromFile = GetStringFromFile(path);
List<Regex>regexList = GetRegexList(stringFromFile);
if(regexList!= null)
{
isQueryInFile = regexList.Any(pattern => pattern.IsMatch(query));
}
return isQueryInFile;
}
private static string GetStringFromFile(string path)
{
var words = String.Empty;
if(!string.IsNullOrEmpty(path))
{
try
{
using (StreamReader sr = File.OpenText(path))
{
words = sr.ReadToEnd().Replace(Environment.Newline, "");
}
}
catch { return words; }
}
return words;
}
private static List<Regex> GetRegexList(string words)
{
if(string.IsNullOrEmpty(words)) { return null; }
return words.Split(',').Select(w=> new Regex(#"\b" + Regex.Escape(w) + #'\b', RegexOptions.Compiled | RegexOptions.IgnoreCase)).ToList();
}
}
My problem is that I neither want to read from the file every time the ContainsKeyword static method is called nor do I want to create a new RegexList every time. Also, I cannot change the SQL script and I have to send the path to the file as an input parameter for the method call in the SQL script since the path might change in the future.
Is there a way to make sure I only read the contents from the input path only once, store them in a string, and use the string for the match with different input queries?
To read the content only once, saving in memory will probaby be needed. Memory capacity could be an issue.
public Dictionary<string, string> FileContentCache { get; set; } // make sure that gets initialized
public string GetFileContentCache(string path)
{
if (FileContentCache == null) FileContentCache = new Dictionary<string, string>();
if (FileContentCache.ContainsKey(path))
return FileContentCache[path];
var fileData = GetStringFromFile(path);
FileContentCache.Add(path, fileData);
return fileData;
}

WMI Query Library C#

I am having hard time understanding why there's not a single WMI query library
in here or google pointed out. Simply interface to query WMI for information. All I can find is "write a new query like this..."-style of topics all over web which makes no sense to me while there's few hundred queries I am working with.
So, here's my current effort building one:
using System;
using System.Management;
namespace hardware
{
public class Info
{
public static Tuple<string[], int, int> Query(string Select, string From, string Where = null, string Answer = null, string Root = "root\\CIMV2")
{
int Count = 0;
string[] Values = Select.Split(',');
if (Where != null && Answer != null)
{
ManagementObjectSearcher s = new ManagementObjectSearcher(Root, "SELECT " + Select + " FROM " + From + " WHERE " + Where + " = '" + Answer + "'");
string[] res = new string[Values.Length];
foreach (ManagementObject wQuery in s.Get())
{
foreach (string value in Values)
{
res[Count] = Convert.ToString(wQuery[value]);
Count++;
}
}
var result = new Tuple<string[], int, int> (res, Values.Length, Count);
return result;
}
else
{
ManagementObjectSearcher s = new ManagementObjectSearcher(Root, "SELECT " + Select + " FROM " + From);
string[] res = new string[Values.Length];
foreach (ManagementObject wQuery in s.Get())
{
foreach (string value in Values)
{
res[Count] = Convert.ToString(wQuery[value]);
Count++;
}
}
var result = new Tuple<string[], int, int>(res, Values.Length, Count);
return result;
}
}
}
}
I've been trying to get above in working order now for a weeks. Coming from basic string multi-dimensional array (return string[,] or string[][]) just started testing, if returning tuple (which I used few years ago successfully) would be better choice.
So, anyone know, if there is library already written?
if not, then best way to do above query from reference DLL library?
I know this question is old. But you can use ORMi library. I think it will just fit on your needs.
For example:
1) Define your class:
[WMIClass("Win32_Processor")]
public class Processor
{
public string Name { get; set; }
[WMIProperty("NumberOfCores")]
public int Cores { get; set; }
public string Description { get; set; }
}
2) Query:
WMIHelper helper = new WMIHelper("root\\CimV2");
List<Processor> processors = helper.Query<Processor>().ToList();
3) Or lazy coding option:
var processors = helper.Query("SELECT * FROM Win32_Processor");
You can find more information in here:
https://github.com/nicoriff/ORMi
https://medium.com/#luque.nicolas/compare-ormi-and-traditional-net-wmi-implementation-f00db26d10a3

Unable to stop SQL injection errors

I finally get it. It's not just the code I use to execute the ExecuteScalar method but it is mainly the code up stream that is executing the class. It is everything calling your code. That said, can someone please see if the code executing the my SQL class has faults. I still cant pass the scans. First I will show you two examples of the code calling my code, then the calling code, and finally the executing code, which I formulated and displayed from a previous post.
Calling code with three parameters:
public bool isTamAsp(int aspKey, int fy, string accountCode)
{
MyParam myParam;
string sqlQuery = "select isTamMacom = count(macom_key) FROM hier_fy " +
"WHERE hier_key = #aspKey AND fy = #fy AND #accountCode NOT IN (3,4,7,8) AND macom_key IN (select hier_key from lkup_e581_MacomThatRequireTAM) AND is_visible = 1 AND is_active = 1";
QueryContainer Instance = new QueryContainer(sqlQuery);
myParam = new MyParam();
myParam.SqlParam = new SqlParameter("#aspKey", Instance.AddParameterType(_DbTypes.Int));
myParam.SqlParam.Value = aspKey;
Instance.parameterList.Add(myParam);
myParam = new MyParam();
myParam.SqlParam = new SqlParameter("#fy", Instance.AddParameterType(_DbTypes.Int));
myParam.SqlParam.Value = fy;
Instance.parameterList.Add(myParam);
myParam = new MyParam();
myParam.SqlParam = new SqlParameter("#accountCode", Instance.AddParameterType(_DbTypes._string));
myParam.SqlParam.Value = accountCode;
Instance.parameterList.Add(myParam);
if (Convert.ToInt32(ExecuteScaler(Instance)) < 1)
return false;
return true;
}
Calling code with no parameters:
public long GetMarinesUploadNextUploadKey()
{
string query = "SELECT MAX(upload_key) FROM temp_auth_usmc_upload";
QueryContainer Instance = new QueryContainer(query);
string result = Convert.ToString(ExecuteScaler(Instance));
if (string.IsNullOrEmpty(result))
return 1;
else
return Convert.ToInt64(result) + 1;
}
Code calling my previous code with three parameters:
public bool isTamAsp(int aspKey, int fy, string accountCode)
{
return e581provider.isTamAsp(aspKey, fy, accountCode);
}
Method calling the SQL executing my code:
DbCommand command = _provider.CreateCommand();
command.Connection = _connection;
{
command.CommandText = Instance.Query;
command.CommandType = CommandType.Text;
if (Instance.parameterList.Count > 0)
{
foreach (var p in Instance.parameterList)
{
command.Parameters.Add(p.SqlParam);
}
}
if (_useTransaction) { command.Transaction = _transaction; }
try
{
returnValue = command.ExecuteScalar();
}
My Class containing the SQL string and the cmd parameter List
public enum _DbTypes
{
Int = 1, _string = 2, _long = 3, _bool = 4, _DateTime = 5,
_decimal = 6, _float = 7, _short = 8, _bite = 9
}
public class MyParam
{
public SqlParameter SqlParam { get; set; }
}
/// <summary>
/// Summary description for QueryContainer SGH
/// </summary>
public class QueryContainer
{
string _query;
public List<MyParam> parameterList = new List<MyParam>();
public QueryContainer(string query) { _query = query; }
public SqlDbType AddParameterType(_DbTypes id)
{
switch (id)
{
case _DbTypes.Int:
return (SqlDbType)Enum.Parse(typeof(SqlDbType), "int", true);
case _DbTypes._string:
return (SqlDbType)Enum.Parse(typeof(SqlDbType), "NVarChar", true);
case _DbTypes._long:
return (SqlDbType)Enum.Parse(typeof(SqlDbType), "SqlDbType.BigInt", true);
case _DbTypes._bool:
return (SqlDbType)Enum.Parse(typeof(SqlDbType), "SqlDbType.Bit", true);
}
return SqlDbType.VarChar;
}
public string Query
{
get
{
return _query;
}
set { _query = value; }
}
}
I don't see a vulnerability in that code, but I have an idea what the scan may be asking for. The problem could be that this code makes it too easy for developers to ignore the parameterList collection in your class. If I'm a new developer in your organization who hasn't discovered Sql injection yet, I'd be tempted to ignore all that complicated query parameter stuff and just use string concatenation before setting your Query property.
Instead of wrapping this in a class, what I'm more used to seeing is a single method that has a signature like this:
IEnumerable<T> GetData<T>(string query, IEnumerable<Sqlparameter> parameters)
...or some permutation of that method signature that may use arrays or lists instead of IEnumerable. This forces downstream developers to deal with that parameters argument to the method. They can't ignore it, and so the temptation to use a quick, lazy string concatenation call to substitute some user-provided data into the query is reduced.

How to check the length of a MSMQ [duplicate]

I was wondering if there is a way to programmatically check how many messages are in a private or public MSMQ using C#? I have code that checks if a queue is empty or not using the peek method wrapped in a try/catch, but I've never seen anything about showing the number of messages in the queue. This would be very helpful for monitoring if a queue is getting backed up.
You can read the Performance Counter value for the queue directly from .NET:
using System.Diagnostics;
// ...
var queueCounter = new PerformanceCounter(
"MSMQ Queue",
"Messages in Queue",
#"machinename\private$\testqueue2");
Console.WriteLine( "Queue contains {0} messages",
queueCounter.NextValue().ToString());
There is no API available, but you can use GetMessageEnumerator2 which is fast enough. Sample:
MessageQueue q = new MessageQueue(...);
int count = q.Count();
Implementation
public static class MsmqEx
{
public static int Count(this MessageQueue queue)
{
int count = 0;
var enumerator = queue.GetMessageEnumerator2();
while (enumerator.MoveNext())
count++;
return count;
}
}
I also tried other options, but each has some downsides
Performance counter may throw exception "Instance '...' does not exist in the specified Category."
Reading all messages and then taking count is really slow, it also removes the messages from queue
There seems to be a problem with Peek method which throws an exception
If you need a fast method (25k calls/second on my box), I recommend Ayende's version based on MQMgmtGetInfo() and PROPID_MGMT_QUEUE_MESSAGE_COUNT:
for C#
https://github.com/hibernating-rhinos/rhino-esb/blob/master/Rhino.ServiceBus/Msmq/MsmqExtensions.cs
for VB
https://gist.github.com/Lercher/5e1af6a2ba193b38be29
The origin was probably http://functionalflow.co.uk/blog/2008/08/27/counting-the-number-of-messages-in-a-message-queue-in/ but I'm not convinced that this implementation from 2008 works any more.
We use the MSMQ Interop. Depending on your needs you can probably simplify this:
public int? CountQueue(MessageQueue queue, bool isPrivate)
{
int? Result = null;
try
{
//MSMQ.MSMQManagement mgmt = new MSMQ.MSMQManagement();
var mgmt = new MSMQ.MSMQManagementClass();
try
{
String host = queue.MachineName;
Object hostObject = (Object)host;
String pathName = (isPrivate) ? queue.FormatName : null;
Object pathNameObject = (Object)pathName;
String formatName = (isPrivate) ? null : queue.Path;
Object formatNameObject = (Object)formatName;
mgmt.Init(ref hostObject, ref formatNameObject, ref pathNameObject);
Result = mgmt.MessageCount;
}
finally
{
mgmt = null;
}
}
catch (Exception exc)
{
if (!exc.Message.Equals("Exception from HRESULT: 0xC00E0004", StringComparison.InvariantCultureIgnoreCase))
{
if (log.IsErrorEnabled) { log.Error("Error in CountQueue(). Queue was [" + queue.MachineName + "\\" + queue.QueueName + "]", exc); }
}
Result = null;
}
return Result;
}
//here queue is msmq queue which you have to find count.
int index = 0;
MSMQManagement msmq = new MSMQManagement() ;
object machine = queue.MachineName;
object path = null;
object formate=queue.FormatName;
msmq.Init(ref machine, ref path,ref formate);
long count = msmq.MessageCount();
This is faster than you selected one.
You get MSMQManagement class refferance inside "C:\Program Files (x86)\Microsoft SDKs\Windows" just brows in this address you will get it. for more details you can visit http://msdn.microsoft.com/en-us/library/ms711378%28VS.85%29.aspx.
I had real trouble getting the accepted answer working because of the xxx does not exist in the specified Category error. None of the solutions above worked for me.
However, simply specifying the machine name as below seems to fix it.
private long GetQueueCount()
{
try
{
var queueCounter = new PerformanceCounter("MSMQ Queue", "Messages in Queue", #"machineName\private$\stream")
{
MachineName = "machineName"
};
return (long)queueCounter.NextValue();
}
catch (Exception e)
{
return 0;
}
}
The fastest method I have found to retrieve a message queue count is to use the peek method from the following site:
protected Message PeekWithoutTimeout(MessageQueue q, Cursor cursor, PeekAction action)
{
Message ret = null;
try
{
ret = q.Peek(new TimeSpan(1), cursor, action);
}
catch (MessageQueueException mqe)
{
if (!mqe.Message.ToLower().Contains("timeout"))
{
throw;
}
}
return ret;
}
protected int GetMessageCount(MessageQueue q)
{
int count = 0;
Cursor cursor = q.CreateCursor();
Message m = PeekWithoutTimeout(q, cursor, PeekAction.Current);
{
count = 1;
while ((m = PeekWithoutTimeout(q, cursor, PeekAction.Next)) != null)
{
count++;
}
}
return count;
}
This worked for me. Using a Enumarator to make sure the queue is empty first.
Dim qMsg As Message ' instance of the message to be picked
Dim privateQ As New MessageQueue(svrName & "\Private$\" & svrQName) 'variable svrnme = server name ; svrQName = Server Queue Name
privateQ.Formatter = New XmlMessageFormatter(New Type() {GetType(String)}) 'Formating the message to be readable the body tyep
Dim t As MessageEnumerator 'declared a enumarater to enable to count the queue
t = privateQ.GetMessageEnumerator2() 'counts the queues
If t.MoveNext() = True Then 'check whether the queue is empty before reading message. otherwise it will wait forever
qMsg = privateQ.Receive
Return qMsg.Body.ToString
End If
If you want a Count of a private queue, you can do this using WMI.
This is the code for this:
// You can change this query to a more specific queue name or to get all queues
private const string WmiQuery = #"SELECT Name,MessagesinQueue FROM Win32_PerfRawdata_MSMQ_MSMQQueue WHERE Name LIKE 'private%myqueue'";
public int GetCount()
{
using (ManagementObjectSearcher wmiSearch = new ManagementObjectSearcher(WmiQuery))
{
ManagementObjectCollection wmiCollection = wmiSearch.Get();
foreach (ManagementBaseObject wmiObject in wmiCollection)
{
foreach (PropertyData wmiProperty in wmiObject.Properties)
{
if (wmiProperty.Name.Equals("MessagesinQueue", StringComparison.InvariantCultureIgnoreCase))
{
return int.Parse(wmiProperty.Value.ToString());
}
}
}
}
}
Thanks to the Microsoft.Windows.Compatibility package this also works in netcore/netstandard.
The message count in the queue can be found using the following code.
MessageQueue messageQueue = new MessageQueue(".\\private$\\TestQueue");
var noOFMessages = messageQueue.GetAllMessages().LongCount();

How to take a CSV field and write to columns in SQL

I have the following code which takes a CSV and writes to a console:
using (CsvReader csv = new CsvReader(
new StreamReader("data.csv"), true))
{
// missing fields will not throw an exception,
// but will instead be treated as if there was a null value
csv.MissingFieldAction = MissingFieldAction.ReplaceByNull;
// to replace by "" instead, then use the following action:
//csv.MissingFieldAction = MissingFieldAction.ReplaceByEmpty;
int fieldCount = csv.FieldCount;
string[] headers = csv.GetFieldHeaders();
while (csv.ReadNextRecord())
{
for (int i = 0; i < fieldCount; i++)
Console.Write(string.Format("{0} = {1};",
headers[i],
csv[i] == null ? "MISSING" : csv[i]));
Console.WriteLine();
}
}
The CSV file has 7 headers for which I have 7 columns in my SQL table.
What is the best way to take each csv[i] and write to a row for each column and then move to the next row?
I tried to add the ccsv[i] to a string array but that didn't work.
I also tried the following:
SqlCommand sql = new SqlCommand("INSERT INTO table1 [" + csv[i] + "]", mysqlconnectionstring);
sql.ExecuteNonQuery();
My table (table1) is like this:
name address city zipcode phone fax device
your problem is simple but I will take it one step further and let you know a better way to approach the issue.
when you have a problem to sold, always break it down into parts and apply each part in each own method. For example, in your case:
1 - read from the file
2 - create a sql query
3 - run the query
and you can even add validation to the file (imagine your file does not even have 7 fields in one or more lines...) and the example below it to be taken, only if your file never passes around 500 lines, as if it does normally you should consider to use a SQL statement that takes your file directly in to the database, it's called bulk insert
1 - read from file:
I would use a List<string> to hold the line entries and I always use StreamReader to read from text files.
using (StreamReader sr = File.OpenText(this.CsvPath))
{
while ((line = sr.ReadLine()) != null)
{
splittedLine = line.Split(new string[] { this.Separator }, StringSplitOptions.None);
if (iLine == 0 && this.HasHeader)
// header line
this.Header = splittedLine;
else
this.Lines.Add(splittedLine);
iLine++;
}
}
2 - generate the sql
foreach (var line in this.Lines)
{
string entries = string.Concat("'", string.Join("','", line))
.TrimEnd('\'').TrimEnd(','); // remove last ",'"
this.Query.Add(string.Format(this.LineTemplate, entries));
}
3 - run the query
SqlCommand sql = new SqlCommand(string.Join("", query), mysqlconnectionstring);
sql.ExecuteNonQuery();
having some fun I end up doing the solution and you can download it here, the output is:
The code can be found here. It needs more tweaks but I will left that for others. Solution written in C#, VS 2013.
The ExtractCsvIntoSql class is as follows:
public class ExtractCsvIntoSql
{
private string CsvPath, Separator;
private bool HasHeader;
private List<string[]> Lines;
private List<string> Query;
/// <summary>
/// Header content of the CSV File
/// </summary>
public string[] Header { get; private set; }
/// <summary>
/// Template to be used in each INSERT Query statement
/// </summary>
public string LineTemplate { get; set; }
public ExtractCsvIntoSql(string csvPath, string separator, bool hasHeader = false)
{
this.CsvPath = csvPath;
this.Separator = separator;
this.HasHeader = hasHeader;
this.Lines = new List<string[]>();
// you can also set this
this.LineTemplate = "INSERT INTO [table1] SELECT ({0});";
}
/// <summary>
/// Generates the SQL Query
/// </summary>
/// <returns></returns>
public List<string> Generate()
{
if(this.CsvPath == null)
throw new ArgumentException("CSV Path can't be empty");
// extract csv into object
Extract();
// generate sql query
GenerateQuery();
return this.Query;
}
private void Extract()
{
string line;
string[] splittedLine;
int iLine = 0;
try
{
using (StreamReader sr = File.OpenText(this.CsvPath))
{
while ((line = sr.ReadLine()) != null)
{
splittedLine = line.Split(new string[] { this.Separator }, StringSplitOptions.None);
if (iLine == 0 && this.HasHeader)
// header line
this.Header = splittedLine;
else
this.Lines.Add(splittedLine);
iLine++;
}
}
}
catch (Exception ex)
{
if(ex.InnerException != null)
while (ex.InnerException != null)
ex = ex.InnerException;
throw ex;
}
// Lines will have all rows and each row, the column entry
}
private void GenerateQuery()
{
foreach (var line in this.Lines)
{
string entries = string.Concat("'", string.Join("','", line))
.TrimEnd('\'').TrimEnd(','); // remove last ",'"
this.Query.Add(string.Format(this.LineTemplate, entries));
}
}
}
and you can run it as:
class Program
{
static void Main(string[] args)
{
string file = Ask("What is the CSV file path? (full path)");
string separator = Ask("What is the current separator? (; or ,)");
var extract = new ExtractCsvIntoSql(file, separator);
var sql = extract.Generate();
Output(sql);
}
private static void Output(IEnumerable<string> sql)
{
foreach(var query in sql)
Console.WriteLine(query);
Console.WriteLine("*******************************************");
Console.Write("END ");
Console.ReadLine();
}
private static string Ask(string question)
{
Console.WriteLine("*******************************************");
Console.WriteLine(question);
Console.Write("= ");
return Console.ReadLine();
}
}
Usually i like to be a bit more generic so i'll try to explain a very basic flow i use from time to time:
I don't like the hard coded attitude so even if your code will work it will be dedicated specifically to one type. I prefer i simple reflection, first to understand what DTO is it and then to understand what repository should i use to manipulate it:
For example:
public class ImportProvider
{
private readonly string _path;
private readonly ObjectResolver _objectResolver;
public ImportProvider(string path)
{
_path = path;
_objectResolver = new ObjectResolver();
}
public void Import()
{
var filePaths = Directory.GetFiles(_path, "*.csv");
foreach (var filePath in filePaths)
{
var fileName = Path.GetFileName(filePath);
var className = fileName.Remove(fileName.Length-4);
using (var reader = new CsvFileReader(filePath))
{
var row = new CsvRow();
var repository = (DaoBase)_objectResolver.Resolve("DAL.Repository", className + "Dao");
while (reader.ReadRow(row))
{
var dtoInstance = (DtoBase)_objectResolver.Resolve("DAL.DTO", className + "Dto");
dtoInstance.FillInstance(row.ToArray());
repository.Save(dtoInstance);
}
}
}
}
}
Above is a very basic class responsible importing the data. Nevertheless of how this piece of code parsing CSV files (CsvFileReader), the important part is thata "CsvRow" is a simple List.
Below is the implementation of the ObjectResolver:
public class ObjectResolver
{
private readonly Assembly _myDal;
public ObjectResolver()
{
_myDal = Assembly.Load("DAL");
}
public object Resolve(string nameSpace, string name)
{
var myLoadClass = _myDal.GetType(nameSpace + "." + name);
return Activator.CreateInstance(myLoadClass);
}
}
The idea is to simple follow a naming convetion, in my case is using a "Dto" suffix for reflecting the instances, and "Dao" suffix for reflecting the responsible dao. The full name of the Dto or the Dao can be taken from the csv name or from the header (as you wish)
Next step is filling the Dto, each dto or implements the following simple abstract:
public abstract class DtoBase
{
public abstract void FillInstance(params string[] parameters);
}
Since each Dto "knows" his structure (just like you knew to create an appropriate table in the database), it can easily implement the FillInstanceMethod, here is a simple Dto example:
public class ProductDto : DtoBase
{
public int ProductId { get; set; }
public double Weight { get; set; }
public int FamilyId { get; set; }
public override void FillInstance(params string[] parameters)
{
ProductId = int.Parse(parameters[0]);
Weight = double.Parse(parameters[1]);
FamilyId = int.Parse(parameters[2]);
}
}
After you have your Dto filled with data you should find the appropriate Dao to handle it
which is basically happens in reflection in this line of the Import() method:
var repository = (DaoBase)_objectResolver.Resolve("DAL.Repository", className + "Dao");
In my case the Dao implements an abstract base class - but it's not that relevant to your problem, your DaoBase can be a simple abstract with a single Save() method.
This way you have a dedicated Dao to CRUD your Dto's - each Dao simply knows how to save for its relevant Dto. Below is the corresponding ProductDao to the ProductDto:
public class ProductDao : DaoBase
{
private const string InsertProductQuery = #"SET foreign_key_checks = 0;
Insert into product (productID, weight, familyID)
VALUES (#productId, #weight, #familyId);
SET foreign_key_checks = 1;";
public override void Save(DtoBase dto)
{
var productToSave = dto as ProductDto;
var saveproductCommand = GetDbCommand(InsertProductQuery);
if (productToSave != null)
{
saveproductCommand.Parameters.Add(CreateParameter("#productId", productToSave.ProductId));
saveproductCommand.Parameters.Add(CreateParameter("#weight", productToSave.Weight));
saveproductCommand.Parameters.Add(CreateParameter("#familyId", productToSave.FamilyId));
ExecuteNonQuery(ref saveproductCommand);
}
}
}
Please ignore the CreateParameter() method, since it's an abstraction from the base classs. you can just use a CreateSqlParameter or CreateDataParameter etc.
Just notice, it's a real naive implementation - you can easily remodel it better, depends on your needs.
From the first impression of your questionc I guess you would be having hugely number of records (more than lacs). If yes I would consider the SQL bulk copies an option. If the record would be less go ahead single record. Insert. The reason for you insert not working is u not providing all the columns of the table and also there's some syntax error.

Categories

Resources