I've been working on a small program for a friend of mine who has a very large file which I read into the a datagridview > Modify data > export to csv. I managed to make everything work relevatively well until recently when he asked me to make some changes to the way the data is exported. For some reason, I am getting an Out of Memory exception when running this function.
private void ExportData(int fileNum = 1, int rowCount = 0)
{
int lastRow = rowCount;
if (!Directory.Exists(ExportPath + dataFilePath.Name))
Directory.CreateDirectory(ExportPath + dataFilePath.Name);
StreamWriter sw = new StreamWriter(ExportPath + dataFilePath.Name + #"\" + dataFilePath.Name + "_" + fileNum + ".csv");
//var headers = dataGridView1.Columns.Cast<DataGridViewColumn>();
//sw.WriteLine(string.Join(",", headers.Select(column => "\"" + column.HeaderText + "\"").ToArray()));
sw.WriteLine("Unit,UPC,Brand,Vendor,List Cost,QTY,Price,Description,Attribute 1,Attribute 2," +
"Descriptor 1,Descriptor 2,Descriptor 3,Descriptor 4,Descriptor 5,Descriptor 6,Descriptor 7,Descriptor 8");
for (int i = 0; i < 50000; i++)
{
rowCount = lastRow + i;
if (rowCount >= dataGridView1.RowCount)
break;
var cells = dataGridView1.Rows[rowCount].Cells.Cast<DataGridViewCell>();
sw.WriteLine(string.Join(",", cells.Select(cell => "\"" + cell.Value + "\"").ToArray()));
}
sw.Close();
sw.Dispose();
lastRow = rowCount + 1;
if (lastRow < dataGridView1.RowCount - 1)
ExportData(fileNum + 1, lastRow);
else
{
progressBar1.BeginInvoke(new MethodInvoker(delegate {
progressBar1.Style = ProgressBarStyle.Blocks;
button_OpenDataFile.Enabled = true;
button_ConvertFromRaw.Enabled = true;
button_exportLS.Enabled = true;
Console.WriteLine("[Main] Export complete.");
}));
}
}
var cells = dataGridView1.Rows[rowCount].Cells.Cast<DataGridViewCell>();
seems to be the line the error occurs on.
Could anyone provide any insight into what Im doing wrong?
Thank you!
Do this experiment: convert your code to a loop, instead of using recursion:
private void ExportData()
{
//You only need to do this once, take it out of the loop.
if (!Directory.Exists(ExportPath + dataFilePath.Name))
Directory.CreateDirectory(ExportPath + dataFilePath.Name);
var fileNum = 0;
var rowCount = 0;
while (rowCount < dataGridView1.RowCount)
{
fileNum = fileNum + 1;
using (StreamWriter sw = new StreamWriter(ExportPath + dataFilePath.Name + #"\" + dataFilePath.Name + "_" + fileNum + ".csv")
{
sw.WriteLine("Unit,UPC,Brand,Vendor,List Cost,QTY,Price,Description,Attribute 1,Attribute 2," +
"Descriptor 1,Descriptor 2,Descriptor 3,Descriptor 4,Descriptor 5,Descriptor 6,Descriptor 7,Descriptor 8");
for (int i = 0; i < 50000; i++)
{
rowCount = rowCount + 1;
if (rowCount >= dataGridView1.RowCount)
break;
var cells = dataGridView1.Rows[rowCount].Cells.Cast<DataGridViewCell>();
sw.WriteLine(string.Join(",", cells.Select(cell => "\"" + cell.Value + "\"").ToArray()));
}
} //sw.Close() and sw.Dispose() not needed because of the 'using'. You may want to do sw.Flush().
}
//The 'else' part of your original recursive method
progressBar1.BeginInvoke(new MethodInvoker(delegate {
progressBar1.Style = ProgressBarStyle.Blocks;
button_OpenDataFile.Enabled = true;
button_ConvertFromRaw.Enabled = true;
button_exportLS.Enabled = true;
Console.WriteLine("[Main] Export complete.");
}));
}
Does the error go away? Probably yes. Recursion uses a lot of memory in a stack and does not release it until the end of the recursion, when going up the stack again. The line where you get the error just happens to try and add to memory the contents of a whole line in your csv file. That may be the last drop that causes the out of memory exception if the memory is already almost full with the recursion stack.
I removed some accumulator vars that seemed redundant, I hope I didn't mess up with the ranges of the loops.
I removed the CreateDirectory from the loop, and added a using statement for the StreamWriter. I don't think those were the reasons for your error, as the directory was created only once, and you were disposing the StreamWriter before the recursive call, but anyway if you want to confirm it you can try undoing those changes in the non-recursive code one by one, and see if the error happens again.
So I figured it out.
I guess iterating through the datagridview just isn't the way to go. Instead, I just exported the data using my data source. It goes a hell of a lot faster... down from 2 minutes to about 2 seconds.
Thank you, everyone, for the help!
private void ExportData()
{
//You only need to do this once, take it out of the loop.
if (!Directory.Exists(ExportPath + dataFilePath.Name))
Directory.CreateDirectory(ExportPath + dataFilePath.Name);
var fileNum = 0;
var rowCount = 0;
while (rowCount < dataGridView1.RowCount)
{
fileNum = fileNum + 1;
using (StreamWriter sw = new StreamWriter(ExportPath + dataFilePath.Name + #"\" + dataFilePath.Name + "_" + fileNum + ".csv"))
{
sw.WriteLine("Unit,UPC,Brand,Vendor,List Cost,QTY,Price,Description,Attribute 1,Attribute 2" +
"Descriptor 1,Descriptor 2,Descriptor 3,Descriptor 4,Descriptor 5,Descriptor 6,Descriptor 7,Descriptor 8");
for (int i = 0; i < 50000; i++)
{
rowCount = rowCount + 1;
if (rowCount >= dataGridView1.RowCount)
break;
var s = new string[]
{
"\"" + DATA[rowCount].Unit + "\"",
"\"" + DATA[rowCount].UPC + "\"",
"\"" + DATA[rowCount].Brand + "\"",
"\"" + DATA[rowCount].Vendor + "\"",
"\"" + DATA[rowCount].List_Cost + "\"",
"\"" + DATA[rowCount].Quantity.ToString() + "\"",
"\"" + DATA[rowCount].Price + "\"",
"\"" + DATA[rowCount].Description + "\"",
"\"" + DATA[rowCount].Attribute_1 + "\"",
"\"" + DATA[rowCount].Attribute_2 + "\"",
"\"" + DATA[rowCount].Descriptor_1 + "\"",
"\"" + DATA[rowCount].Descriptor_2 + "\"",
"\"" + DATA[rowCount].Descriptor_3 + "\"",
"\"" + DATA[rowCount].Descriptor_4 + "\"",
"\"" + DATA[rowCount].Descriptor_5 + "\"",
"\"" + DATA[rowCount].Descriptor_6 + "\"",
"\"" + DATA[rowCount].Descriptor_7 + "\"",
"\"" + DATA[rowCount].Descriptor_8 + "\""
};
sw.WriteLine(string.Join(",", s));
sw.Flush();
}
} //sw.Close() and sw.Dispose() not needed because of the 'using'. You may want to do sw.Flush().
}
//The 'else' part of your original recursive method
progressBar1.BeginInvoke(new MethodInvoker(delegate
{
progressBar1.Style = ProgressBarStyle.Blocks;
button_OpenDataFile.Enabled = true;
button_ConvertFromRaw.Enabled = true;
button_exportLS.Enabled = true;
Console.WriteLine("[Main] Export complete.");
}));
}'
Related
I'm working on a bit of code for school but I keep getting an ArgumentOutOfRangeException
With this code I'm trying to read some data from a .csv file and if it equals the name of the image I want it to remove it from the .csv file whilst keeping the structure intact.
public void checkPair(Image card1, Image card2)
{
this.Image1 = card1;
this.Image2 = card2;
if (Convert.ToString(card1.Source) == Convert.ToString(card2.Source) && (card1 != card2))
{
getPoint(card1, card2);
string path = #"Save1.csv";
var reader = new StreamReader(File.OpenRead(path));
var data = new List<List<string>>();
while (!reader.EndOfStream)
{
var line = reader.ReadLine();
var values = line.Split(';');
data.Add(new List<String> { values[0], values[1]
});
}
reader.Close();
string delimiter = ";";
for (int i = 1; i < 5; i++)
{
for (int x = 0; x < 4; x++)
{
if (data[i][x] == Convert.ToString(card1.Source))
{
data[i][x] = null;
}
}
}
File.WriteAllText(path, data[0][0] + delimiter + data[0][1] + Environment.NewLine + data[1][0] + delimiter + data[1][1] + delimiter + data[1][2] + delimiter + data[1][3] + Environment.NewLine + data[2][0] + delimiter + data[2][1] + delimiter + data[2][2] + delimiter + data[2][3] + Environment.NewLine + data[3][0] + delimiter + data[3][1] + delimiter + data[3][2] + delimiter + data[3][3] + Environment.NewLine + data[4][0] + delimiter + data[4][1] + delimiter + data[4][2] + delimiter + data[4][3] + Environment.NewLine + "ready");
I have no idea why I get this error and how to fix it
Initially, I'd change your last line from
File.WriteAllText(path, data[0][0] + delimiter + data[0][1] ....
to something like
var obj1 = data[0][0];
var obj2 = data[0][1];
File.WriteAllText(path, obj1 + delimiter + obj2 .... etc)
If you over inline functions or array accessing, when you get an exception the stack trace won't be that helpful. At least you'll have an idea of the statement that caused the issue.
This technique can prove to be very helpful, if you are looking at an in exception in the logs, after the fact.
How do I extract all VBA Code (Forms, Moduls, Reports) out of more than hundred Access databases via c#.
It is not possible to change the databases an/or add vba code to them. The extraction of the Code has to be done read-only.
I tried this Code:
var appClass = new ApplicationClass();
appClass.OpenCurrentDatabase(#"C:\Temp\Test\FBIOE.mdb", false, "");
Console.WriteLine(appClass.Version);
Console.WriteLine(appClass.Modules.Count.ToString());
Console.WriteLine(appClass.Modules.Parent.ToString());
int NumOfLines = 0;
Console.WriteLine("Anzahl Module:" + appClass.Modules.Count.ToString());
for (int i = 0; i < appClass.Modules.Count; i++)
{
Console.WriteLine(appClass.Modules[i].Name + " : " + appClass.Modules[i].CountOfLines);
NumOfLines += appClass.Modules[i].CountOfLines;
//Console.WriteLine(appClass.Modules[i].Name + " : " + appClass.Modules[i].ToString());
}
Console.WriteLine("Number of Lines : " + NumOfLines);
Console.ReadKey();
But this Code has some Problems:
It executes the autoexec-macro which is a very bad thing as all of the databases are doing different dangerouse things when starting.
it seems not to get every module. A lot of them seems to be skippted. In my test db there are more than 53 modules (not counting forms and reports) but appClass.Modules.Count is 44 (and there are forms and Reports included)
Edit:
I found a way to read all the Code:
appClass.OpenCurrentDatabase(tempFile, false, "");
Debug.WriteLine("appClass.CurrentProject.AllForms.Count:" + appClass.CurrentProject.AllForms.Count.ToString());
Debug.WriteLine("appClass.CurrentProject.AllMacros.Count:" + appClass.CurrentProject.AllMacros.Count.ToString());
Debug.WriteLine("appClass.CurrentProject.AllModules.Count:" + appClass.CurrentProject.AllModules.Count.ToString());
Debug.WriteLine("appClass.CurrentProject.AllReports.Count:" + appClass.CurrentProject.AllReports.Count.ToString());
var currentProject = appClass.CurrentProject;
for (int i = 0; i < appClass.CurrentProject.AllForms.Count; i++)
{
var form = appClass.CurrentProject.AllForms[i];
Debug.WriteLine("Erledige: " + file+ " Item: " + form.FullName);
appClass.SaveAsText(AcObjectType.acForm, form.FullName, absolutesVerzeichnis + "Form_"+form.FullName + ".txt");
}
for (int i = 0; i < appClass.CurrentProject.AllMacros.Count; i++)
{
var macro = appClass.CurrentProject.AllMacros[i];
Debug.WriteLine("Erledige: " + file + " Item: " + macro.FullName);
appClass.SaveAsText(AcObjectType.acMacro, macro.FullName, absolutesVerzeichnis + "Makro_" + macro.FullName + ".txt");
}
for (int i = 0; i < appClass.CurrentProject.AllModules.Count; i++)
{
var module = appClass.CurrentProject.AllModules[i];
Debug.WriteLine("Erledige: " + file + " Item: " + module.FullName);
appClass.SaveAsText(AcObjectType.acModule, module.FullName, absolutesVerzeichnis + module.FullName + ".txt");
}
for (int i = 0; i < appClass.CurrentProject.AllReports.Count; i++)
{
var report = appClass.CurrentProject.AllReports[i];
Debug.WriteLine("Erledige: " + file + " Item: " + report.FullName);
appClass.SaveAsText(AcObjectType.acReport, report.FullName, absolutesVerzeichnis + "Report_" + report.FullName + ".txt");
}
This works fine. But the main problem stays the same:
The autoexec code is executed and if there are errors inside, the extraction stops.
I'm getting a OutOfMemory exception when running the following code, it happens on the File.ReadLines line, it processes most files fine until it hits larger files.
It's consistantly using tons of memory and cpu during the whole process though.
The file it crashed on is only 156,000KB, which is 156mb
static void Main(string[] args)
{
Console.CursorVisible = false;
Console.ForegroundColor = ConsoleColor.Green;
Console.WriteLine();
Console.WriteLine(" [" + DateTime.Now.ToShortTimeString() + "]" + " Connected to the Cassandra Database");
Console.WriteLine();
Console.ForegroundColor = ConsoleColor.White;
string filepath = #"C:\Users\admin\Desktop\wecrack lists";
DirectoryInfo directory = new DirectoryInfo(filepath);
int fileCount = 0;
var client = new MongoClient("mongodb://localhost:27017");
var database = client.GetDatabase("cracking");
var collection = database.GetCollection<Password>("passwords");
foreach (var file in directory.GetFiles("*"))
{
fileCount++;
Console.WriteLine(" [" + DateTime.Now.ToShortTimeString() + "]" + " Working through file: {" + file + "} {" + fileCount + "/" + directory.GetFiles("*").Count() + "}");
List<Password> entitys = new List<Password>();
foreach (string line in File.ReadLines(filepath + #"\" + file.ToString()))
{
entitys.Add(new Password { password = line });
}
collection.InsertManyAsync(entitys);
}
Console.WriteLine();
Console.WriteLine(" [" + DateTime.Now.ToShortTimeString() + "]" + " Finished inserting records, press any key to get the count.");
Console.ReadKey(true);
while (true)
{
Console.ReadKey(true);
}
}
Try batching your updates. That way you won't have all that data in memory at the same time. It may also help you not totally lock up your database.
...
foreach (var file in directory.GetFiles("*"))
{
fileCount++;
Console.WriteLine(" [" + DateTime.Now.ToShortTimeString() + "]" + " Working through file: {" + file + "} {" + fileCount + "/" + directory.GetFiles("*").Count() + "}");
System.IO.StreamReader file = new System.IO.StreamReader(filepath + #"\" + file.ToString());
while(!file.EndOfStream)
{
int passwordBatchCount = 0;
List<Password> entitysBatch = new List<Password>();
while ((string line = file.ReadLine()) != null && passwordBatchCount < BATCH_SIZE)
{
entitysBatch.Add(new Password { password = line });
passwordBatchCount++;
}
collection.InsertManyAsync(entitysBatch);
}
file.Close();
}
}
...
I'm having issues with a C# Winforms file IO. The code complies just fine, but then it returns errors on execution.
The output code is here:
private void saveData()
{
string fullPath = System.Environment.GetEnvironmentVariable(#"%MyDocuments%\HellsingRPG\");
StreamWriter writer = new StreamWriter(fullPath + textBox2.Text + ".txt");
writer.WriteLine(textBox1.Text + "," + textBox2.Text + "," + textBox3.Text + "," + textBox4.Text + "," + comboBox1.SelectedText + "," +
numericUpDown25.Value + "," + numericUpDown1.Value + "," + numericUpDown2.Value + "," + numericUpDown3.Value + "," + numericUpDown4.Value + "," +
numericUpDown5.Value + "," + numericUpDown6.Value + "," + numericUpDown7.Value + "," + numericUpDown8.Value + "," + numericUpDown9.Value + "," +
numericUpDown10.Value + "," + numericUpDown11.Value + "," + numericUpDown12.Value + "," + numericUpDown13.Value + "," + numericUpDown14.Value
+ "," + numericUpDown15.Value + "," + numericUpDown16.Value + "," + numericUpDown17.Value + "," + numericUpDown18.Value + "," +
numericUpDown19.Value + "," + numericUpDown20.Value + "," + numericUpDown21.Value + "," + numericUpDown22.Value);
writer.Close();
}
And the code to load the data is here:
private void loadData()
{
Stream myStream = null;
OpenFileDialog openFileDialog1 = new OpenFileDialog();
openFileDialog1.InitialDirectory = System.Environment.GetEnvironmentVariable(#"%MyDocuments%\HellsingRPG\");
openFileDialog1.Filter = "txt files (*.txt)|*.txt|All files (*.*)|*.*";
openFileDialog1.FilterIndex = 2;
openFileDialog1.RestoreDirectory = true;
if (openFileDialog1.ShowDialog() == DialogResult.OK)
{
try
{
if ((myStream = openFileDialog1.OpenFile()) != null)
{
using (myStream)
{
List<string> myData = parseCSV(System.Convert.ToString(myStream));
textBox1.Text = myData[0];
textBox2.Text = myData[1];
textBox3.Text = myData[3];
textBox4.Text = myData[4];
comboBox1.SelectedText = myData[5];
numericUpDown25.Value = System.Convert.ToDecimal(myData[6]);
numericUpDown1.Value = System.Convert.ToDecimal(myData[7]);
numericUpDown2.Value = System.Convert.ToDecimal(myData[8]);
numericUpDown3.Value = System.Convert.ToDecimal(myData[9]);
numericUpDown4.Value = System.Convert.ToDecimal(myData[10]);
numericUpDown5.Value = System.Convert.ToDecimal(myData[11]);
numericUpDown6.Value = System.Convert.ToDecimal(myData[12]);
numericUpDown7.Value = System.Convert.ToDecimal(myData[13]);
numericUpDown8.Value = System.Convert.ToDecimal(myData[14]);
numericUpDown9.Value = System.Convert.ToDecimal(myData[15]);
numericUpDown10.Value = System.Convert.ToDecimal(myData[16]);
numericUpDown11.Value = System.Convert.ToDecimal(myData[17]);
numericUpDown12.Value = System.Convert.ToDecimal(myData[18]);
numericUpDown13.Value = System.Convert.ToDecimal(myData[19]);
numericUpDown14.Value = System.Convert.ToDecimal(myData[20]);
numericUpDown15.Value = System.Convert.ToDecimal(myData[21]);
numericUpDown16.Value = System.Convert.ToDecimal(myData[22]);
numericUpDown17.Value = System.Convert.ToDecimal(myData[23]);
numericUpDown18.Value = System.Convert.ToDecimal(myData[24]);
numericUpDown19.Value = System.Convert.ToDecimal(myData[25]);
numericUpDown20.Value = System.Convert.ToDecimal(myData[26]);
numericUpDown21.Value = System.Convert.ToDecimal(myData[27]);
numericUpDown22.Value = System.Convert.ToDecimal(myData[28]);
}
}
}
catch (Exception ex)
{
MessageBox.Show("Error: Could not read file from disk. Original error: " + ex.Message);
}
}
}
And that compiles just fine. But when I use it, I get the following errors:
"Could not find file "C:\Users\collmark\Documents\Visual Studio
2015\Projects\WindowsFormsApplication1\WindowsFormsApplication1\bin\Release\System.IO.Filestream".
"Error: Could not read file from disk. Original error: Index out of
range. Must be non-negative and less than the size of the collection.
Parameter name: index."
Thanks
your save data seems to save 22 fields while the read expects 28.
I suspect the myData object does not contain the fields index you are trying to read, hence index out of range.
do yourself a favour when printing exception data don't limit yourself to the message but print the whole stack trace, it will tell you which line is faulty giving you a hint at the actual problem.
MessageBox.Show("Error: Could not read file from disk. Original error: " + ex.ToString());
I need to create x number of files (a set) but I must first check to see if the files exist with a similar name.
For example, tiftest1.tif, tiftest2.tif, ... exist and I must write tiftest again to the same directory. I'd like to append _x to the end of the filename, where x is a number that is auto-incremented, each time that I want to create the set. So I can have tiftest1_1.tif,tiftest2_1.tif, tiftest1_2.tif, tiftest2_2.tif, tiftest1_3.tif, tiftest2_3.tif and so forth.
Here is what I have so far:
...
DirectoryInfo root = new DirectoryInfo(fileWatch.Path);
FileInfo[] exist = root.GetFiles(fout + "*.tif");
if (exist.Length > 0)
{
int cnt = 0;
do
{
cnt++;
DirectoryInfo root1 = new DirectoryInfo(fileWatch.Path);
FileInfo[] exist1 = root.GetFiles(fout + "*" + "_" + cnt + ".tif");
arg_proc = "-o " + "\"" + fileWatch.Path
+ "\\" + fout + "%03d_" + cnt + ".tif\" -r " + "\"" + openDialog.FileName + "\"";
} while (exist1.Length > 0); //exist1 is out of scope so this doesn't work
}
else
{
arg_proc = "-o " + "\"" + fileWatch.Path
+ "\\" + fout + "%03d.tif\" -r " + "\"" + openDialog.FileName + "\"";
}
...
exist1.length is out of scope so the loop will continually run. I'm not certain how to correct this. My method was to originally scan the directory for a match and see if the length of the array is greater than 0. If it is > 0 then the _x will autoincrement until a match isn't found. arg_proc is a string used in a function (not included) which will create the files.
Can't you just reuse your exist variable?
FileInfo[] exist = root.GetFiles(fout + "*.tif");
if (exist.Length > 0)
{
int cnt = 0;
do
{
cnt++;
DirectoryInfo root1 = new DirectoryInfo(fileWatch.Path);
exist = root.GetFiles(fout + "*" + "_" + cnt + ".tif");
arg_proc = "-o " + "\"" + fileWatch.Path + "\\"
+ fout + "%03d_" + cnt + ".tif\" -r " + "\"" + openDialog.FileName + "\"";
} while (exist.Length > 0);
}
...
This way, exist won't be out of scope. It didn't look like you needed the original list of files once you started incrementing your counter, so if that is the case, you can just keep reusing exist to count your existing filenames.