I want to write a file of about 10 million rows. I'm using StreamWriter with a using statement, however it seems that the StreamWriter is not flushing. Here is my code:
public void ExportRecords(IEnumerable<Record> records, string path)
{
using(TextWriter writer = new StreamWriter(path))
{
writer.WriteLine("header");
foreach(var record in records)
{
string line = "";
//Fill line with record properties
writer.WriteLine(line);
}
}
}
At a certain line the StreamWriter doesn't add any rows. So where is the issue?
EDIT
As mentioned is the comments the code is run async.
Here is how it is:
Task.Run(() =>
{
if (!String.IsNullOrEmpty(folderPath))
{
var records = _generator.GenerateData();
_dataService.ExportRecords(records, $"{folderPath}/RECORDS.csv");
//Just a message box
_reporter.ReportMessage($"Generation and exportation finished");
}
else
{
IsGenerating = false;
_reporter.ReportMessage("No output file was created");
}
});
So I did put a breaking point to check if the method starts executing, and it does. I'm just indicating the the code is not stuck in generation method.
Change temporarily the type of your application to Console Application (if it is Windows Application), and add some Console.WriteLine at strategic points so that you have some feedback about the progress of the procedure. Even better you could add logging to your application, using a library like log4net or Serilog. It will be useful in the production phase of your application too, not only in the development/debugging phase.
Flush() has to be manually called at some point if you need it in the middle of the operation, otherwise the stream will only be flushed at the end of the statement, when it is disposed.
Related
I have an app that reads from text files to determine which reports should be generated. It works as it should most of the time, but once in awhile, the program deletes one of the text files it reads from/writes to. Then an exception is thrown ("Could not find file") and progress ceases.
Here is some pertinent code.
First, reading from the file:
List<String> delPerfRecords = ReadFileContents(DelPerfFile);
. . .
private static List<String> ReadFileContents(string fileName)
{
List<String> fileContents = new List<string>();
try
{
fileContents = File.ReadAllLines(fileName).ToList();
}
catch (Exception ex)
{
RoboReporterConstsAndUtils.HandleException(ex);
}
return fileContents;
}
Then, writing to the file -- it marks the record/line in that file as having been processed, so that the same report is not re-generated the next time the file is examined:
MarkAsProcessed(DelPerfFile, qrRecord);
. . .
private static void MarkAsProcessed(string fileToUpdate, string
qrRecord)
{
try
{
var fileContents = File.ReadAllLines(fileToUpdate).ToList();
for (int i = 0; i < fileContents.Count; i++)
{
if (fileContents[i] == qrRecord)
{
fileContents[i] = string.Format("{0}{1} {2}"
qrRecord, RoboReporterConstsAndUtils.COMPLETED_FLAG, DateTime.Now);
}
}
// Will this automatically overwrite the existing?
File.Delete(fileToUpdate);
File.WriteAllLines(fileToUpdate, fileContents);
}
catch (Exception ex)
{
RoboReporterConstsAndUtils.HandleException(ex);
}
}
So I do delete the file, but immediately replace it:
File.Delete(fileToUpdate);
File.WriteAllLines(fileToUpdate, fileContents);
The files being read have contents such as this:
Opas,20170110,20161127,20161231-COMPLETED 1/10/2017 12:33:27 AM
Opas,20170209,20170101,20170128-COMPLETED 2/9/2017 11:26:04 AM
Opas,20170309,20170129,20170225-COMPLETED
Opas,20170409,20170226,20170401
If "-COMPLETED" appears at the end of the record/row/line, it is ignored - will not be processed.
Also, if the second element (at index 1) is a date in the future, it will not be processed (yet).
So, for these examples shown above, the first three have already been done, and will be subsequently ignored. The fourth one will not be acted on until on or after April 9th, 2017 (at which time the data within the data range of the last two dates will be retrieved).
Why is the file sometimes deleted? What can I do to prevent it from ever happening?
If helpful, in more context, the logic is like so:
internal static string GenerateAndSaveDelPerfReports()
{
string allUnitsProcessed = String.Empty;
bool success = false;
try
{
List<String> delPerfRecords = ReadFileContents(DelPerfFile);
List<QueuedReports> qrList = new List<QueuedReports>();
foreach (string qrRecord in delPerfRecords)
{
var qr = ConvertCRVRecordToQueuedReport(qrRecord);
// Rows that have already been processed return null
if (null == qr) continue;
// If the report has not yet been run, and it is due, add i
to the list
if (qr.DateToGenerate <= DateTime.Today)
{
var unit = qr.Unit;
qrList.Add(qr);
MarkAsProcessed(DelPerfFile, qrRecord);
if (String.IsNullOrWhiteSpace(allUnitsProcessed))
{
allUnitsProcessed = unit;
}
else if (!allUnitsProcessed.Contains(unit))
{
allUnitsProcessed = allUnitsProcessed + " and "
unit;
}
}
}
foreach (QueuedReports qrs in qrList)
{
GenerateAndSaveDelPerfReport(qrs);
success = true;
}
}
catch
{
success = false;
}
if (success)
{
return String.Format("Delivery Performance report[s] generate
for {0} by RoboReporter2017", allUnitsProcessed);
}
return String.Empty;
}
How can I ironclad this code to prevent the files from being periodically trashed?
UPDATE
I can't really test this, because the problem occurs so infrequently, but I wonder if adding a "pause" between the File.Delete() and the File.WriteAllLines() would solve the problem?
UPDATE 2
I'm not absolutely sure what the answer to my question is, so I won't add this as an answer, but my guess is that the File.Delete() and File.WriteAllLines() were occurring too close together and so the delete was sometimes occurring on both the old and the new copy of the file.
If so, a pause between the two calls may have solved the problem 99.42% of the time, but from what I found here, it seems the File.Delete() is redundant/superfluous anyway, and so I tested with the File.Delete() commented out, and it worked fine; so, I'm just doing without that occasionally problematic call now. I expect that to solve the issue.
// Will this automatically overwrite the existing?
File.Delete(fileToUpdate);
File.WriteAllLines(fileToUpdate, fileContents);
I would simply add an extra parameter to WriteAllLines() (which could default to false) to tell the function to open the file in overwrite mode, and not call File.Delete() at all then.
Do you currently check the return value of the file open?
Update: ok, it looks like WriteAllLines() is a .Net Framework function and therefore cannot be changed, so I deleted this answer. However now this shows up in the comments, as a proposed solution on another forum:
"just use something like File.WriteAllText where if the file exists,
the data is just overwritten, if the file does not exist it will be
created."
And this was exactly what I meant (while thinking WriteAllLines() was a user defined function), because I've had similar problems in the past.
So, a solution like that could solve some tricky problems (instead of deleting/fast reopening, just overwriting the file) - also less work for the OS, and possibly less file/disk fragmentation.
I'm starting a 3rd party command line application from my C# program using this part of code.
// define Process Start Info and give filename of the 3rd Party app
ProcessStartInfo myProcessStartInfo = new ProcessStartInfo(FileName);
myProcessStartInfo.CreateNoWindow = true;
myProcessStartInfo.UseShellExecute = false;
myProcessStartInfo.RedirectStandardInput = true;
myProcessStartInfo.RedirectStandardOutput = true;
theProcess.StartInfo = myProcessStartInfo;
// instantiate observer class that contains a function
// (ProcessStandardOutput) that is run in separate thread
observer = new StreamObserver();
// create the thread
outputObserverThread = new Thread(observer.ProcessStandardOutput);
// start the 3rd party console application
theProcess.Start();
// set the input stream (to send commands to the console app)
commandStream = theProcess.StandardInput;
// connect the output stream with the observer class
observer.SetOutputStream(theProcess.StandardOutput);
// start the observer thread
outputObserverThread.Start();
// send any command to the console app
SendCommand("init");
This is actually nothing special and has been taken 70% from C# documentation example.
It works so far. I can send commands using the SendCommand() to the console application and I'm getting the response back.
But at one point the output stream gets stuck. Meaning I do not get any text back and even the end of the previous textblock is missing.
Sending commands that result usually in just one line of reply will not be passed to the stream.
A command that usually produces a hugh reply (e.g. "help") will "flush" the stream and I'm getting text through the stream (including the missing data)
This is the (stripped down) implementation of the StreamObserver which processes the output stream:
public class StreamObserver
{
// Volatile is used as hint to the compiler that this data
// member will be accessed by multiple threads.
private volatile bool _shouldStop;
private volatile StreamReader outputStream;
// This method will be called when the thread is started.
public void ProcessStandardOutput()
{
while (!_shouldStop)
{
string line = outputStream.ReadLine();
Console.WriteLine(line);
}
}
public void SetOutputStream(StreamReader stream)
{
outputStream = stream;
}
}
Nothing magic here either....
Any idea what could cause the stream to get stuck and recover when hugh data is present?
I have just counted the returned text. It looks like a block has to contain ~4k data before it is send to the output stream.
Does that ring any bell???
Btw, I'm running on Windows 7 64bit and Use Visual Studio 2013 prof.
i am very new to C#, and this is my first question, please be gentle on me
I am trying to write a application to capture some tick data from the data provider, below is the main part of the program
void zf_TickEvent(object sender, ZenFire.TickEventArgs e)
{
output myoutput = new output();
myoutput.time = e.TimeStamp;
myoutput.product = e.Product.ToString();
myoutput.type = Enum.GetName(typeof(ZenFire.TickType), e.Type);
myoutput.price = e.Price;
myoutput.volume = e.Volume;
using (StreamWriter writer = File.AppendText("c:\\log222.txt"))
{
writer.Write(myoutput.time.ToString(timeFmt) + ",");
writer.Write(myoutput.product + "," );
writer.Write(myoutput.type + "," );
writer.Write(myoutput.price + ",");
writer.Write(myoutput.volume + ",");
}
i have successfully write the data into the text file, however i know that this method will be call like 10000 times a second during peak time, and open a file and append it many times a second is very inefficient, i was pointed to use a buffer or some sort, but i have no idea how to do it, i try reading the document but i still dont understand, thats why i turn in here for help.
Please give me some (working) snippet code so i can pointed to the write direction. thanks
EDIT: i have simplified the code as much as possible
using (StreamWriter streamWriter = File.AppendText("c:\\output.txt"))
{
streamWriter.WriteLine(string.Format("{0},{1},{2},{3},{4}",
e.TimeStamp.ToString(timeFmt),
e.Product.ToString(),
Enum.GetName(typeof(ZenFire.TickType), e.Type),
e.Price,
e.Volume));
}
ED has told me to make my stream to a field, how is the syntax looks like? can anyone post some code to help me? thanks a lot
You need to create a field for the stream instead of a local variable. Initialize it in constructor once and don't forget to close it somewhere. It's better to implement IDisposable interface and close the stream in Dispose() method.
IDisposable
class MyClass : IDisposable {
private StreamWriter _writer;
MyClass() {
_writer = File.App.....;
}
void zf_TickEvent(object sender, ZenFire.TickEventArgs e)
{
output myoutput = new output();
myoutput.time = e.TimeStamp;
myoutput.product = e.Product.ToString();
myoutput.type = Enum.GetName(typeof(ZenFire.TickType), e.Type);
myoutput.price = e.Price;
myoutput.volume = e.Volume;
_writer.Write(myoutput.time.ToString(timeFmt) + ",");
_writer.Write(myoutput.product + "," );
_writer.Write(myoutput.type + "," );
_writer.Write(myoutput.price + ",");
_writer.Write(myoutput.volume + ",");
}
public void Dispose() { /*see the documentation*/ }
}
There are many things you can do
Step 1. Make sure you don't make many io calls and string concatenations.
Output myOutput = new Outoput(e); // Maybe consruct from event args?
// Single write call, single string.format
writer.Write(string.Format("{0},{1},{2},{3},{4},{5}",
myOutput.Time.ToString(),
myOutput.Product,
...);
This I recommend regardless of what your current performance is. I also made some cosmetic changes (variable/property/class name casing. You should look up the difference between variables and properties and their recommended case etc.)
Step 2. Analyse your performance to see if it does what you want. If it does, no need to do anything further. If performance is still too bad, you can
Keep the file open and close it when your handler shuts down.
Write to a buffer and flush it at regular intervals.
Use a logger framework like log4net that internally handles the above for you, and takes care of hairy issues like access to the log file from multiple threads.
I would use String.Format:
using (StreamWriter writer = new StreamWriter(#"c:\log222.txt", true))
{
writer.AutoFlush = true;
writer.Write(String.Format("{0},{1},{2},{3},{4},", myoutput.time.ToString(timeFmt),
myoutput.product, myoutput.type, myoutput.price, myoutput.volume);
}
If you use # before string you don't have to use double \.
This is much faster - you write only once to the file instead of 5 times. Additionally you don't use + operator with strings which is not the fastest operation ;)
Also - if this is multithreading application - you should consider using some lock. It would prevent application from trying to write to the file from eg. 2 threads at one time.
I am developing a c# application, backend as sqlite.In my application i have an option for clean databse.It means the curren .db file will delete using File.Delete method and again it create empty databse using File.create method.Now let me explain the problem.
To perform cleandatabse task, i have to stop all the process which is running ,after doing that if i click on clean database it is throwing an error that file cannot delete, it is being used by another process.i am able to stop all the thread which is running.
Somehow i am able to find which process is blocikng the file ,
foreach (var process in Process.GetProcesses()) {
var files = GetFilesLockedBy(process);
if (files.Contains(filePath))
{
procs.Add(process);
Console.WriteLine(process.ProcessName);
process.Kill();
File.Delete(filePath);
}
}
But in the above code i used process.Kill, which close the window form which i am running.
without using kill, i tried close and dispose which doesn't work for me.
Can you please help me to release the file from the process without closing the application and then yo delete the db file.
Thank you in advance
Best regards
Sangita.
You should make sure you close every stream you open it:
using (Stream str = File.Create("C:\\h.txt"))
{
// your code here
} // the stream will be automatically closed here
if you don't put this using statement, it will cause you a lot of bugs, even if you close it manually str.Close();
Streamss are disposable types, you must manage their lifetime manually, either by that using syntax, e.g.:
using (StreamReader f = new ...) {
}
... or by doing it more verbosely (this syntax is required if you allocate and delete the Stream in different code-blocks/functions):
try {
StreamReader f = new ...;
...
} finally {
if (null != f) f.Dispose();
}
... or by making the holding class an IDisposable by itself. See also What Your Mother Never Told You About Resource Deallocation.
Interestingly, this seems to be a practical incarnation of one of those https://stackoverflow.com/questions/2245196/c-urban-myths/2245382#2245382 :
0) In C++, you must mess around with pointers, that's old and dangerous, use C#
Gee, #include boost/shared_ptr> or one of the like. Actually, it is often easier to produce mess in your sowonderful C#:
static void Main () {
foo();
bar();
}
static void foo () {
var f = new StreamWriter ("hello.txt");
f.Write ("hello world");
}
static void bar () {
var f = new StreamReader ("hello.txt");
Console.WriteLine (f.ReadToEnd ());
}
"Unhandled IOException: The process cannot access the file 'hello.txt' because it is being used by another process."
Those claims, btw, are often made by those who happen to never have heard of RAII, and about how far you can get without even smart-pointers.
Not sure but you may be calling Kill on the current process.
EDIT : call Delte after the loop.
Try this :
int currentProcessId = Process.GetCurrentProcess().Id;
foreach (var process in Process.GetProcesses()) {
if (process.Id != currentProcessId)
{
var files = GetFilesLockedBy(process);
if (files.Contains(filePath))
{
procs.Add(process);
Console.WriteLine(process.ProcessName);
process.Kill();
}
}
}
File.Delete(filePath);
Moreover Close doesn't terminate the process, you have to call CloseMainWindow or Kill.
I have a method in my windows service as follows:
System.IO.File.Copy(path, ConfigurationManager.AppSettings["BulkInsertGiftRegisterCreatorDirectory"] + System.IO.Path.GetFileName(path));
Loyalty.Entity.Base.FileInfo file = new Loyalty.Entity.Base.FileInfo();
file.FileName = path;
request.Object = file;
ResponseBase response = new ResponseBase(request);
RequestConnection connection = new RequestConnection("cn");
FileManager fileManager = new FileManager(request, connection);
response = fileManager.OfflineGiftRegisterBulkInsert();
System.IO.File.Delete(ConfigurationManager.AppSettings["BulkInsertGiftRegisterCreatorDirectory"] + System.IO.Path.GetFileName(path));
// here is the part of stored procedure that uses file
SELECT #SCRIPT= 'BULK INSERT GIFT_CARD.GIFT_TEMP'
+' FROM '''
+ #FILE_PATH
+''' WITH ('
+'FIELDTERMINATOR = '','','
+ 'KEEPNULLS'
+');'
I can delete the file from file system by hand, but this code says me "Ooops! System.IO.IOException: The process cannot access the file 'filename'
because it is being used by another process."
I've searched the similar questions on stackoverflow and else where. But I could not find anything to help me. Copy or Delete methods return void and I have no stream in my code to dispose.
How can I fix it?
Thanks in advance.
Here is a method for you to check if a file is in use:
public static System.Boolean FileInUse(System.String file)
{
try
{
if (!System.IO.File.Exists(file)) // The path might also be invalid.
{
return false;
}
using (System.IO.FileStream stream = new System.IO.FileStream(file, System.IO.FileMode.Open))
{
return false;
}
}
catch
{
return true;
}
}
Also, to wait for a file I have made:
public static void WaitForFile(System.String file)
{
// While the file is in use...
while (FileInUse(file)) ; // Do nothing.
}
I hope this helps!
Before you go looking through the code you might want to use process explorer to Identify what process has the handle. This might rule out some issue you haven't thought of
update
Since you are using a timer you must make sure that your method is reentrant and you don't have any race conditions.. E.g. the timer ticks faster than you can process the event.
See this question
And this answer
Even better solution, is to add this two lines of code before using the function FileInUse that Vercas showed:
GC.Collect();
GC.WaitForPendingFinalizers();