Hi I have created a console application for kafka consumer for receiving messages.
class Program
{
static void Main(string[] args)
{
string topic = "IDGTestTopic";
Uri uri = new Uri("http://localhost:9092");
var options = new KafkaOptions(uri);
var router = new BrokerRouter(options);
var consumer = new Consumer(new ConsumerOptions(topic, router));
foreach (var message in consumer.Consume())
{
Console.WriteLine(Encoding.UTF8.GetString(message.Value));
//Saving messages in files
string lines = Encoding.UTF8.GetString(message.Value);
System.IO.File.WriteAllText(#"C:\Project\Kafka Research\Kafka_Consumer\Kafka_Consumer\KafkaMessages\Messages.txt", lines);
}
}
}
But it only store the current messages. if you see the console all the messages are displaying.
But if you see the text file it only contain current messages
How to save all the messages in a file ?
For each message, System.IO.File.WriteAllText overwrites the file and therefore the created file will contain only the latest message.
In order to keep all the messages in a single file, you can replace System.IO.File.WriteAllText with System.IO.File.AppendAllText as shown below:
foreach (var message in consumer.Consume()) {
Console.WriteLine(Encoding.UTF8.GetString(message.Value));
//Saving messages in files
string lines = Encoding.UTF8.GetString(message.Value);
System.IO.File.AppendAllText(#"C:\Project\Kafka Research\Kafka_Consumer\Kafka_Consumer\KafkaMessages\Messages.txt", lines);
}
According to the docs,
File.AppendAllText Method (String, String)
Opens a file, appends the specified string to the file, and then
closes the file. If the file does not exist, this method creates a
file, writes the specified string to the file, then closes the file.
and File.WriteAllText Method (String, String)
Creates a new file, writes the specified string to the file, and then
closes the file. If the target file already exists, it is overwritten.
Every time you consume a message, you overwrite the whole file:
System.IO.File.WriteAllText
You need to do this outside the consume loop.
# Giorgos Myrianthous Your previous second choice was better in some ways. Appending to a stringbuilder and writing only once outside the loop to a file is most likely much faster then going through IO many times in every loop. Here is what I was suggesting:
StringBuilder linebuilder = new StringBuilder(); //this line outside the loop
foreach (var message in consumer.Consume()) {
Console.WriteLine(Encoding.UTF8.GetString(message.Value));
//Saving messages in files
linebuilder.Append(Encoding.UTF8.GetString(message.Value)); //this line inside the loop
}
System.IO.File.AppendAllText(#"C:\Project\Kafka Research\Kafka_Consumer\Kafka_Consumer\KafkaMessages\Messages.txt", linebuilder.ToString(());
Related
I need to create a file, write one line of text in the file and then delete the file and estimate how long it will take to do it.
Unfortunately, I am running in couple of problems, first I cannot write in the file, it succesfully creates it but nothing is written to it.
Secondly, I cannot delete the file because it has been used by another process.
Please help.
I have been trying to delete it for quite some time.
I have also tried wrapping it in usings, to no avail.
Writing to the file is the same situation. I even changed it so the file ends in .txt but that does not make any difference.
public static void ProcessFile(string path)
{
string fullpath = path;
int lastBracket = path.LastIndexOf("\\");
// the filename does not contain .html so it can be named to .txt
string newFileName = path.Substring(lastBracket + 1, path.Length - lastBracket - 6) + " hrefcount.txt";
string newPath = Path.Combine(fullpath.Substring(0, lastBracket), newFileName);
Console.WriteLine(newPath);
int counter = 0;
foreach (var line in File.ReadAllLines(path))
{
if (line.Contains("href="))
{
counter++;
}
}
var fileCreated = File.CreateText(newPath);
fileCreated.WriteLine("The number of times href appears is " + counter);
Console.WriteLine();
File.Delete(newPath);
}
File created, nothing written to it, unable to delete due to has been used by another process.
Instead of File.CreateText() use File.WriteAllText(path, content). It writes the text and then closes the file allowing you to delete it if necessary
Instead of the following
var fileCreated = File.CreateText(newPath);
fileCreated.WriteLine("The number of times href appears is " + counter);
You may write
File.WriteAllText(newPath, $"The number of times href appears is {counter}");
Refer documentation here
The issue with your approach is that CreateText() is used to write to a stream. But in your case, it is not necessary since you're writing all the text at once to the file and that text is small in size.
The cause of your error is the fact that you don't close and dispose the variable fileCreated. This, is a FileStream and until you close and dispose this variable the file is not available to anyone, even your own code that has opened the file.
So the first thing to do is
using (var fileCreated = File.CreateText(newPath))
{
fileCreated.WriteLine("The number of times href appears is " + counter);
}
The using block ensure the proper disposal of the variable.
However there are other parts of your code that you can simplify
public static void ProcessFile(string path)
{
string folder = Path.GetDirectoryName(path);
string file = Path.GetFileName(path);
// Keep the first 6 characters from the source file?
string newFile = file.Substring(0, 6) + " hrefcount.txt";
string newPath = Path.Combine(folder, newFile);
// A single line to retrieve your counter thanks to IEnumerables and Linq
int counter = File.ReadLines(path).Count(x => x.Contains("href="));
// Create, but dispose also the file
using (var fileCreated = File.CreateText(newPath))
{
fileCreated.WriteLine("The number of times href appears is " + counter);
}
// Now you should be free to delete the file
File.Delete(newPath);
}
I cannot delete the file because it has been used by another process.
Probably you're not disposed your files after creating. To do that, you should additionally use FileStream.Dispose method:
File.Create(path).Dispose();
estimate how long it will take to do it
You can use System.Diagnostics.Stopwatch class to do that:
Stopwatch stopwatch = new Stopwatch();
stopwatch.Start();
/*
do the magic
*/
stopwatch.Stop();
Console.WriteLine(stopwatch.ElapsedMilliseconds);
You can use File.WriteAllText method instead of File.CreateText to write your text to file:
File.WriteAllText(path, myText);
Remember that since the .NET 4 you can use this method with array or List<T> too instead of string.
File.Create() supports Dispose method which help you to release that file resource to perform further operations
To perform operations on file follow below steps:
Create file and free up the resource using Dispose().
File.Create(newPath).Dispose();
or Use StreamWriter to create file and write text to it.
using( StreamWriter sw = new StreamWriter(newPath, true)
{
sw.Write($"The number of times href appears is {counter}"); //Used string interpolation
}
StreamWriter calls Dispose() function to release file resource.
When Writer release control over file, then you will not face issue related to I cannot delete the file because it has been used by another process.
Now you can delete file using,
File.Delete(newPath);
MSDN : IDisposable.Dispose Method
Performs application-defined tasks associated with freeing, releasing,
or resetting unmanaged resources.
In the .net framework using a Windows Forms app I can purge a file, then write the data that I want back to into that file.
Here is the code that I use in Windows Forms:
var openFile = File.OpenText(fullFileName);
var fileEmpty = openFile.ReadLine();
if (fileEmpty != null)
{
var lines = File.ReadAllLines(fullFileName).Skip(4); //Will skip the first 4 then rewrite the file
openFile.Close();//Close the reading of the file
File.WriteAllLines(fullFileName, lines); //Reopen the file to write the lines
openFile.Close();//Close the rewriting of the file
}
openFile.Close();
openFile.Dispose();
I am trying to do the same thing the compact framework. I can keep the lines that I want, and then delete all the lines in the file. However I am not able to rewrite the file.
Here is my compact framework code:
var sb = new StringBuilder();
using (var sr = new StreamReader(fullFileName))
{
// read the first 4 lines but do nothing with them; basically, skip them
for (int i = 0; i < 4; i++)
sr.ReadLine();
string line1;
while ((line1 = sr.ReadLine()) != null)
{
sb.AppendLine(line1);
}
}
string allines = sb.ToString();
openFile.Close();//Close the reading of the file
openFile.Dispose();
//Reopen the file to write the lines
var writer = new StreamWriter(fullFileName, false); //Don't append!
foreach (char line2 in allines)
{
writer.WriteLine(line2);
}
openFile.Close();//Close the rewriting of the file
}
openFile.Close();
openFile.Dispose();
Your code
foreach (char line2 in allines)
{
writer.WriteLine(line2);
}
is writing out the characters of the original file, each on a separate line.
Remember, allines is a single string that happens to have Environment.NewLine between the original strings of the file.
What you probably intend to do is simply
writer.WriteLine(allines);
UPDATE
You are closing openFile a number of times (you should only do this once), but you are not flushing or closing your writer.
Try
using (var writer = new StreamWriter(fullFileName, false)) //Don't append!
{
writer.WriteLine(allines);
}
to ensure the writer is disposed and therefore flushed.
If you plan to do this to have something like a "rotating" buffer for a log file consider that most Windows CE devices uses flash as storage media and your approach will generate a full re-write of the whole file (whole - 4 lines) every time. If this happens quite often (every few seconds) this may wear our the flash, reaching its maximum number of erase cycles quickly (quickly may mean a few weeks or months).
An alternative approach would be rename the old log file when it has reached the maximum size (deleting any existing file with the same name) and create a new one.
In this was you logging info would be split on two files but you'll always append to the existing files, limiting the number of writes you perform. Also renaming or deleting a file aren't heavy operations from the point of view of a flash file system.
In my application the user can drag and drop multiple text files onto a GUI control to convert them to another format. Here is the relevant code:
private void panelConverter_DragDrop(object sender, DragEventArgs e)
{
string[] filenames = (string[])e.Data.GetData(DataFormats.FileDrop);
foreach (string filename in filenames)
{
convertFile(filename);
}
}
private void convertFile(string filename)
{
// build name of output file
string convertedFile = Path.ChangeExtension(filename, ".out");
// open input file for reading
FileInfo source = new FileInfo(filename);
StreamReader srcStream = source.OpenText();
// open output file for writing
StreamWriter dstStream = new StreamWriter(convertedFile);
// loop over input file
string line;
do
{
// get next line from input file
line = srcStream.ReadLine();
if (!Regex.IsMatch(line, #"fred=\d+"))
{
dstStream.WriteLine(line);
dstStream.Flush();
}
} while (line != null);
}
The problem is that when I drop multiple files on the GUI, only one of them actually gets processed. I have found that if I comment out the Regex line, all of the dropped files are processed. Am I missing something in my handling of regular expressions in this context?
Try following variation of the method:
private void convertFile(string filename)
{
// build name of output file
string convertedFile = Path.ChangeExtension(filename, ".out");
// open input file for reading
FileInfo source = new FileInfo(filename);
StreamReader srcStream = source.OpenText();
// open output file for writing
using (StreamWriter dstStream = File.CreateText(convertedFile))
{
// loop over input file
string line;
do
{
// get next line from input file
line = srcStream.ReadLine();
if (!Regex.IsMatch(line, #"fred=\d+"))
{
dstStream.WriteLine(line);
dstStream.Flush();
}
} while (line != null);
}
Debug.WriteLine(string.Format("File written to: {0}", convertedFile));
}
The main modification is use of using keyword which would guarantee disposal and closing of the file resource. If problem is not still resolved then try followings:
Do you have any global exception handlers? Make sure you check Debug > Exceptions... so that Visual Studio automatically breaks on the line where exception is thrown. See this article on how-to.
Make sure files are written at correct places. If files have full path then the Debug.WriteLine statement above would tell you were the files are being written.
You should get at least 0 length file written on the disk if no exceptions are occurring.
Hi all i write a code to write my last row of datagrid view to a file as follows
private void Save_Click(object sender, EventArgs e)
{
if (dataGridView1.Rows.Count > 0)
{
List<string> lstContent = new List<string>();
foreach (DataGridViewRow row in dataGridView1.Rows)
{
if ((string)row.Cells[0].Value == "FileControl")
{
lstContent.Add((string)row.Cells[1].Value);
string mydata = string.Join(",", lstContent.ToArray());
using (StreamWriter sw = new StreamWriter(Append.FileName, true))
{
sw.WriteLine();
sw.Write(mydata);
}
}
}
}
}
But if i click multiple times on save this is writing that line multiple times what i need is if already that line exists in the file i have to replace that line with new line. Any help please
Your StreamWriter is explicitly using the file with append = true. Change the second parameter of the constructor to false if you want to overwrite the file each time. Docs are here. Quote:
append
Type: System.Boolean
Determines
whether data is to be appended to the
file. If the file exists and append is
false, the file is overwritten. If the
file exists and append is true, the
data is appended to the file.
Otherwise, a new file is created.
Revised code:
using (StreamWriter sw = new StreamWriter(Append.FileName, false))
{
sw.WriteLine();
sw.Write(mydata);
}
Replacing a given line in your file rather than just overwriting the whole file is a lot more difficult - this code is not going to get it done. StreamWriter is not great for this, you need random access and the ability to replace one data segment (line) by a different data segment of different length, which is an expensive operation on disk.
You might want to keep the files in memory as a container of Strings and do your required line replacement within the container, then write out the file to disk using File.WriteAllLines - that's if the file is not too big.
The following code gives me a System.IO.IOException with the message 'The process cannot access the file'.
private void UnPackLegacyStats()
{
DirectoryInfo oDirectory;
XmlDocument oStatsXml;
//Get the directory
oDirectory = new DirectoryInfo(msLegacyStatZipsPath);
//Check if the directory exists
if (oDirectory.Exists)
{
//Loop files
foreach (FileInfo oFile in oDirectory.GetFiles())
{
//Check if file is a zip file
if (C1ZipFile.IsZipFile(oFile.FullName))
{
//Open the zip file
using (C1ZipFile oZipFile = new C1ZipFile(oFile.FullName, false))
{
//Check if the zip contains the stats
if (oZipFile.Entries.Contains("Stats.xml"))
{
//Get the stats as a stream
using (Stream oStatsStream = oZipFile.Entries["Stats.xml"].OpenReader())
{
//Load the stats as xml
oStatsXml = new XmlDocument();
oStatsXml.Load(oStatsStream);
//Close the stream
oStatsStream.Close();
}
//Loop hit elements
foreach (XmlElement oHitElement in oStatsXml.SelectNodes("/*/hits"))
{
//Do stuff
}
}
//Close the file
oZipFile.Close();
}
}
//Delete the file
oFile.Delete();
}
}
}
I am struggling to see where the file could still be locked. All objects that could be holding onto a handle to the file are in using blocks and are explicitly closed.
Is it something to do with using FileInfo objects rather than the strings returned by the static GetFiles method?
Any ideas?
I do not see problems in your code, everything look ok. To check is the problem lies in C1ZipFile I suggest you initialize zip from stream, instead of initialization from file, so you close stream explicitly:
//Open the zip file
using (Stream ZipStream = oFile.OpenRead())
using (C1ZipFile oZipFile = new C1ZipFile(ZipStream, false))
{
// ...
Several other suggestions:
You do not need to call Close() method, with using (...), remove them.
Move xml processing (Loop hit elements) outsize zip processing, i.e. after zip file closeing, so you keep file opened as least as possible.
I assume you're getting the error on the oFile.Delete call. I was able to reproduce this error. Interestingly, the error only occurs when the file is not a zip file. Is this the behavior you are seeing?
It appears that the C1ZipFile.IsZipFile call is not releasing the file when it's not a zip file. I was able to avoid this problem by using a FileStream instead of passing the file path as a string (the IsZipFile function accepts either).
So the following modification to your code seems to work:
if (oDirectory.Exists)
{
//Loop files
foreach (FileInfo oFile in oDirectory.GetFiles())
{
using (FileStream oStream = new FileStream(oFile.FullName, FileMode.Open))
{
//Check if file is a zip file
if (C1ZipFile.IsZipFile(oStream))
{
// ...
}
}
//Delete the file
oFile.Delete();
}
}
In response to the original question in the subject: I don't know if it's possible to know if a file can be deleted without attempting to delete it. You could always write a function that attempts to delete the file and catches the error if it can't and then returns a boolean indicating whether the delete was successful.
I'm just guessing: are you sure that oZipFile.Close() is enough? Perhaps you have to call oZipFile.Dispose() or oZipFile.Finalize() to be sure it has actually released the resources.
More then Likely it's not being disposed, anytime you access something outside of managed code(streams, files, etc.) you MUST dispose of them. I learned the hard way with Asp.NET and Image files, it will fill up your memory, crash your server, etc.
In the interest of completeness I am posing my working code as the changes came from more than one source.
private void UnPackLegacyStats()
{
DirectoryInfo oDirectory;
XmlDocument oStatsXml;
//Get the directory
oDirectory = new DirectoryInfo(msLegacyStatZipsPath);
//Check if the directory exists
if (oDirectory.Exists)
{
//Loop files
foreach (FileInfo oFile in oDirectory.GetFiles())
{
//Set empty xml
oStatsXml = null;
//Load file into a stream
using (Stream oFileStream = oFile.OpenRead())
{
//Check if file is a zip file
if (C1ZipFile.IsZipFile(oFileStream))
{
//Open the zip file
using (C1ZipFile oZipFile = new C1ZipFile(oFileStream, false))
{
//Check if the zip contains the stats
if (oZipFile.Entries.Contains("Stats.xml"))
{
//Get the stats as a stream
using (Stream oStatsStream = oZipFile.Entries["Stats.xml"].OpenReader())
{
//Load the stats as xml
oStatsXml = new XmlDocument();
oStatsXml.Load(oStatsStream);
}
}
}
}
}
//Check if we have stats
if (oStatsXml != null)
{
//Process XML here
}
//Delete the file
oFile.Delete();
}
}
}
The main lesson I learned from this is to manage file access in one place in the calling code rather than letting other components manage their own file access. This is most apropriate when you want to use the file again after the other component has finished it's task.
Although this takes a little more code you can clearly see where the stream is disposed (at the end of the using), compared to having to trust that a component has correctly disposed of the stream.