I have a text file. Several processes can simultaneously try to read and edit this file. I have a problem with FileStream.Unlock() method:
using System;
using System.IO;
using System.Text;
static class Program
{
static void Main()
{
var fileName = #"c:\temp\data.txt";
// Content of the 'c:\temp\data.txt' file:
// Hello!
// The magic number is 000. :)))
// Good luck...
using (var stream = new FileStream(fileName, FileMode.Open,
FileAccess.ReadWrite, FileShare.ReadWrite))
{
using(var reader = new StreamReader(stream))
{
var value = 0;
Console.Write("New value [0-999]: ");
while(int.TryParse(Console.ReadLine(), out value))
{
var prevPosition = stream.Position;
stream.Position = 28;
var data = Encoding.UTF8.GetBytes(value.ToString());
try
{
stream.Lock(stream.Position, data.LongLength);
Console.WriteLine("Data locked. Press any key for continuation...");
Console.ReadKey();
stream.Write(data, 0, data.Length);
stream.Flush();
// I get the Exception here: The segment already unlocked.
stream.Unlock(stream.Position, data.LongLength);
}
catch(Exception ex)
{
Console.WriteLine("Error: {0}", ex.Message);
}
stream.Position = prevPosition;
Console.Write("New value: ");
}
}
}
}
}
Why my stream is unlocked before I do it myself?
The reason is stream.Position is advanced after you locked the file (because you write to it), and you use stream.Position (which is now different) to unlock a file. In result - you are trying to unlock not the same range you locked. Instead, save stream.Position:
var position = stream.Position; // < save
stream.Lock(position, data.LongLength);
Console.WriteLine("Data locked. Press any key for continuation...");
stream.Write(data, 0, data.Length); // < this changes stream.Position, breaking your old logic
stream.Flush();
// I get the Exception here:
// The blocking of the segment already taken off.
stream.Unlock(position, data.LongLength); // < now you unlock the same range
Not sure, but maybe when you Write, the Stream.Position changes.
Related
I want to read file continuously like GNU tail with "-f" param. I need it to live-read log file.
What is the right way to do it?
More natural approach of using FileSystemWatcher:
var wh = new AutoResetEvent(false);
var fsw = new FileSystemWatcher(".");
fsw.Filter = "file-to-read";
fsw.EnableRaisingEvents = true;
fsw.Changed += (s,e) => wh.Set();
var fs = new FileStream("file-to-read", FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
using (var sr = new StreamReader(fs))
{
var s = "";
while (true)
{
s = sr.ReadLine();
if (s != null)
Console.WriteLine(s);
else
wh.WaitOne(1000);
}
}
wh.Close();
Here the main reading cycle stops to wait for incoming data and FileSystemWatcher is used just to awake the main reading cycle.
You want to open a FileStream in binary mode. Periodically, seek to the end of the file minus 1024 bytes (or whatever), then read to the end and output. That's how tail -f works.
Answers to your questions:
Binary because it's difficult to randomly access the file if you're reading it as text. You have to do the binary-to-text conversion yourself, but it's not difficult. (See below)
1024 bytes because it's a nice convenient number, and should handle 10 or 15 lines of text. Usually.
Here's an example of opening the file, reading the last 1024 bytes, and converting it to text:
static void ReadTail(string filename)
{
using (FileStream fs = File.Open(filename, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
// Seek 1024 bytes from the end of the file
fs.Seek(-1024, SeekOrigin.End);
// read 1024 bytes
byte[] bytes = new byte[1024];
fs.Read(bytes, 0, 1024);
// Convert bytes to string
string s = Encoding.Default.GetString(bytes);
// or string s = Encoding.UTF8.GetString(bytes);
// and output to console
Console.WriteLine(s);
}
}
Note that you must open with FileShare.ReadWrite, since you're trying to read a file that's currently open for writing by another process.
Also note that I used Encoding.Default, which in US/English and for most Western European languages will be an 8-bit character encoding. If the file is written in some other encoding (like UTF-8 or other Unicode encoding), It's possible that the bytes won't convert correctly to characters. You'll have to handle that by determining the encoding if you think this will be a problem. Search Stack overflow for info about determining a file's text encoding.
If you want to do this periodically (every 15 seconds, for example), you can set up a timer that calls the ReadTail method as often as you want. You could optimize things a bit by opening the file only once at the start of the program. That's up to you.
To continuously monitor the tail of the file, you just need to remember the length of the file before.
public static void MonitorTailOfFile(string filePath)
{
var initialFileSize = new FileInfo(filePath).Length;
var lastReadLength = initialFileSize - 1024;
if (lastReadLength < 0) lastReadLength = 0;
while (true)
{
try
{
var fileSize = new FileInfo(filePath).Length;
if (fileSize > lastReadLength)
{
using (var fs = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
fs.Seek(lastReadLength, SeekOrigin.Begin);
var buffer = new byte[1024];
while (true)
{
var bytesRead = fs.Read(buffer, 0, buffer.Length);
lastReadLength += bytesRead;
if (bytesRead == 0)
break;
var text = ASCIIEncoding.ASCII.GetString(buffer, 0, bytesRead);
Console.Write(text);
}
}
}
}
catch { }
Thread.Sleep(1000);
}
}
I had to use ASCIIEncoding, because this code isn't smart enough to cater for variable character lengths of UTF8 on buffer boundaries.
Note: You can change the Thread.Sleep part to be different timings, and you can also link it with a filewatcher and blocking pattern - Monitor.Enter/Wait/Pulse. For me the timer is enough, and at most it only checks the file length every second, if the file hasn't changed.
This is my solution
static IEnumerable<string> TailFrom(string file)
{
using (var reader = File.OpenText(file))
{
while (true)
{
string line = reader.ReadLine();
if (reader.BaseStream.Length < reader.BaseStream.Position)
reader.BaseStream.Seek(0, SeekOrigin.Begin);
if (line != null) yield return line;
else Thread.Sleep(500);
}
}
}
so, in your code you can do
foreach (string line in TailFrom(file))
{
Console.WriteLine($"line read= {line}");
}
You could use the FileSystemWatcher class which can send notifications for different events happening on the file system like file changed.
private void button1_Click(object sender, EventArgs e)
{
if (folderBrowserDialog.ShowDialog() == DialogResult.OK)
{
path = folderBrowserDialog.SelectedPath;
fileSystemWatcher.Path = path;
string[] str = Directory.GetFiles(path);
string line;
fs = new FileStream(str[0], FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
tr = new StreamReader(fs);
while ((line = tr.ReadLine()) != null)
{
listBox.Items.Add(line);
}
}
}
private void fileSystemWatcher_Changed(object sender, FileSystemEventArgs e)
{
string line;
line = tr.ReadLine();
listBox.Items.Add(line);
}
If you are just looking for a tool to do this then check out free version of Bare tail
I have an active audio recording happening in WAV format with NAudio Library.
private void RecordStart() {
try {
_sourceStream = new WaveIn {
DeviceNumber = _recordingInstance.InputDeviceIndex,
WaveFormat =
new WaveFormat(
44100,
WaveIn.GetCapabilities(_recordingInstance.InputDeviceIndex).Channels)
};
_sourceStream.DataAvailable += SourceStreamDataAvailable;
if (!Directory.Exists(_recordingInstance.AudioFilePath)) {
Directory.CreateDirectory(_recordingInstance.AudioFilePath);
}
WaveFileWriter _waveWriter = new WaveFileWriter(
_recordingInstance.AudioFilePath + _recordingInstance.AudioFileName,
_sourceStream.WaveFormat);
_sourceStream.StartRecording();
}
catch (Exception exception) {
Log.Error("Recording failes", exception);
}
}
private void SourceStreamDataAvailable(object sender, WaveInEventArgs e) {
if (_waveWriter == null) return;
_waveWriter.Write(e.Buffer, 0, e.BytesRecorded);
_waveWriter.Flush();
}
I want to copy the latest available content to another location. The copied file should be in WAV format, and should be able to play the available duration. Update the destination file, whenever more content is available.
I have Tried the following sample code (using NAudio) with a static WAV file, but the solution is not working.
The resulting WAV file created is corrupted - not in the correct format.
using (WaveFileReader reader = new WaveFileReader(remoteWavFile))
{
byte[] buffer = new byte[reader.Length];
int read = reader.Read(buffer, 0, buffer.Length);
}
When the recording is in progress, the code throws an exception "File is in use by another application".
I have solved the problem with help of NAudio Library itself.
When we only use the WaveFileReader class of NAudio. It will throw the exception - "file is in use by another application".
So I had to create a file stream, which opens the source file - live recording file, with File.Open(inPath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite) then pass this stream as an input of WaveFileReader.
Then create a WaveFileWritter class of NAudio, with the same WavFormat of the reader.
copied below is the code, i have used.
public static void CopyWavFile(string inPath, string outPath){
using (var fs = File.Open(inPath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite)){
using (var reader = new WaveFileReader(fs)){
using (var writer = new WaveFileWriter(outPath, reader.WaveFormat)){
reader.Position = 0;
var endPos = (int)reader.Length;
var buffer = new byte[1024];
while (reader.Position < endPos){
var bytesRequired = (int)(endPos - reader.Position);
if (bytesRequired <= 0) continue;
var bytesToRead = Math.Min(bytesRequired, buffer.Length);
var bytesRead = reader.Read(buffer, 0, bytesToRead);
if (bytesRead > 0){
writer.Write(buffer, 0, bytesRead);
}
}
}
}
}
}
I am currently trying to construct file B by extracting a certain length of bytes from file A (chunk-by-chunk). The size of file B is 38052441 bytes, and its location in file A is from byte 34 onward. If I do it in one shot, I manage to extract file B from file A without any issue, as shown in the snippet below.
test = new byte[38052441];
//madefilePath: file A, madecabfilePath: file B
using (BinaryReader reader = new BinaryReader(new FileStream(madefilePath, FileMode.Open)))
using (BinaryWriter bw = new BinaryWriter(File.Open(madecabfilePath, FileMode.OpenOrCreate)))
{
reader.BaseStream.Seek(34, SeekOrigin.Begin);
reader.Read(test, 0, 38052441);
bw.Write(test);
bw.Close();
reader.Close();
}
Howerver, if I try to do it in multiple query (I have to do this, because this feature will be ported to compact framework in the future), I kept on getting a corrupted file. Currently, I am testing by getting the first 20Mb, write into a file, then get the remaining bytes and write it into the file again.
int max = 38052474;
int offset = 34;
int weight = 20000000;
bool isComplete = false;
test = null;
test = new byte[weight];
using (BinaryWriter bw = new BinaryWriter(File.Open(madecabfilePath, FileMode.OpenOrCreate)))
using (BinaryReader reader = new BinaryReader(new FileStream(madefilePath, FileMode.Open)))
{
while (!isComplete)
{
if (offset + weight < max)
{
reader.BaseStream.Seek(offset, SeekOrigin.Begin);
reader.Read(test, 0, weight);
bw.Write(test);
offset = offset + weight;
}
else
{
weight = max - offset;
test = null;
test = new byte[weight];
reader.BaseStream.Seek(offset, SeekOrigin.Begin);
reader.Read(test, 0, weight);
bw.Write(test);
//Terminate everything
reader.Close();
bw.Close();
isComplete = true;
}
}
}
I think the issue lies with my logic, but I can't figure out why. Any help is appreciated. Thank you.
BinaryReader.Read() returns the number of bytes that were actually read. So you can simplify your logic and probably fix some issues with something like:
using (BinaryWriter bw = new BinaryWriter(File.Open(madecabfilePath, FileMode.OpenOrCreate)))
using (BinaryReader reader = new BinaryReader(new FileStream(madefilePath, FileMode.Open)))
{
reader.BaseStream.Seek(offset, SeekOrigin.Begin);
while (!isComplete)
{
int charsRead = reader.Read(test, 0, weight);
if (charsRead == 0)
{
isComplete = true;
}
else
{
bw.Write(test, 0, charsRead);
}
}
}
Note that you don't need to explicitly close bw or reader, as the using statement will do that for you. Also note that after the first Seek() call the position in the BinaryReader will be kept track of.
I want to save the incoming stream data to a WAV file on my hard disk drive. How can I change the code below to be able to record the stream into a valid WAV file?
From the demo here:
private void StreamMP3(object state)
{
this.fullyDownloaded = false;
string url = (string)state;
webRequest = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse resp = null;
try
{
resp = (HttpWebResponse)webRequest.GetResponse();
}
catch(WebException e)
{
if (e.Status != WebExceptionStatus.RequestCanceled)
{
ShowError(e.Message);
}
return;
}
byte[] buffer = new byte[16384 * 4]; // Needs to be big enough to hold a decompressed frame
IMp3FrameDecompressor decompressor = null;
try
{
using (var responseStream = resp.GetResponseStream())
{
var readFullyStream = new ReadFullyStream(responseStream);
do
{
if (bufferedWaveProvider != null &&
bufferedWaveProvider.BufferLength - bufferedWaveProvider.BufferedBytes <
bufferedWaveProvider.WaveFormat.AverageBytesPerSecond / 4)
{
Debug.WriteLine("Buffer getting full, taking a break");
Thread.Sleep(500);
}
else
{
Mp3Frame frame = null;
try
{
frame = Mp3Frame.LoadFromStream(readFullyStream);
}
catch (EndOfStreamException)
{
this.fullyDownloaded = true;
// Reached the end of the MP3 file / stream
break;
}
catch (WebException)
{
// Probably we have aborted download from the GUI thread
break;
}
if (decompressor == null)
{
// I don't think these details matter too much - just help ACM select the right codec.
// However, the buffered provider doesn't know what sample rate it is working at
// until we have a frame.
WaveFormat waveFormat = new Mp3WaveFormat(
frame.SampleRate,
frame.ChannelMode == ChannelMode.Mono ? 1 : 2,
frame.FrameLength,
frame.BitRate);
decompressor = new AcmMp3FrameDecompressor(waveFormat);
this.bufferedWaveProvider = new BufferedWaveProvider(decompressor.OutputFormat);
this.bufferedWaveProvider.BufferDuration = TimeSpan.FromSeconds(20); // Allow us to get well ahead of ourselves
//this.bufferedWaveProvider.BufferedDuration = 250;
}
int decompressed = decompressor.DecompressFrame(frame, buffer, 0);
//Debug.WriteLine(String.Format("Decompressed a frame {0}", decompressed));
bufferedWaveProvider.AddSamples(buffer, 0, decompressed);
}
} while (playbackState != StreamingPlaybackState.Stopped);
Debug.WriteLine("Exiting");
// I was doing this in a finally block, but for some reason
// we are hanging on response stream .Dispose, so we never get there.
decompressor.Dispose();
}
}
finally
{
if (decompressor != null)
{
decompressor.Dispose();
}
}
}
I wouldn't take that particular approach to saving to disk. It's a bit too hands-on, because it has to deal with playing back at the right rate. Just buffer up the response, and then wrap it in an Mp3FileReader stream and use WaveFileWriter to write the WAV file:
MemoryStream mp3Buffered = new MemoryStream();
using (var responseStream = resp.GetResponseStream())
{
byte[] buffer = new byte[65536];
int bytesRead = responseStream.Read(buffer, 0, buffer.Length);
while (bytesRead > 0)
{
mp3Buffered.Write(buffer, 0, bytesRead);
bytesRead = responseStream.Read(buffer, 0, buffer.Length);
}
}
mp3Buffered.Position = 0;
using (var mp3Stream = new Mp3FileReader(mp3Buffered))
{
WaveFileWriter.CreateWaveFile("file.wav", mp3Stream);
}
That does, of course, assume that your MP3 file's wave format is compatible with WAV and in particular, your WAV player. If it isn't, you'll need to inject and add a WaveFormatConversion stream as well.
You can use following line to save to MemoryStream :
mp3Buffered.Write(frame.RawData, 0, frame.RawData.Length);
Saving stream to file is described in MattW's answer.
As part of an upcoming project at my university, I need to write a client that downloads a media file from a server and writes it to the local disk. Since these files can be very large, I need to implement partial download and serialization in order to avoid excessive memory use.
What I came up with:
namespace PartialDownloadTester
{
using System;
using System.Diagnostics.Contracts;
using System.IO;
using System.Net;
using System.Text;
public class DownloadClient
{
public static void Main(string[] args)
{
var dlc = new DownloadClient(args[0], args[1], args[2]);
dlc.DownloadAndSaveToDisk();
Console.ReadLine();
}
private WebRequest request;
// directory of file
private string dir;
// full file identifier
private string filePath;
public DownloadClient(string uri, string fileName, string fileType)
{
this.request = WebRequest.Create(uri);
this.request.Method = "GET";
var sb = new StringBuilder();
sb.Append("C:\\testdata\\DownloadedData\\");
this.dir = sb.ToString();
sb.Append(fileName + "." + fileType);
this.filePath = sb.ToString();
}
public void DownloadAndSaveToDisk()
{
// make sure directory exists
this.CreateDir();
var response = (HttpWebResponse)request.GetResponse();
Console.WriteLine("Content length: " + response.ContentLength);
var rStream = response.GetResponseStream();
int bytesRead = -1;
do
{
var buf = new byte[2048];
bytesRead = rStream.Read(buf, 0, buf.Length);
rStream.Flush();
this.SerializeFileChunk(buf);
}
while (bytesRead != 0);
}
private void CreateDir()
{
if (!Directory.Exists(dir))
{
Directory.CreateDirectory(dir);
}
}
private void SerializeFileChunk(byte[] bytes)
{
Contract.Requires(!Object.ReferenceEquals(bytes, null));
FileStream fs = File.Open(filePath, FileMode.Append);
fs.Write(bytes, 0, bytes.Length);
fs.Flush();
fs.Close();
}
}
}
For testing purposes, I've used the following parameters:
"http://itu.dk/people/janv/mufc_abc.jpg" "mufc_abc" "jpg"
However, the picture is incomplete (only the first ~10% look right) even though the content length prints 63780 which is the actual size of the image.
So my questions are:
Is this the right way to go for partial download and serialization or is there a better/easier approach?
Is the full content of the response stream stored in client memory? If this is the case, do I need to use HttpWebRequest.AddRange to partially download data from the server in order to conserve my client's memory?
How come the serialization fails and I get a broken image?
Do I introduce a lot of overhead when I use the FileMode.Append? (msdn states that this option "seeks to the end of the file")
Thanks in advance
You could definitely simplify your code using a WebClient:
class Program
{
static void Main()
{
DownloadClient("http://itu.dk/people/janv/mufc_abc.jpg", "mufc_abc.jpg");
}
public static void DownloadClient(string uri, string fileName)
{
using (var client = new WebClient())
{
using (var stream = client.OpenRead(uri))
{
// work with chunks of 2KB => adjust if necessary
const int chunkSize = 2048;
var buffer = new byte[chunkSize];
using (var output = File.OpenWrite(fileName))
{
int bytesRead;
while ((bytesRead = stream.Read(buffer, 0, buffer.Length)) > 0)
{
output.Write(buffer, 0, bytesRead);
}
}
}
}
}
}
Notice how I am writing only the number of bytes I have actually read from the socket to the output file and not the entire 2KB buffer.
I don't know if this is the source of the problem, however I would change the loop like this
const int ChunkSize = 2048;
var buf = new byte[ChunkSize];
var rStream = response.GetResponseStream();
do {
int bytesRead = rStream.Read(buf, 0, ChunkSize);
if (bytesRead > 0) {
this.SerializeFileChunk(buf, bytesRead);
}
} while (bytesRead == ChunkSize);
The serialize method would get an additional argument
private void SerializeFileChunk(byte[] bytes, int numBytes)
and then write the right number of bytes
fs.Write(bytes, 0, numBytes);
UPDATE:
I do not see the need for closing and reopening the file each time. I also would use the using statement, which closes the resources, even if an exception should occur. The using statement calls the Dispose() method of the resource at the end, which in turn calls Close() in the case of file streams. using can be applied to all types implementing IDisposable.
var buf = new byte[2048];
using (var rStream = response.GetResponseStream()) {
using (FileStream fs = File.Open(filePath, FileMode.Append)) {
do {
bytesRead = rStream.Read(buf, 0, buf.Length);
fs.Write(bytes, 0, bytesRead);
} while (...);
}
}
The using statement does something like this
{
var rStream = response.GetResponseStream();
try
{
// do some work with rStream here.
} finally {
if (rStream != null) {
rStream.Dispose();
}
}
}
Here is the solution from Microsoft: http://support.microsoft.com/kb/812406
Updated 2021-03-16: seems the original article is not available now. Here is the archived one: https://mskb.pkisolutions.com/kb/812406