I'm using StreamReader, but if try to read from the same stream using two StreamReader-objects I get an error saying I can't read from dispose object (reader3.ReadLine).
Since I'm not disposing any object, what am I doing wrong?
Stream responseStream2;
FtpWebResponse ftpResponse2;
string casefile = CNCElement.ID_CASE_TEST_FILE;
string casepath;
if (FileManager.PathCombine(result, lock_root_folder, casefile, out casepath) == false)
return false;
if (fm.DownloadFtp(result, casepath, out responseStream2, out ftpResponse2) == false)
return false;
StreamReader reader2 = new StreamReader(responseStream2);
StreamReader reader3 = new StreamReader(responseStream2);
byte[] contents=null;
//if cycle is not present update case file
//if cycle is present, case file is already correct
if (reader2.ReadToEnd().Contains(cycle) == false)
{
byte seekcase = CNCElement.ID_CASE.Value;
int casecount = 1;
string line;
using (MemoryStream ms = new MemoryStream())
{
while ((line = reader3.ReadLine()) != null
|| casecount <= seekcase)
{
if (line.Contains("\"\"") == true)
{
if (casecount == seekcase)
line = line.Replace("\"\"", "\"" + cycle + "\"");
}
byte[] app = StrToByteArray(line);
ms.Write(app, 0, line.Length);
contents = ms.ToArray();
}
}
}
if (reader2 != null)
reader2.Close();
if (ftpResponse2 != null)
ftpResponse2.Close();
When you read to the end of reader2 you are really reading to the end of the underlying stream (responseStream2). At this point another read from that stream will fail.
While the specific exception is slightly expected, wrapping the same stream in different StreamReaders is going to do weird things because it is a weird thing to do.
If you need to read a stream twice you need to either use a stream that supports resetting its position to the beginning (ie. random access) and then create a new reader for the second read; or (as would seem likely in this case: I doubt any network stream will support random access) buffer the stream content yourself.
When you call ReadToEnd() the underlying steam is all read into memory and you have reached the end.
Each time you call the function ReadLine() then underlying stream is moved to the next line.
This means that when your application reaches Reader3.ReadLine() loop, as you have already reached the end of the file, the reader fails.
If the expected file stream is not too large, I would suggest you assign the result of the ReadToEnd() call to a variable and perform subsequent operations on this variable.
If the stream is large, then try resetting the Position property (provided it is supported - See the docs).
Related
Background: I need to relay the content of the request to multiple other servers (via client.SendAsync(request)).
Problem: After first request the content stream is empty
[HttpPost]
public async Task<IActionResult> PostAsync() {
for (var n = 0; n <= 1; n++) {
using (var stream = new MemoryStream()) {
await Request.Body.CopyToAsync(stream);
// why is stream.length == 0 in the second iteration?
}
}
return StatusCode((int)HttpStatusCode.OK);
}
Streams have a pointer indicating at which position the stream is; after copying it, the pointer is at the end. You need to rewind a stream by setting its position to 0.
This is however only supported in streams that support seeking. You can read the request stream only once. This is because it's read "from the wire", and therefore doesn't support seeking.
When you want to copy the request stream to multiple output streams, you have two options:
Forward while you read
Read once into memory, then forward at will
The first option means all forwards happen at the same speed; the entire transfer goes as slow as the input, or as slow as the slowest reader. You read a chunk from the caller, and forward that chunk to all forward addresses.
For the second approach, you'll want to evaluate whether you can hold the entire request body plus the body for each forward address in memory. If that's not expected to be a problem and properly configured with sensible limits, then simply copy the request stream to a single MemoryStream and copy and rewind that one after every call:
using (var bodyStream = new MemoryStream())
{
await Request.Body.CopyToAsync(bodyStream);
for (...)
{
using (var stream = new MemoryStream())
{
await bodyStream.CopyToAsync(stream);
// Rewind for next copy
bodyStream.Position = 0;
}
}
}
I found out that the CopyToAsync function sets the origin stream position to the last read position. The next time I use CopyToAsync the stream starts reading from the last read position and does not find more content. However I could not use Request.Body.Position = 0 since it is not supported. I ended up copying the stream once more and reset the position after each copy.
If someone knows a cleaner solution you are welcome to point it out.
using (var contentStream = new MemoryStream()) {
await Request.Body.CopyToAsync(contentStream);
for (var n = 0; n <= 1; n++) {
using (var stream = new MemoryStream()) {
contentStream.Position = 0;
await contentStream.CopyToAsync(stream);
// works
}
}
}
I'm trying to process part of a text file, and write the remainder of the text file to a cloud blob using UploadFromStream. The problem is that the StreamReader appears to be grabbing too much content from the underlying stream, and so the subsequent write does nothing.
Text file:
3
Col1,String
Col2,Integer
Col3,Boolean
abc,123,True
def,3456,False
ghijkl,532,True
mnop,1211,False
Code:
using (var stream = File.OpenRead("c:\\test\\testinput.txt"))
using (var reader = new StreamReader(stream))
{
var numColumns = int.Parse(reader.ReadLine());
while (numColumns-- > 0)
{
var colDescription = reader.ReadLine();
// do stuff
}
// Write remaining contents to another file, for testing
using (var destination = File.OpenWrite("c:\\test\\testoutput.txt"))
{
stream.CopyTo(destination);
destination.Flush();
}
// Actual intended usage:
// CloudBlockBlob blob = ...;
// blob.UploadFromStream(stream);
}
When debugging, I observe that stream.Position jumps to the end of the file on the first call to reader.ReadLine(), which I don't expect. I expected the stream to be advanced only as many positions as the reader needed to read some content.
I imagine that the stream reader is doing some buffering for performance reasons, but there doesn't seem to be a way to ask the reader where in the underlying stream it "really" is. (If there was, I could manually Seek the stream to that position before CopyingTo).
I know that I could keep taking lines using the same reader and sequentially append them to the text file I'm writing, but I'm wondering if there's a cleaner way?
EDIT:
I found a StreamReader constructor which leaves the underlying stream open when it is disposed, so I tried this, hoping that the reader would set the stream's position as it's being disposed:
using (var stream = File.OpenRead("c:\\test\\testinput.txt"))
{
using (var reader = new StreamReader(stream, Encoding.UTF8,
detectEncodingFromByteOrderMarks: true,
bufferSize: 1 << 12,
leaveOpen: true))
{
var numColumns = int.Parse(reader.ReadLine());
while (numColumns-- > 0)
{
var colDescription = reader.ReadLine();
// do stuff
}
}
// Write remaining contents to another file
using (var destination = File.OpenWrite("c:\\test\\testoutput.txt"))
{
stream.CopyTo(destination);
destination.Flush();
}
}
But it doesn't. Why would this constructor be exposed if it doesn't leave the stream in an intuitive state/position?
Sure, there's a cleaner way. Use ReadToEnd to read the remaining data, and then write it to a new file. For example:
using (var reader = new StreamReader("c:\\test\\testinput.txt"))
{
var numColumns = int.Parse(reader.ReadLine());
while (numColumns-- > 0)
{
var colDescription = reader.ReadLine();
// do stuff
}
// write everything else to another file.
File.WriteAllText("c:\\test\\testoutput.txt", reader.ReadToEnd());
}
Edit after comment
If you want to read the text and upload it to a stream, you could replace the File.WriteAllText with code that reads the remaining text, writes it to a StreamWriter backed by a MemoryStream, and then sends the contents of that MemoryStream. Something like:
using (var memStream = new MemoryStream())
{
using (var writer = new StreamWriter(memStream))
{
writer.Write(reader.ReadToEnd());
writer.Flush();
memStream.Position = 0;
blob.UploadFromStream(memStream);
}
}
You should never access the underlying stream of a StreamReader. Trying to use both is going to have an undefined behavior.
What's going on here is that the reader is buffering the data from the underlying stream. It doesn't read each byte exactly when you request it, because that's often going to be very inefficient. Instead it will grab chunks, put them in a buffer, and then provide you with data from that buffer, grabbing a new chunk when it needs to.
You should continue to use the StreamReader throughout the remainder of that block, instead of using stream. To minimize the memory footprint of the program, the most effective way of doing this would be to read the next line from the reader in a loop until it his the end of the file, writing each line to the output stream as you go.
Also note that you don't need to be disposing of both the stream reader and the underlying stream. The stream reader will dispose of the underlying stream itself, so you can simply adjust your header to:
using (var reader = new StreamReader(
File.OpenRead("c:\\test\\testinput.txt")))
In an utility method, which accepts a Stream parameter, I rely on some StreamReader to analyse data.
I don't want to close the incoming stream in my method. I want to let the caller method to take the decision to dispose the stream.
Is it safe to not dispose the opened StreamReader? I mean, will it eventually be automatically disposed? Will it lead to memory leaks?
Here is my utility method. Its goal is to read a Stream, and return its content as a string, regardless of how the data is encoded:
public static string GetStringAutoDetectEncoding(Stream data, out Encoding actualEncoding)
{
// 1. Is there a Bye Order Mask ?
var candidateEncoding = DetectEncodingWithByteOrderMask(data);
// 2a. No BOM, the data is either UTF8 no BOM or ANSI
if (candidateEncoding == Encoding.Default)
{
var utf8NoBomEncoding = Encoding.GetEncoding("utf-8",new EncoderExceptionFallback(), new DecoderExceptionFallback());
var positionBackup = data.Position;
var sr = new StreamReader(data, utf8NoBomEncoding);
try
{
// 3. Try as UTF8 With no BOM
var result = sr.ReadToEnd(); // will throw error if not UTF8
actualEncoding = utf8NoBomEncoding; // Probably an UTF8 no bom string
return result;
}
catch (DecoderFallbackException)
{
// 4. Rewind the stream and fallback to ASNI
data.Position = positionBackup;
var srFallback = new StreamReader(data, candidateEncoding);
actualEncoding = candidateEncoding;
return srFallback.ReadToEnd(); ;
}
}
// 2b. There is a BOM. Use the detected encoding
else
{
var sr = new StreamReader(data, candidateEncoding);
actualEncoding = candidateEncoding;
return sr.ReadToEnd(); ;
}
}
Then, I can have some methods in the like this:
void Foo(){
using(var stream = File.OpenRead(#"c:\somefile")) {
Encoding detected;
var fileContent = MyUtilityClass.GetStringAutoDetectEncoding(stream, detected);
Console.WriteLine("Detected encoding: {0}", encoding);
Console.WriteLine("File content: {0}", fileContent);
}
}
You could invert control using a closure. That is, create a method like so:
// This method will open the stream, execute the streamClosure, and then close the stream.
public static String StreamWork(Func<Stream, String> streamClosure) {
// Set up the stream here.
using (Stream stream = new MemoryStream()) { // Pretend the MemoryStream is your actual stream.
// Execute the closure. Return it's results.
return streamClosure(stream);
}
}
which is responsible for opening / closing the stream within the method.
Then you simply wrap up all the code that needs the stream into a Func<Stream, String> closure, and pass it in. The StreamWork method will open the stream, execute your code, then close the stream.
public static void Main()
{
// Wrap all of the work that needs to be done in a closure.
// This represents all the work that needs to be done while the stream is open.
Func<Stream, String> streamClosure = delegate(Stream stream) {
using (StreamReader streamReader = new StreamReader(stream)) {
return streamReader.ReadToEnd();
}
};
// Call StreamWork. This method handles creating/closing the stream.
String result = StreamWork(streamClosure);
Console.WriteLine(result);
Console.ReadLine();
}
UPDATE
Of course, this method of inversion is a matter of preference as mentioned in the comments below. The key point is to ensure that the stream is closed rather than allowing it to float around until the GC cleans it up (since the whole point of having stuff implement IDisposable is to avoid that sort of situation to begin with). Since this is a library function that accepts a Stream as input, the assumption is that the method-consumer will be creating the stream, and therefore as you point out, has the responsibility of ultimately closing the stream as well. But for sensitive resources where you are concerned about ensuring clean up occurs absolutely, inversion is sometimes a useful technique.
StreamReader close/dispose their underlying streams only when you call Dispose on them. They don't dispose of the stream if the reader/writer is just garbage collected.
I am trying to write an object to an Xml string and take that string and save it to a DB. But first I need to get the string...
private static readonly Encoding LocalEncoding = Encoding.UTF8;
public static string SaveToString<T> (T settings)
{
Stream stream = null;
TextWriter writer = null;
string settingsString = null;
try
{
stream = new MemoryStream();
var serializer = new XmlSerializer(typeof(T));
writer = new StreamWriter(stream, LocalEncoding);
serializer.Serialize(writer, settings);
var buffer = new byte[stream.Length];
stream.Read(buffer, 0, (int)stream.Length);
settingsString = LocalEncoding.GetString(buffer);
}
catch (Exception ex)
{
// If the action cancels we don't want to throw, just return null.
}
finally
{
if (stream != null)
stream.Close();
if (writer != null)
writer.Close();
}
return settingsString;
}
This seems to work, the stream gets filled with bytes. But when I come to read it back into the buffer and then into the string... the buffer is filled with '0'! Not sure what I doing wrong here guys.
If you'd checked the results of stream.Read, you'd have seen that it hadn't read anything - because you haven't rewound the stream. (You could do this with stream.Position = 0;.) However, it's easier to just call ToArray:
settingsString = LocalEncoding.GetString(stream.ToArray());
(You'll need to change the type of stream from Stream to MemoryStream, but that's okay as it's in the same method where you create it.)
Alternatively - and even more simply - just use StringWriter instead of StreamWriter. You'll need to create a subclass if you want to use UTF-8 instead of UTF-16, but that's pretty easy. See this answer for an example.
I'm concerned by the way you're just catching Exception and assuming that it means something harmless, by the way - without even logging anything. Note that using statements are generally cleaner than writing explicit finally blocks.
string result = System.Text.Encoding.UTF8.GetString(fs.ToArray());
string result = Encoding.UTF8.GetString((stream as MemoryStream).ToArray());
In case of a very large stream length there is the hazard of memory leak due to Large Object Heap. i.e. The byte buffer created by stream.ToArray creates a copy of memory stream in Heap memory leading to duplication of reserved memory. I would suggest to use a StreamReader, a TextWriter and read the stream in chunks of char buffers.
In netstandard2.0 System.IO.StreamReader has a method ReadBlock
you can use this method in order to read the instance of a Stream (a MemoryStream instance as well since Stream is the super of MemoryStream):
private static string ReadStreamInChunks(Stream stream, int chunkLength)
{
stream.Seek(0, SeekOrigin.Begin);
string result;
using(var textWriter = new StringWriter())
using (var reader = new StreamReader(stream))
{
var readChunk = new char[chunkLength];
int readChunkLength;
//do while: is useful for the last iteration in case readChunkLength < chunkLength
do
{
readChunkLength = reader.ReadBlock(readChunk, 0, chunkLength);
textWriter.Write(readChunk,0,readChunkLength);
} while (readChunkLength > 0);
result = textWriter.ToString();
}
return result;
}
NB. The hazard of memory leak is not fully eradicated, due to the usage of MemoryStream, that can lead to memory leak for large memory stream instance (memoryStreamInstance.Size >85000 bytes). You can use Recyclable Memory stream, in order to avoid LOH. This is the relevant library
I have a textReader that in a specific instance I want to be able to advance to the end of file quickly so other classes that might hold a reference to this object will not be able to call tr.ReadLine() without getting a null.
This is a large file. I cannot use TextReader.ReadToEnd() as it will often lead to an OutOfMemoryException
I thought I would ask the community if there was a way SEEK the stream without using TextReader.ReadToEnd() which returns a string of all data in the file.
Current method, inefficient.
The following example code is a mock up. Obviously I am not opening a file with an if statement directly following it asking if I want to read to the end.
TextReader tr = new StreamReader("Largefile");
if(needToAdvanceToEndOfFile)
{
while(tr.ReadLine() != null) { }
}
Desired solution (Note this code block contains fake 'concept' methods or methods that cannot be used due to risk of outofmemoryexception)
TextReader tr = new StreamReader("Largefile");
if(needToAdvanceToEndOfFile)
{
tr.SeekToEnd(); // A method that does not return anything. This method does not exist.
// tr.ReadToEnd() not acceptable as it can lead to OutOfMemoryException error as it is very large file.
}
A possible alternative is to read through the file in bigger chunks using tr.ReadBlock(args).
I poked around ((StreamReader)tr).BaseStream but could not find anything that worked.
As I am new to the community I figured I would see if someone knew the answer off the top of their head.
You have to discard any buffered data if you have read any file content - since data is buffered you might get content even if you seek the underlying stream to the end - working example:
StreamReader sr = new StreamReader(fileName);
string sampleLine = sr.ReadLine();
//discard all buffered data and seek to end
sr.DiscardBufferedData();
sr.BaseStream.Seek(0, SeekOrigin.End);
The problem as mentioned in the documentation is
The StreamReader class buffers input from the underlying stream when
you call one of the Read methods. If you manipulate the position of
the underlying stream after reading data into the buffer, the position
of the underlying stream might not match the position of the internal
buffer. To reset the internal buffer, call the DiscardBufferedData
method
Use
reader.BaseStream.Seek(0, SeekOrigin.End);
Test:
using (StreamReader reader = new StreamReader(#"Your Large File"))
{
reader.BaseStream.Seek(0, SeekOrigin.End);
int read = reader.Read();//read will be -1 since you are at the end of the stream
}
Edit: Test it with your code:
using (TextReader tr = new StreamReader("C:\\test.txt"))//test.txt is a file that has data and lines
{
((StreamReader)tr).BaseStream.Seek(0, SeekOrigin.End);
string foo = tr.ReadLine();
Debug.WriteLine(foo ?? "foo is null");//foo is null
int read = tr.Read();
Debug.WriteLine(read);//-1
}