I have a code written in MFC for file reading and writing. I am rewriting it for C#.
My file consists of three parts, Header *Body* and Footer. Now in MFC code CArchive can write/read any of these parts. This is done by COleStreamFile::OpenStream. In this method we gave which part to read and this returns stream pointing to that location in the file. CArchive then uses stream and reads/writes to the file.
COleStreamFile stream;
//Stream is pointed to footer location.
stream.OpenStream(m_pStg, "Footer", nOpenFlags, pError); // pStg is LPSTORAGE
CArchive ar(&stream, CArchive::load);
Now after this code when I do ar >> or ar << I didn't read file from start. It is reading from middle or end (depends on stream). Now what I want is that to convert this code to C#. Whats the replacement for COleStreamFile::OpenStream in C#.
Here is what I have done so far.
using (var stream = new FileStream(filePath, FileMode.Open))
{
using (var binaryReader = new BinaryReader(stream)
{
}
}
Now here stream is pointing to the start. I think I can give it to read from specific byte. But I don't know that byte location. What I know is Header, Body and Footer names which are being used by MFC code.
Or is there any way to find out the current location of CArchive when it is reading or writing. If I get byte location from there I can use that as well.
Related
I'm trying to implement a system for a game editor wherein it will load a predefined texture in the library. The texture uses a proprietary format that the library is already capable of reading, but currently only supports doing so using a pointer address and length as input.
I've currently gotten it working by moving the texture into a temporary file, but this is a fairly unoptimized workaround. I was wondering if there was any way in particular to get the address of the internal file, either with or without the stream. Here is my current (working) code, I can provide additional information on how NodeFactory works if necessary but the main thing is that it requires an address and length to work properly.
var resourceName = ("BrawlLib.HardcodedFiles.metal00.tex0");
string listDefault = "";
using (Stream stream = assembly.GetManifestResourceStream(resourceName))
{
string file;
FileStream fstream = new FileStream((file = Path.GetTempFileName()), FileMode.OpenOrCreate);
stream.CopyTo(fstream);
tNode = NodeFactory.FromFile(null, file) as TEX0Node;
Texture.Attach(tNode, _palette);
}
I am trying to consume a streamed response in Python from a soap API, and output a CSV file. The response outputs a string coded in base 64, which I do not know what to do with. Also the api documentation says that the response must be read to a destination buffer-by-buffer.
Here is the C# code was provided by the api's documentation:
byte[] buffer = new byte[4000];
bool endOfStream = false;
int bytesRead = 0;
using (FileStream localFileStream = new FileStream(destinationPath, FileMode.Create, FileAccess.Write))
{
using (Stream remoteStream = client.DownloadFile(jobId))
{
while (!endOfStream)
{
bytesRead = remoteStream.Read(buffer, 0, buffer.Length);
if (bytesRead > 0)
{
localFileStream.Write(buffer, 0, bytesRead);
totalBytes += bytesRead;
}
else
{
endOfStream = true;
}
}
}
}
I have tried many different things to get this stream to a readable csv file, but non have worked.
with open('test.csv', 'w') as f: f.write(FileString)
Returns a csv with the base64 string spread over multiple lines
Here is my latest attempt:
with open('csvfile13.csv', 'wb') as csvfile:
FileString = client.service.DownloadFile(yyy.JobId, False)
stream = io.BytesIO(str(FileString))
with open(stream,"rt",4000) as readstream:
csvfile.write(readstream)
This produces the error:
TypeError: coercing to Unicode: need string or buffer, _io.BytesIO
Any help would be greatly appreciated, even if it is just to point me in the right direction. I will be ensure to award the points to whoever is the most helpful, even if I do not completely solve the issue!
I have asked several questions similar to this one, but I have yet to find an answer that works completely:
What is the Python equivalent to FileStream in C#?
Write Streamed Response(file-like object) to CSV file Byte by Byte in Python
How to replicate C# 'byte' and 'Write' in Python
Let me know if you need further clarification!
Update:
I have tried print(base64.b64decode(str(FileString)))
This gives me a page full of webdings like
]�P�O�J��Y��KW �
I have also tried
for data in client.service.DownloadFile(yyy.JobId, False):
print data
But this just loops through the output character by characater like any other string.
I have also managed to get a long string of bytes like \xbc\x97_D\xfb(not actual bytes, just similar format) by decoding the entire string, but I do not know how to make this readable.
Edit: Corrected the output of the sample python, added more example code, formatting
It sounds like you need to use the base64 module to decode the downloaded data.
It might be as simple as:
with open(destinationPath, 'w') as localFile:
remoteFile = client.service.DownloadFile(yyy.JobId, False)
remoteData = str(remoteFile).decode('base64')
localFile.write(remoteData)
I suggest you break the problem down and determine what data you have at each stage. For example what exactly are you getting back from client.service.DownloadFile?
Decoding your sample downloaded data (given in the comments):
'UEsYAItH7brgsgPutAG\AoAYYAYa='.decode('base64')
gives
'PK\x18\x00\x8bG\xed\xba\xe0\xb2\x03\xee\xb4\x01\x80\xa0\x06\x18\x01\x86'
This looks suspiciously like a ZIP file header. I suggest you rename the file .zip and open it as such to investigate.
If remoteData is a ZIP something like the following should extract and write your CSV.
import io
import zipfile
remoteFile = client.service.DownloadFile(yyy.JobId, False)
remoteData = str(remoteFile).decode('base64')
zipStream = io.BytesIO(remoteData)
z = zipfile.ZipFile(zipStream, 'r')
csvData = z.read(z.infolist()[0])
with open(destinationPath, 'w') as localFile:
localFile.write(csvData)
Note: BASE64 can have some variations regarding padding and alternate character mapping but once you can see the data it should be reasonably clear what you need. Of course carefully read the documentation on your SOAP interface.
Are you sure FileString is a Base64 string? Based on the source code here, suds.sax.text.Text is a subclass of Unicode. You can write this to a file as you would a normal string but whatever you use to read the data from the file may corrupt it unless it's UTF-8-encoded.
You can try writing your Text object to a UTF-8-encoded file using io.open:
import io
with io.open('/path/to/my/file.txt', 'w', encoding='utf_8') as f:
f.write(FileString)
Bear in mind, your console or text editor may have trouble displaying non-ASCII characters but that doesn't mean they're not encoded properly. Another way to inspect them is to open the file back up in the Python interactive shell:
import io
with io.open('/path/to/my/file.txt', 'r', encoding='utf_8') as f:
next(f) # displays the representation of the first line of the file as a Unicode object
In Python 3, you can even use the built-in csv to parse the file, however in Python 2, you'll need to pip install backports.csv because the built-in module doesn't work with Unicode objects:
from backports import csv
import io
with io.open('/path/to/my/file.txt', 'r', encoding='utf_8') as f:
r = csv.reader(f)
next(r) # displays the representation of the first line of the file as a list of Unicode objects (each value separated)
I am trying to use a LibTiff.Net library and rewriting a merge tool TiffCP api to use memory streams.
This library has a Tiff class and by passing a stream to this class, it can merge tiff images into this stream.
For testing, I passed on a Filestream and I got what i wanted - it merged and I was able to see multipage tif.
But when I pass a MemoryStream, I am able to verify that the page data is being added to the stream as I loop through but when I write it to the file at the end, I could see only 1st page.
var mso = new MemoryStream();
var fso = new FileStream(#"C:\test\ttest.tif",FileMode.OpenOrCreate); //This works
using (Tiff outImage = Tiff.ClientOpen("custom", "w", mso, tso))
{
//...
//..
System.Drawing.Image tiffImg = System.Drawing.Image.FromStream(mso, true);
tiffImg.Save(#"C:\test\test2.tiff", System.Drawing.Imaging.ImageFormat.Tiff);
tiffImg.Dispose();
//..
//..
}
P.S: I need it in memorystream because, of some folder permissions on servers + vendor API reasons.
You probably using the memory stream before data is actually written into the stream.
Please use Tiff.Flush() method before accessing data in the memory stream. And please make sure you call Tiff.WriteDirectory() method for each page you create.
EDIT:
Please also take a look at Bob Powell's article on Generating Multi-Page TIFF files. The article shows how to use EncoderParameters to actually generate a multipage TIFF.
Using
tiffImg.Save(#"C:\test\test2.tiff", System.Drawing.Imaging.ImageFormat.Tiff);
you are probably save only first frame.
I am developing an application which requires to read a text file which is continuously updating. I need to read the file till the end of file(at that very moment) and need to remember this location for my next file read. I am planning to develop this application in C#.net . How can I perform these partial reads and remember the locations as C# does not provide pointers in file handling ?
Edit : File is updated every 30 seconds and new data is appended to the old file.
I tried with maintaining the length of previous read and then reading the data from that location but the file can not be accessed by two applications at the same time.
You can maintain the last offset of the read pointer in the file. You can do sth like this
long lastOffset = 0;
using (var fs = new FileStream("myFile.bin", FileMode.Open))
{
fs.Seek(lastOffset, SeekOrigin.Begin);
// Read the file here
// Once file is read, update the lastOffset
lastOffset=fs.Seek(0, SeekOrigin.End);
}
Open the file, read everything, save the count of bytes you read in a variable (let's assume it's read_until_here).
next time you read the file, just take the new information (whichever comes after the location of your read_until_here variable ...
I am planning to develop this application in C#.net . How can I perform these partial reads and remember the locations as C# does not provide pointers in file handling ?
Not entirely sure why you'd be concerned about the supposed lack of pointers... I'd look into int FileStream.Read(byte[] array, int offset, int count) myself.
It allows you to read from an offset into a buffer, as many bytes as you want, and tells you how many bytes were actually read... which looks to be all the functionality you'd need.
While opening a file in C# using stream reader is the file going to remain in memory till it closed.
For eg if a file of size 6MB is opened by a program using streamreader to append a single line at the end of the file. Will the program hold the entire 6 MB in it's memory till file is closed. OR is a file pointer returned internally by .Net code and the line is appended at the end. So the 6MB memory will not be taken up by the program
The whole point of a stream is so that you don't have to hold an entire object in memory. You read from it piece by piece as needed.
If you want to append to a file, you should use File.AppendText which will create a StreamWriter that appends to the end of a file.
Here is an example:
string path = #"c:\temp\MyTest.txt";
// This text is always added, making the file longer over time
// if it is not deleted.
using (StreamWriter sw = File.AppendText(path))
{
sw.WriteLine("This");
sw.WriteLine("is Extra");
sw.WriteLine("Text");
}
Again, the whole file will not be stored in memory.
Documentation: http://msdn.microsoft.com/en-us/library/system.io.file.appendtext.aspx
The .NET FileStream will buffer a small amount of data (you can set this amount with some of the constructors).
The Windows OS will do more significant caching of the file, if you have plenty of RAM this might be the whole file.
A StreamReader uses FileStream to open the file. FileStream stores a Windows handle, returned by the CreateFile() API function. It is 4 bytes on a 32-bit operating system. FileStream also has a byte[] buffer, it is 4096 bytes by default. This buffer avoids having to call the ReadFile() API function for every single read call. StreamReader itself has a small buffer to make decoding the text in the file more efficient, it is 128 bytes by default. And it has some private variables to keep track of the buffer index and whether or not a BOM has been detected.
This all adds up to a few kilobytes. The data you read with StreamReader will of course take space in your program's heap. That could add up to 12 megabytes if you store every string in, say, a List. You usually want to avoid that.
StreamReader will not read the 6 MB file into memory. Also, you can't append a line to the end of the file using StreamReader. You might want to use StreamWriter.
update: not counting buffering and OS caching as someone else mentioned