Develop Reference

Decode Stream to CSV in Python by Byte (Translate from C# code)

I am trying to consume a streamed response in Python from a soap API, and output a CSV file. The response outputs a string coded in base 64, which I do not know what to do with. Also the api documentation says that the response must be read to a destination buffer-by-buffer.
Here is the C# code was provided by the api's documentation:
byte[] buffer = new byte[4000];
bool endOfStream = false;
int bytesRead = 0;
using (FileStream localFileStream = new FileStream(destinationPath, FileMode.Create, FileAccess.Write))
{
using (Stream remoteStream = client.DownloadFile(jobId))
{
while (!endOfStream)
{
bytesRead = remoteStream.Read(buffer, 0, buffer.Length);
if (bytesRead > 0)
{
localFileStream.Write(buffer, 0, bytesRead);
totalBytes += bytesRead;
}
else
{
endOfStream = true;
}
}
}
}
I have tried many different things to get this stream to a readable csv file, but non have worked.
with open('test.csv', 'w') as f: f.write(FileString)
Returns a csv with the base64 string spread over multiple lines
Here is my latest attempt:
with open('csvfile13.csv', 'wb') as csvfile:
FileString = client.service.DownloadFile(yyy.JobId, False)
stream = io.BytesIO(str(FileString))
with open(stream,"rt",4000) as readstream:
csvfile.write(readstream)
This produces the error:
TypeError: coercing to Unicode: need string or buffer, _io.BytesIO
Any help would be greatly appreciated, even if it is just to point me in the right direction. I will be ensure to award the points to whoever is the most helpful, even if I do not completely solve the issue!
I have asked several questions similar to this one, but I have yet to find an answer that works completely:
What is the Python equivalent to FileStream in C#?
Write Streamed Response(file-like object) to CSV file Byte by Byte in Python
How to replicate C# 'byte' and 'Write' in Python
Let me know if you need further clarification!
Update:
I have tried print(base64.b64decode(str(FileString)))
This gives me a page full of webdings like
]�P�O�J��Y��KW �
I have also tried
for data in client.service.DownloadFile(yyy.JobId, False):
print data
But this just loops through the output character by characater like any other string.
I have also managed to get a long string of bytes like \xbc\x97_D\xfb(not actual bytes, just similar format) by decoding the entire string, but I do not know how to make this readable.
Edit: Corrected the output of the sample python, added more example code, formatting

It sounds like you need to use the base64 module to decode the downloaded data.
It might be as simple as:
with open(destinationPath, 'w') as localFile:
remoteFile = client.service.DownloadFile(yyy.JobId, False)
remoteData = str(remoteFile).decode('base64')
localFile.write(remoteData)
I suggest you break the problem down and determine what data you have at each stage. For example what exactly are you getting back from client.service.DownloadFile?
Decoding your sample downloaded data (given in the comments):
'UEsYAItH7brgsgPutAG\AoAYYAYa='.decode('base64')
gives
'PK\x18\x00\x8bG\xed\xba\xe0\xb2\x03\xee\xb4\x01\x80\xa0\x06\x18\x01\x86'
This looks suspiciously like a ZIP file header. I suggest you rename the file .zip and open it as such to investigate.
If remoteData is a ZIP something like the following should extract and write your CSV.
import io
import zipfile
remoteFile = client.service.DownloadFile(yyy.JobId, False)
remoteData = str(remoteFile).decode('base64')
zipStream = io.BytesIO(remoteData)
z = zipfile.ZipFile(zipStream, 'r')
csvData = z.read(z.infolist()[0])
with open(destinationPath, 'w') as localFile:
localFile.write(csvData)
Note: BASE64 can have some variations regarding padding and alternate character mapping but once you can see the data it should be reasonably clear what you need. Of course carefully read the documentation on your SOAP interface.

Are you sure FileString is a Base64 string? Based on the source code here, suds.sax.text.Text is a subclass of Unicode. You can write this to a file as you would a normal string but whatever you use to read the data from the file may corrupt it unless it's UTF-8-encoded.
You can try writing your Text object to a UTF-8-encoded file using io.open:
import io
with io.open('/path/to/my/file.txt', 'w', encoding='utf_8') as f:
f.write(FileString)
Bear in mind, your console or text editor may have trouble displaying non-ASCII characters but that doesn't mean they're not encoded properly. Another way to inspect them is to open the file back up in the Python interactive shell:
import io
with io.open('/path/to/my/file.txt', 'r', encoding='utf_8') as f:
next(f) # displays the representation of the first line of the file as a Unicode object
In Python 3, you can even use the built-in csv to parse the file, however in Python 2, you'll need to pip install backports.csv because the built-in module doesn't work with Unicode objects:
from backports import csv
import io
with io.open('/path/to/my/file.txt', 'r', encoding='utf_8') as f:
r = csv.reader(f)
next(r) # displays the representation of the first line of the file as a list of Unicode objects (each value separated)

convert binary file to text

I have a program that gets a response from a url in binary format and I do not know how to convert this to a text file.
byte[] postBytes = System.Text.Encoding.UTF8.GetBytes(postString);
request.ContentLength = postBytes.Length;
Stream stream = request.GetRequestStream();
stream.Write(postBytes, 0, postBytes.Length);
stream.Close();
response = (HttpWebResponse)request.GetResponse();
Stream ReceiveStream = response.GetResponseStream();
string filename = "C:\\responseGot.txt";
byte[] buffer = new byte[1024];
FileStream outFile = new FileStream(filename, FileMode.Create);
int bytesRead;
while ((bytesRead = ReceiveStream.Read(buffer, 0, buffer.Length)) != 0)
outFile.Write(buffer, 0, bytesRead);
When I open responseGot.txt it is a binary file how do I get text file.

In what format is the response you get? There is no such thing as a text file. There are only binary files. HTTP is also 100% binary.
Text is the interpretation of bytes, and it only exists as part of running application. You can never, ever write text to a file. You can only convert the text to bytes (using various ways) and write the bytes.
Therefore, ask yourself why the bytes you received cannot be interpreted by notepad.exe as text. Maybe the response is not directly text but a ZIP file or something.
You can guess the format with a hex editor
You can ask the website owner

You don't show in your code sample saving the file anywhere.
But to convert the response to string you can use:
using (HttpWebResponse response = req.GetResponse() as HttpWebResponse)
{
StreamReader reader = new StreamReader(response.GetResponseStream());
string ResponseTXT = reader.ReadToEnd();
}
Then you can save it with usual techniques
http://msdn.microsoft.com/en-us/library/6ka1wd3w%28v=vs.110%29.aspx
Did you mean that?

Every data represented in digital computing these days is based on 2 bits ie. binary (electrical/magnetic signals: on/off or north/south).
Every file written to disk is also a binary file ie. a sequence of (8 bit) bytes.
ASCII/ANSI defines character map for each byte sequence and only about 95 of the 256 bytes are referred to as printable (text) characters.
Your downloaded file seems to have more than just the printable characters (usually referred to as a plain text file).
To view the file as it is (in your current encoding settings):
type <file.ext>
To view in a different code page:
chcp <codepage>
type <file.ext>
To view a (plain)text representation of your file, you'd encode it first (ie. translate it to a text file) eg. hex coded string via some hex editor.
The first few characters of the hex sequence should give a magic number, indicating the type of file being read. You'd then open the file with the associated program (that is capable of opening those types of files).
If it is a text file you were expecting and instead got a file which has more than just printable (plain text) characters, then it's more likely there has been some sort of compression/encryption applied to it. Once again, the magic number should hint how the file should be treated eg. decompressed before attempting to read the data/file. (Encrypted files should come with a decryption hint/key, unless exchanged/agreed earlier)

Use the ReadFully method in this topic Creating a byte array from a stream
Get the string representation to an actual string:
string text = System.Text.Encoding.Default.GetString(byteArray);
And finally create the text file and write the content:
using(StreamWriter sw = new StreamWriter("C:\\responseGot.txt"))
{
sw.WriteLine(text);
}

Compressing a file from memory with SevenZipSharp, stranges mistakes

I download the SevenZipSharp Lib in order to compress some files.
I used this in order to compress a file :
var libPath = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.ProgramFiles), "7-zip", "7z.dll");
SevenZip.SevenZipCompressor.SetLibraryPath(libPath);
SevenZip.SevenZipCompressor compressor = new SevenZipCompressor();
compressor.CompressFiles(#"C:\myTestFile.mdf", new string[] { #"C:\myTestFileCompressed.7z" });
With this, my file is compressed whitout problem. I can decompressed it.
Now...i would like to compress the same file, but, instead of compress directly the file, i would like to :
Read the file in a string. Yes, in a string, and not in a byte[].
Convert my string to byte[]
Compress the byte[] to another byte[].
Here is my try :
string strToCompress = File.ReadAllText(#"C:\myTestFile.mdf");
SevenZipCompressor compressor = new SevenZipCompressor();
byte[] byteArrayToCompress = Encoding.ASCII.GetBytes(text);
MemoryStream stream = new MemoryStream(byteArrayToCompress);
MemoryStream streamOut = new MemoryStream();
compressor.CompressStream(stream, streamOut);
string strcompressed = Encoding.ASCII.GetString(streamOut.ToArray());
File.WriteAllText(#"C:\myfileCompressed.7z",strcompressed);
My problem is very simple :
If i compare the size produced by these 2 methods, it's 3 603 443 bytes vs 3 604 081 bytes.
In addition, i cannot uncompressed the file produced by the second method.
Maybe it's because i used ASCII encoding, but my file to compress is not a Text, it's a binary file.
Anyone could explain me how solving it please ? I need to read my file to a string and compress it. ( i don't want to read the file directly to a byte[]).
Thanks a lot,
Best regards,
Nixeus

You cannot put binary data into a string, not every byte value has a Unicode codepoint. Using ASCII encoding will similarly always cause irretrievable data loss, it only has characters for byte values 0 through 127, higher values will produce a ?
You certainly can convert a byte[] to a string, it needs to be encoded. The standard encoding that's used for that is available in .NET from the Convert.ToBase64String() method. You recover the byte[] again with Convert.FromBase64String(). Inevitably it won't be as compact, it will be 4/3 bigger as the original data in a byte[].
You can never produce a valid .7z archive that way, it of course uses the most compact possible storage and that is bytes. You must pass a FileStream to the CompressStream() method.

Convert the contents of a byte array to string

I have a byte array that contains the data of an uploaded file which happens to be a Resume of an employee(.doc file). I did it with the help of the following lines of code
AppSettingsReader rd = new AppSettingsReader();
FileUpload arr = (FileUpload)upresume;
Byte[] arrByte = null;
if (arr.HasFile && arr.PostedFile != null)
{
//To create a PostedFile
HttpPostedFile File = upresume.PostedFile;
//Create byte Array with file len
arrByte = new Byte[File.ContentLength];
//force the control to load data in array
File.InputStream.Read(arrByte, 0, File.ContentLength);
}
Now, I would like to get the contents of the uploaded file(resume) in string format either from the byte array or any other methods.
PS: 'contents' literally refers to the contents of the resume; for example if the resume(uploaded file) contains a word 'programming', I would like to have the same word contained in the string.
Please help me to solve this.

I worked on a similar project a few years ago. Long story short... I ended up reconstructing the file and saving it on the server, then programmatically convert it to pdf, and then index the contents of the pdf, this proved much easier in practice at the time.
Alternatively, if you can restrict resume uploads to docx file format, you can use Microsofts OpenXML library to parse and index the content very easily. But in practict this may cause usability issues for users of the web site.

get the video file after convert from base64 and UTF8 in c#

what i need to do is sending a file from java to c#. the java act as the client meanwhile, c# act as server.
the file is loaded in java through fileinputstream and its been converted to utf8 then base64. see the code.
FileInputStream fin=new FileInputStream(fileName);
byte[] content = new byte[fin.available()];
fin.read(content, 0, content.length);
String asString = new String(content, "UTF8");
byte[] newBytes = asString.getBytes("UTF8");
String base64 = Base64.encodeToString(newBytes, Base64.DEFAULT);
and the server (using c# language) will read the data send and convert it back as a file. im using base64 then to utf8 and last i am not sure how to make it. what im trying to send is video.mp4 size of 144kb or less. so far, the output shows the catch of "WRONG FORMAT". see the code.
try
{
for (int i = 0; i <= _server.Q.NoOfItem - 1; i++)
{
words = _server.Q.ElementAtBuffer(i).ToString();
//textBox1.Text = words;
byte[] encodedDataAsBytes = System.Convert.FromBase64String(words);
string returnValue = System.Text.Encoding.UTF8.GetString(encodedDataAsBytes);
textBox1.Text = returnValue;
}
}
catch (ArgumentNullException argNull)
{
textBox1.Text = "Received null value";
}
catch (FormatException FrmtEx)
{
textBox1.Text = "Wrong format";
}
you can ignore the for (int i = 0; i <= _server.Q.NoOfItem - 1; i++) as this is the way i want to capture/retrieve the data sent.
p/s: it works when im just trying to pass any string without load the file (string >> utf8 >> base64) and to receive (base64 >> utf8 >> string).

the file is loaded in java through fileinputstream and its been converted to utf8
Then you've lost data. Video data is not text data, so don't load it as text data. Treat it as binary data - by all means encode it to base64 if you need to represent it as a string somewhere but don't perform any text decoding on it, as that's only meant for encoded text data, which this isn't.
It's really important to understand what's wrong here. The only thing the two lines below can do is lose data. If they don't lose data, they serve no purpose - and if they do lose data, they're clearly a bad idea:
String asString = new String(content, "UTF8");
byte[] newBytes = asString.getBytes("UTF8");
You should analyze how you ended up with this code in the first place... why did you feel the need to convert the byte array to a string and back?
jowierun's answer is also correct - you shouldn't be using available() at all. You might want to use utility methods from Guava, such as Files.toByteArray if you definitely need to read the whole file into memory in one go.
p/s: it works when im just trying to pass any string without load the file (string >> utf8 >> base64) and to receive (base64 >> utf8 >> string).
Well yes - if you start with text data, then that's fine - UTF-8 can represent every valid string, and base64 is lossless, so you're fine. (Admittedly you could break it by presenting an invalid string with half of a surrogate pair, but...) The problem is at the point where you treat non-text data as text in the first place.

You shouldn't use fin.available() to assume you can read the file in one go. That is likely to work for only small files. Instead you need to do the read in a loop and collect all the contents together before you encode it.
It would make sense to (on the java side at least) to have a decode that routine you can use to TEST that your encode is working (a unit test perhaps?). You will probably find that test is failing consistently with the problem you are getting.