Develop Reference

Decode Stream to CSV in Python by Byte (Translate from C# code)

I am trying to consume a streamed response in Python from a soap API, and output a CSV file. The response outputs a string coded in base 64, which I do not know what to do with. Also the api documentation says that the response must be read to a destination buffer-by-buffer.
Here is the C# code was provided by the api's documentation:
byte[] buffer = new byte[4000];
bool endOfStream = false;
int bytesRead = 0;
using (FileStream localFileStream = new FileStream(destinationPath, FileMode.Create, FileAccess.Write))
{
using (Stream remoteStream = client.DownloadFile(jobId))
{
while (!endOfStream)
{
bytesRead = remoteStream.Read(buffer, 0, buffer.Length);
if (bytesRead > 0)
{
localFileStream.Write(buffer, 0, bytesRead);
totalBytes += bytesRead;
}
else
{
endOfStream = true;
}
}
}
}
I have tried many different things to get this stream to a readable csv file, but non have worked.
with open('test.csv', 'w') as f: f.write(FileString)
Returns a csv with the base64 string spread over multiple lines
Here is my latest attempt:
with open('csvfile13.csv', 'wb') as csvfile:
FileString = client.service.DownloadFile(yyy.JobId, False)
stream = io.BytesIO(str(FileString))
with open(stream,"rt",4000) as readstream:
csvfile.write(readstream)
This produces the error:
TypeError: coercing to Unicode: need string or buffer, _io.BytesIO
Any help would be greatly appreciated, even if it is just to point me in the right direction. I will be ensure to award the points to whoever is the most helpful, even if I do not completely solve the issue!
I have asked several questions similar to this one, but I have yet to find an answer that works completely:
What is the Python equivalent to FileStream in C#?
Write Streamed Response(file-like object) to CSV file Byte by Byte in Python
How to replicate C# 'byte' and 'Write' in Python
Let me know if you need further clarification!
Update:
I have tried print(base64.b64decode(str(FileString)))
This gives me a page full of webdings like
]�P�O�J��Y��KW �
I have also tried
for data in client.service.DownloadFile(yyy.JobId, False):
print data
But this just loops through the output character by characater like any other string.
I have also managed to get a long string of bytes like \xbc\x97_D\xfb(not actual bytes, just similar format) by decoding the entire string, but I do not know how to make this readable.
Edit: Corrected the output of the sample python, added more example code, formatting

It sounds like you need to use the base64 module to decode the downloaded data.
It might be as simple as:
with open(destinationPath, 'w') as localFile:
remoteFile = client.service.DownloadFile(yyy.JobId, False)
remoteData = str(remoteFile).decode('base64')
localFile.write(remoteData)
I suggest you break the problem down and determine what data you have at each stage. For example what exactly are you getting back from client.service.DownloadFile?
Decoding your sample downloaded data (given in the comments):
'UEsYAItH7brgsgPutAG\AoAYYAYa='.decode('base64')
gives
'PK\x18\x00\x8bG\xed\xba\xe0\xb2\x03\xee\xb4\x01\x80\xa0\x06\x18\x01\x86'
This looks suspiciously like a ZIP file header. I suggest you rename the file .zip and open it as such to investigate.
If remoteData is a ZIP something like the following should extract and write your CSV.
import io
import zipfile
remoteFile = client.service.DownloadFile(yyy.JobId, False)
remoteData = str(remoteFile).decode('base64')
zipStream = io.BytesIO(remoteData)
z = zipfile.ZipFile(zipStream, 'r')
csvData = z.read(z.infolist()[0])
with open(destinationPath, 'w') as localFile:
localFile.write(csvData)
Note: BASE64 can have some variations regarding padding and alternate character mapping but once you can see the data it should be reasonably clear what you need. Of course carefully read the documentation on your SOAP interface.

Are you sure FileString is a Base64 string? Based on the source code here, suds.sax.text.Text is a subclass of Unicode. You can write this to a file as you would a normal string but whatever you use to read the data from the file may corrupt it unless it's UTF-8-encoded.
You can try writing your Text object to a UTF-8-encoded file using io.open:
import io
with io.open('/path/to/my/file.txt', 'w', encoding='utf_8') as f:
f.write(FileString)
Bear in mind, your console or text editor may have trouble displaying non-ASCII characters but that doesn't mean they're not encoded properly. Another way to inspect them is to open the file back up in the Python interactive shell:
import io
with io.open('/path/to/my/file.txt', 'r', encoding='utf_8') as f:
next(f) # displays the representation of the first line of the file as a Unicode object
In Python 3, you can even use the built-in csv to parse the file, however in Python 2, you'll need to pip install backports.csv because the built-in module doesn't work with Unicode objects:
from backports import csv
import io
with io.open('/path/to/my/file.txt', 'r', encoding='utf_8') as f:
r = csv.reader(f)
next(r) # displays the representation of the first line of the file as a list of Unicode objects (each value separated)

How to read a binary stream from C# BinaryWriter in PHP?

Is there anything open source available for this?
or
Is there a way to parse a stream of bytes received from a POST request manually and convert the chunks of bytes to the appropriate data types?

I'm not sure if there is anything open source for this, but PHP does support the needed features out of the box.
the contents of a POST request can be retrieved as follows:
$data = file_get_contents("php://input");
// or to handle the data as a stream
$stream = fopen("php://input", "rb");
The above is the preferred method, as $HTTP_RAW_POST_DATA is deprecated.
The data can then be parsed using the PHP unpack() function.

PDF Image from Binary in MTOM

I am making a custom HttpWebRequest call to a GSoap web service that returns an XML + XOP envelope which contains a PDF Image as binary.
I am grabbing the response and taking the binary code in between the boundary string.
Finally, I convert the binary to byte[] and save it as a PDF.
Now, I can see the PDF metadata so the encoding was right, but when I try to open it, I get insufficient data for an image error and the image inside the PDF is not displayed.
I am converting the binary string via this:
retBytes = System.Text.Encoding.UTF8.GetBytes(modStr);
Where modStr is the string starting with %PDF-1.1 and ending with %%EOF. Do I need to do more encoding/decoding so that the image shows up granted I can see everything else (pages/metadata, etc)?

Google protocol buffers-Sending a message form C# client to a java Server

The Client sends a 1481 bytes array.
The server can read all the 1481 bytes message without any problems but by parsing the given messsage from the received binary array i get this exeption:
com.google.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
The binary data is the same. I checked that I am using the right version of the proto files. I am a bit at a loss tbh. Any help appreciated.
Code
byte [] data= IOUtils.toByteArray(br1, "ASCII");
System.out.println("SIZE:" + data.length);
AddressBook adb1 = AddressBook.parseFrom(data); System.out.println("Server: Addressbook:" + adb1.getPersonCount()); System.out.println("Server: Addressbook:" + adb1.getPerson(0).getName());
Question:
I need to find a way to correctly parse the received Adressbook msg from the read 1481 bytes arry.
Thanks.

This is the problem:
br1 = new InputStreamReader(s.getInputStream());
That's trying to treat opaque binary data as text. It's not text, it's binary data. So when you convert that Reader into a byte array, you've lost a load of the original data - no wonder it's an invalid protocol buffer.
Just use:
AddressBook adb1 = AddressBook.parseFrom(s.getInputStream());
and avoid the lossy text conversion. That's assuming you haven't got something equally broken on the C# side, of course.
If you must go via text, you should use base64 encoding on both sides.

Now it works I had same mistake by Serializing and Sending the Protocol Buffers Message

Pass object from webserver to client

I developed a C# web application that calls a web-service which returns a base64 encoded array (PDF file). I then convert that array into a UCOMIStream object (I know it is obsolete, but the DLL that I am using requires it as a parameter). I use the following code to do the conversion which works perfectly. I can pass this object to the DLL so that I can print the PDF.
This works great on the Webserver, but the requirement is to print it locally.
Byte[] bBuffer = statementOut.statementcycle.statementdata.content;
int size = bBuffer.Length;
IntPtr mem = Marshal.AllocHGlobal(size);
Marshal.Copy(bBuffer, 0, mem, size);
// Create an OLE Stream object.
System.Runtime.InteropServices.UCOMIStream str; //obsolete but the createstreamonhglobal outputs it
CreateStreamOnHGlobal(mem, true, out str);
The DLL resides on the client so I am able to use ActiveX to create the object using javascript and/or VBscript;however, I have not been able to figure out how to get the stream object to the client to pass to the DLL.
How can this be achieved?

Couldn't you just generate the pdf on the server and have the client download it?

Have the client download that base64 encoded array and then translate the data into a UCOMIStream object and generate the PDF the client side.