Defined a gRPC Python service in a Docker.
The services in my PROTO file:
rpc ClientCommand(ClientRequest) returns (ClientResponse){}
Definition of "ClientResponse":
message ClientResponse{
int32 request_id = 1;
repeated int32 prediction_status = 2;
repeated string prediction_info = 3;
repeated string prediction_error = 4;
repeated string prediction_result_name = 5;
repeated bytes prediction_result = 6;
repeated bytes prediction_config = 7;
repeated bytes prediction_log = 8;
}
On client side, I want to catch the repeated bytes and convert them into a file(I know it better works with a stream but for the moment I would like to do it like this).
The repeated strings and integer I can perfectly convert to List --> OK
The repeated bytes I would like to convert to Byte[]. Their type: Google.ProtoBuf.Collections.RepeatedField<Google.ProtoBuf.ByteString>.
At first it seems to be impossible to convert this type to a Byte[]. Could somebody help me with this please? My solution temporary:
byte[] test = new byte[100];
Google.Protobuf.ByteString[] test2 = new Google.Protobuf.ByteString[100];
response.PredictionResult.CopyTo(test2,0);
test2.CopyTo(test,0);
WriteFile(#"C:\programs\file.txt", test);
Meanwhile I figure it out:
public void WriteFileResult(string fileName,Google.Protobuf.Collections.RepeatedField<ByteString> data)
{
byte[] bytes = new byte[data.Count];
ByteString[] dataByteString = new ByteString[data.Count];
string path = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);
if (!path.EndsWith(#"\")) path += #"\";
if (File.Exists(Path.Combine(path, fileName)))
File.Delete(Path.Combine(path, fileName));
using (FileStream fs = new FileStream(Path.Combine(path, fileName), FileMode.CreateNew, FileAccess.Write))
{
data.CopyTo(dataByteString, 0);
bytes = dataByteString[0].ToByteArray();
fs.Write(bytes, 0, (int)bytes.Length);
_serviceResponseModel.Json = false;
//fs.Close()
}
}
I am using ZXing.Net library to encode and decode my video file using RS Encoder. It works well by adding and and removing parity after encoding and decoding respectively. But When writing decoded file back it is adding "?" characters in file on different locations which was not part of original file. I am not getting why this problem is occurring when writing file back.
Here is my code
using ZXing.Common.ReedSolomon;
namespace zxingtest
{
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
string inputFileName = #"D:\JM\bin\baseline_30.264";
string outputFileName = #"D:\JM\bin\baseline_encoded.264";
string Content = File.ReadAllText(inputFileName, ASCIIEncoding.Default);
//File.WriteAllText(outputFileName, Content, ASCIIEncoding.Default);
ReedSolomonEncoder enc = new ReedSolomonEncoder(GenericGF.AZTEC_DATA_12);
ReedSolomonDecoder dec = new ReedSolomonDecoder(GenericGF.AZTEC_DATA_12);
//string s = "1,2,4,6,1,7,4,0,0";
//int[] array = s.Split(',').Select(str => int.Parse(str)).ToArray();
int parity = 10;
List<byte> toBytes = ASCIIEncoding.Default.GetBytes(Content.Substring(0, 500)).ToList();
for (int index = 0; index < parity; index++)
{
toBytes.Add(0);
}
int[] bytesAsInts = Array.ConvertAll(toBytes.ToArray(), c => (int)c);
enc.encode(bytesAsInts, parity);
bytesAsInts[1] = 3;
dec.decode(bytesAsInts, parity);
string st = new string(Array.ConvertAll(bytesAsInts.ToArray(), z => (char)z));
File.WriteAllText(outputFileName, st, ASCIIEncoding.Default);
}
}
}
And here is the Hex file view of H.264 bit stream
The problem is that you're handling a binary format as if it is a Text file with an encoding. But based on what you are doing you only seem to be interested in reading some bytes, process them (encode, decode) and then write the bytes back to a file.
If that is what you need then use the proper reader and writer for your files, in this case the BinaryReader and BinaryWriter. Using your code as a starting point this is my version using the earlier mentioned readers/writers. My inputfile and outputfile are similar for the bytes read and written.
string inputFileName = #"input.264";
string outputFileName = #"output.264";
ReedSolomonEncoder enc = new ReedSolomonEncoder(GenericGF.AZTEC_DATA_12);
ReedSolomonDecoder dec = new ReedSolomonDecoder(GenericGF.AZTEC_DATA_12);
const int parity = 10;
// open a file as stream for reading
using (var input = File.OpenRead(inputFileName))
{
const int max_ints = 256;
int[] bytesAsInts = new int[max_ints];
// use a binary reader
using (var binary = new BinaryReader(input))
{
for (int i = 0; i < max_ints - parity; i++)
{
//read a single byte, store them in the array of ints
bytesAsInts[i] = binary.ReadByte();
}
// parity
for (int i = max_ints - parity; i < max_ints; i++)
{
bytesAsInts[i] = 0;
}
enc.encode(bytesAsInts, parity);
bytesAsInts[1] = 3;
dec.decode(bytesAsInts, parity);
// create a stream for writing
using(var output = File.Create(outputFileName))
{
// write bytes back
using(var writer = new BinaryWriter(output))
{
foreach(var value in bytesAsInts)
{
// we need to write back a byte
// not an int so cast it
writer.Write((byte)value);
}
}
}
}
}
I am trying to decompress a GZipped string which is part of response from a webservice. The string that I have is:
"[31,-117,8,0,0,0,0,0,0,0,109,-114,65,11,-62,48,12,-123,-1,75,-50,-61,-42,-127,30,122,21,111,-126,94,60,-119,-108,-72,102,44,-48,-75,-93,-21,100,56,-6,-33,-19,20,20,101,57,37,95,-14,94,-34,4,-63,-5,-72,-73,-44,-110,-117,-96,38,-88,26,-74,38,-112,3,117,-7,25,-82,5,24,-116,56,-97,-44,108,-23,28,24,-44,-85,83,34,-41,97,-88,24,-99,23,36,124,-120,94,99,-120,15,-42,-91,-108,91,45,-11,70,119,60,-110,21,-20,12,-115,-94,111,-80,-93,89,-41,-65,-127,-82,76,41,51,-19,52,90,-5,69,-85,76,-96,-128,64,22,35,-33,-23,-124,-79,-55,-1,-2,-10,-87,0,55,-76,55,10,-57,122,-9,73,42,-45,98,-44,5,-77,101,-3,58,-91,39,38,51,-15,121,21,1,0,0]"
I'm trying to decompress that string using the following method:
public static string UnZip(string value)
{
// Removing brackets from string
value = value.TrimStart('[');
value = value.TrimEnd(']');
//Transform string into byte[]
string[] strArray = value.Split(',');
byte[] byteArray = new byte[strArray.Length];
for (int i = 0; i < strArray.Length; i++)
{
if (strArray[i][0] != '-')
byteArray[i] = Convert.ToByte(strArray[i]);
else
{
int val = Convert.ToInt16(strArray[i]);
byteArray[i] = (byte)(val + 256);
}
}
//Prepare for decompress
System.IO.MemoryStream ms = new System.IO.MemoryStream(byteArray);
System.IO.Compression.GZipStream sr = new System.IO.Compression.GZipStream(ms,
System.IO.Compression.CompressionMode.Decompress);
//Reset variable to collect uncompressed result
byteArray = new byte[byteArray.Length];
//Decompress
int rByte = sr.Read(byteArray, 0, byteArray.Length);
//Transform byte[] unzip data to string
System.Text.StringBuilder sB = new System.Text.StringBuilder(rByte);
//Read the number of bytes GZipStream red and do not a for each bytes in
//resultByteArray;
for (int i = 0; i < rByte; i++)
{
sB.Append((char)byteArray[i]);
}
sr.Close();
ms.Close();
sr.Dispose();
ms.Dispose();
return sB.ToString();
}
The method is a modified version of the one in the following link:
http://www.codeproject.com/Articles/27203/GZipStream-Compress-Decompress-a-string
Sadly, the result of that method is a corrupted string. More specifically, I know that the input string contains a compressed JSON object and the output string has only some of the expected string:
"{\"rootElement\":{\"children\":[{\"children\":[],\"data\":{\"fileUri\":\"file:////Luciano/e/orto_artzi_2006_0_5_pixel/index/shapefiles/index_cd20/shp_all/index_cd2.shp\",\"relativePath\":\"/i"
Any idea what could be the problem and how to solve it?
Try
public static string UnZip(string value)
{
// Removing brackets from string
value = value.TrimStart('[');
value = value.TrimEnd(']');
//Transform string into byte[]
string[] strArray = value.Split(',');
byte[] byteArray = new byte[strArray.Length];
for (int i = 0; i < strArray.Length; i++)
{
byteArray[i] = unchecked((byte)Convert.ToSByte(strArray[i]));
}
//Prepare for decompress
using (System.IO.MemoryStream output = new System.IO.MemoryStream())
{
using (System.IO.MemoryStream ms = new System.IO.MemoryStream(byteArray))
using (System.IO.Compression.GZipStream sr = new System.IO.Compression.GZipStream(ms, System.IO.Compression.CompressionMode.Decompress))
{
sr.CopyTo(output);
}
string str = Encoding.UTF8.GetString(output.GetBuffer(), 0, (int)output.Length);
return str;
}
}
The MemoryBuffer() doesn't "duplicate" the byteArray but is directly backed by it, so you can't reuse the byteArray.
I'll add that I find funny that they "compressed" a json of 277 characters to a stringized byte array of 620 characters.
As a sidenote, the memory occupation of this method is out-of-the-roof... The 620 character string (that in truth is a 277 byte array) to be decompressed causes the creation of strings/arrays for a total size of 4887 bytes (including the 620 initial character string) (disclaimer: the GC can reclaim part of this memory during the execution of the method). This is ok for byte arrays of 277 bytes... But for bigger ones the memory occupation will become quite big.
Following on from Xanatos's answer in C# slightly modified to return a simple byte array. This takes a gzip compressed byte array and returns the inflated gunzipped array.
public static byte[] Decompress(byte[] compressed_data)
{
var outputStream = new MemoryStream();
using (var compressedStream = new MemoryStream(compressed_data))
using (System.IO.Compression.GZipStream sr = new System.IO.Compression.GZipStream(
compressedStream, System.IO.Compression.CompressionMode.Decompress))
{
sr.CopyTo(outputStream);
outputStream.Position = 0;
return outputStream.ToArray();
}
}
I have a problem with tcpclient, I need to send one or more files, so I have an application with server and client, the protocol is this :
1) I send some strings with information about number of files, files name and their sizes, all this with streamwriter.writeline (received from server with the function
streamreader.readline)
2) After these strings I send the files, after each file the server answers to the client with a streamwriter.writeline of "ACK". The file is sent with the
networkstream.write method, and received with networkstream.read.
The problem is that the server reads till the received bytes are equal to the file size, but... despite the client "seems" to send every byte of the file, the server
receives fewer bytes in total! So the application is blocked in this step, the server is waiting for the next bytes and the client is waiting for the string of "ACK"
with the streamreader.readline before to send the next file or just to finish the operation.
I also wanted to check what the server receives, so I print the number of bytes received during the reading cycle , discovering that sometimes the server reads fewer bytes than the buffer size of the stream (fixed to 1024). This should be normal because TCP reads as soon as it can, it should not be the real problem, right? I can't
believe that tcp loses bytes, but I don't know how to resolve.
Here you can find some part of codes :
----SERVER SIDE----------
..........Doing Stuffs.............
//secServer is the TCPListener socket.
this.secSHandler = secServer.AcceptTcpClient();
this.secSHandler.ReceiveBufferSize = 1024;
this.secSHandler.SendBufferSize = 1024;
this.is_connected_now = true;
print("is connected!!! ");
//Taking streams...
this.stream = this.secSHandler.GetStream();
this.sr = new StreamReader(stream);
this.sw = new StreamWriter(stream);
string first = sr.ReadLine();
print("I read + " + first + " .");
int numFiles = 0;
string connType = first.Substring(0, 6);
if (connType.CompareTo("CLIENT") == 0)
{
//SINCR CLIENT -> SERVER
string clipType = first.Substring(7, 4);
if (clipType.CompareTo("FILE") == 0)
{
//CASE RECEIVE FILE
int posSeparator = first.IndexOf('*');
string nFiles = first.Substring(12, first.Length - 13);
numFiles = Convert.ToInt16(nFiles);
string[] fileNames = new string[numFiles];
int[] fileSizes = new int[numFiles];
//TAKE FROM THE CLIENT ALL FILE NAMES AND SIZES
for (int i = 0; i < numFiles; i++)
{
fileNames[i] = sr.ReadLine();
print("Name file : I read " + fileNames[i]);
string dim = sr.ReadLine();
fileSizes[i] = Convert.ToInt32(dim);
print("Size file : I read " + fileSizes[i]);
}
//RECEVING FILES
for (int i = 0; i < numFiles; i++)
{
receive_single_file_1(stream, fileSizes[i], fileNames[i]); //CANNOT GO AFTER THIS POINT
sw.WriteLine("File sent - number " + i);
sw.Flush();
}
}
}
.............Doing others stuffs.............
sr.Close();
sw.Close();
THE FUNCTION RECEIVE SINGLE FILE IS HERE BELOW
public bool receive_single_file_1(NetworkStream netstream, int size, string filename)
{
int sizeToRead = 0;
string f_name = "";
//...f_name is the result of another operation, for the sake of the example i write only the final instruction
f_name = filename;
byte[] RecData = new byte[1024];
int RecBytes = 0;
try
{
int totalrecbytes = 0;
FileStream Fs = new FileStream((tempFold + f_name), FileMode.OpenOrCreate, FileAccess.Write);
//COUNTER OF THE WHILE
int nciclo = 0;
while ((RecBytes = netstream.Read(RecData, 0, 1024)) > 0)
{
//I defined print in another context...
totalrecbytes += RecBytes;
if(RecBytes!=1024)
print("Cycle : "+ nciclo +" Content RecBytes : " + RecBytes + " e RecData.Length : " + RecData.Length + " byte reads : " + totalrecbytes + ".");
Fs.Write(RecData, 0, RecBytes);
if (totalrecbytes >= size)
{
print("Read all bytes " + totalrecbytes + " over " + size + " .");
break;
}
//Refresh the buffer
RecData = new byte[1024];
nciclo++;
}
print("End of transfer. Received " + totalrecbytes + "File :" + filename + " Saved on " + tempFold);
Fs.Close();
}
catch (Exception ex)
{
//Ok here i return false, but i do some other stuff before
return false;
}
return true;
}
----END OF SERVER SIDE--------------
--------CLIENT SIDE--------------
.....DOING STUFFS....
//sw is the streamWriter, Sr the streamReader and the stream is the networkstream
System.Collections.Specialized.StringCollection formats = Clipboard.GetFileDropList();
sw.WriteLine("CLIENT:FILE:" + formats.Count + "*");
sw.Flush();
//Sending to the server filename and relative size
foreach (string filename in formats)
{
//Ok the * has sense in my protocol...ignore it.
sw.WriteLine((Directory.Exists(filename)) ? System.IO.Path.GetFileName(filename) + "*" : System.IO.Path.GetFileName(filename));
sw.Flush();
FileStream Fs = new FileStream(filename, FileMode.Open, FileAccess.Read);
sw.WriteLine(Fs.Length);
sw.Flush();
stream.Flush();
Fs.Close();
}
//Sending files
foreach (string filename in formats)
{
//client_sync is the class that wrap the TcpClient socket
client_sync.send_single_file(stream, filename);
resp = sr.ReadLine();
}
....DOING STUFF AND end of this function...
The send file function is defined in this way :
(note : i take this function from code project some weeks ago)
public void send_single_file(NetworkStream netstream, string filename)
{
//connected is a param of my class
if (!connected) return;
byte[] SendingBuffer = null;
try
{
FileStream Fs = new FileStream(filename, FileMode.Open, FileAccess.Read);
int NoOfPackets = Convert.ToInt32(Math.Ceiling(Convert.ToDouble(Fs.Length) / Convert.ToDouble(this.BufferSize)));
//NOTE: BUFFERSIZE IS 1024
int TotalLength = (int)Fs.Length, CurrentPacketLength = 0;
int bytes_send = 0;
for (int i = 0; i < NoOfPackets; i++)
{
if (TotalLength > this.BufferSize)
{
CurrentPacketLength = this.BufferSize;
TotalLength = TotalLength - CurrentPacketLength;
}
else
CurrentPacketLength = TotalLength;
SendingBuffer = new byte[CurrentPacketLength];
Fs.Read(SendingBuffer, 0, CurrentPacketLength);
netstream.Write(SendingBuffer, 0, SendingBuffer.Length);
bytes_send += CurrentPacketLength;
}
Fs.Close();
}
catch (Exception ex)
{
netstream.Close();
//my function
close_connection();
}
netstream.Flush();
}
---------END OF CLIENT SIDE------
So...someone can help me to escape from this hell??? THX :)
I am trying to encrypt and then decrypt an XML file using RSA and C# and while I'm really close, there's a problem. Once it's decrypted, almost all of the file is there but there's a hiccup toward the end. It's either a gap toward the end of the file or more data is appended to the very end of the file.
Here is my encrypt method:
public static bool Encrypt(ProcessingHolder ph)
{
FileInfo inFile = ph.encryptedFI;
FileInfo outFile = ph.unEncryptedFI;
X509Certificate2 daCert = new X509Certificate2(keyFP, daCertPassword);
RSACryptoServiceProvider RSA = (RSACryptoServiceProvider)daCert.PrivateKey;
bool done = false;
FileStream fs = null;
FileStream fso = null;
try
{
//opens the file to encrypt into a filestream object
fs = inFile.OpenRead();
//240 is what the iOS side is using
//algorithm that calculates max bytes ((KeySize - 384) / 8) + 37
//(returns 245)
int chunkSize = 245;
fso = outFile.OpenWrite();
byte[] buffer = new byte[chunkSize];
int totalRead = 0;
while (totalRead < fs.Length)
{
int readBytes = fs.Read(buffer,0, chunkSize);
totalRead += readBytes;
//check to see if the final chunk of data is less than 245 so as not to write empty buffer
if (readBytes < chunkSize) buffer = new byte[readBytes];
//byte[] encr = new byte[readBytes];
//actual encryption
//encr = RSA.Encrypt(buffer, false);
byte[] encr = RSA.Encrypt(buffer, false);
fso.Write(encr, 0, encr.Length);
}
fso.Flush();
fso.Close();
fs.Close();
done = true;
}
catch (Exception ex)
{
Debug.WriteLine("Decrypt failed with message " + ex.Message);
done = false;
}
finally
{
if (fs != null) fs.Close();
if (fso != null) fso.Close();
}
return done;
}
}
and here is my decrypt method:
public static bool Decrypt(ProcessingHolder ph)
{
FileInfo inFile = ph.encryptedFI;
FileInfo outFile = ph.unEncryptedFI;
X509Certificate2 daCert = new X509Certificate2(keyFP, daCertPassword);
RSACryptoServiceProvider RSA = (RSACryptoServiceProvider)daCert.PrivateKey;
bool done = false;
FileStream fs = null;
FileStream fso = null;
try
{
fs = inFile.OpenRead();
int chunkSize = 256;
fso = outFile.OpenWrite();
byte[] buffer = new byte[chunkSize];
int totalRead = 0;
while (totalRead < fs.Length)
{
int readBytes = fs.Read(buffer, 0, chunkSize);
totalRead += readBytes;
//check to see if the final chunk of data is less than 245 so as not to write empty buffer
//if (readBytes < chunkSize) buffer = new byte[readBytes];
byte[] decr = RSA.Decrypt(buffer, false);
fso.Write(decr, 0, decr.Length);
}
fso.Flush();
fso.Close();
fs.Close();
done = true;
}
catch (Exception ex)
{
Debug.WriteLine("Decrypt failed with message " + ex.Message);
done = false;
}
finally
{
if (fs != null) fs.Close();
if (fso != null) fso.Close();
}
return done;
}
banging my head against the wall here, thanks in advance
What happens during encrypting if the file is not a multiple of the length of the chunk size? Ie. a file 500 bytes long would read two sets of 245, but they have 10 bytes left over? This might be loosing the last few bytes at the end or adding extra values?
Maybe you need to add a header to the file with the size in bytes of the decrypted file so that the decrypter knows where to stop and a way to pad out the final block during encryption