Unusual Character addition after writing back decoded file - c#

I am using ZXing.Net library to encode and decode my video file using RS Encoder. It works well by adding and and removing parity after encoding and decoding respectively. But When writing decoded file back it is adding "?" characters in file on different locations which was not part of original file. I am not getting why this problem is occurring when writing file back.
Here is my code
using ZXing.Common.ReedSolomon;
namespace zxingtest
{
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
string inputFileName = #"D:\JM\bin\baseline_30.264";
string outputFileName = #"D:\JM\bin\baseline_encoded.264";
string Content = File.ReadAllText(inputFileName, ASCIIEncoding.Default);
//File.WriteAllText(outputFileName, Content, ASCIIEncoding.Default);
ReedSolomonEncoder enc = new ReedSolomonEncoder(GenericGF.AZTEC_DATA_12);
ReedSolomonDecoder dec = new ReedSolomonDecoder(GenericGF.AZTEC_DATA_12);
//string s = "1,2,4,6,1,7,4,0,0";
//int[] array = s.Split(',').Select(str => int.Parse(str)).ToArray();
int parity = 10;
List<byte> toBytes = ASCIIEncoding.Default.GetBytes(Content.Substring(0, 500)).ToList();
for (int index = 0; index < parity; index++)
{
toBytes.Add(0);
}
int[] bytesAsInts = Array.ConvertAll(toBytes.ToArray(), c => (int)c);
enc.encode(bytesAsInts, parity);
bytesAsInts[1] = 3;
dec.decode(bytesAsInts, parity);
string st = new string(Array.ConvertAll(bytesAsInts.ToArray(), z => (char)z));
File.WriteAllText(outputFileName, st, ASCIIEncoding.Default);
}
}
}
And here is the Hex file view of H.264 bit stream


The problem is that you're handling a binary format as if it is a Text file with an encoding. But based on what you are doing you only seem to be interested in reading some bytes, process them (encode, decode) and then write the bytes back to a file.
If that is what you need then use the proper reader and writer for your files, in this case the BinaryReader and BinaryWriter. Using your code as a starting point this is my version using the earlier mentioned readers/writers. My inputfile and outputfile are similar for the bytes read and written.
string inputFileName = #"input.264";
string outputFileName = #"output.264";
ReedSolomonEncoder enc = new ReedSolomonEncoder(GenericGF.AZTEC_DATA_12);
ReedSolomonDecoder dec = new ReedSolomonDecoder(GenericGF.AZTEC_DATA_12);
const int parity = 10;
// open a file as stream for reading
using (var input = File.OpenRead(inputFileName))
{
const int max_ints = 256;
int[] bytesAsInts = new int[max_ints];
// use a binary reader
using (var binary = new BinaryReader(input))
{
for (int i = 0; i < max_ints - parity; i++)
{
//read a single byte, store them in the array of ints
bytesAsInts[i] = binary.ReadByte();
}
// parity
for (int i = max_ints - parity; i < max_ints; i++)
{
bytesAsInts[i] = 0;
}
enc.encode(bytesAsInts, parity);
bytesAsInts[1] = 3;
dec.decode(bytesAsInts, parity);
// create a stream for writing
using(var output = File.Create(outputFileName))
{
// write bytes back
using(var writer = new BinaryWriter(output))
{
foreach(var value in bytesAsInts)
{
// we need to write back a byte
// not an int so cast it
writer.Write((byte)value);
}
}
}
}
}

Related

Convert RepeatedField<ByteString> to Byte[]

Defined a gRPC Python service in a Docker.
The services in my PROTO file:
rpc ClientCommand(ClientRequest) returns (ClientResponse){}
Definition of "ClientResponse":
message ClientResponse{
int32 request_id = 1;
repeated int32 prediction_status = 2;
repeated string prediction_info = 3;
repeated string prediction_error = 4;
repeated string prediction_result_name = 5;
repeated bytes prediction_result = 6;
repeated bytes prediction_config = 7;
repeated bytes prediction_log = 8;
}
On client side, I want to catch the repeated bytes and convert them into a file(I know it better works with a stream but for the moment I would like to do it like this).
The repeated strings and integer I can perfectly convert to List --> OK
The repeated bytes I would like to convert to Byte[]. Their type: Google.ProtoBuf.Collections.RepeatedField<Google.ProtoBuf.ByteString>.
At first it seems to be impossible to convert this type to a Byte[]. Could somebody help me with this please? My solution temporary:
byte[] test = new byte[100];
Google.Protobuf.ByteString[] test2 = new Google.Protobuf.ByteString[100];
response.PredictionResult.CopyTo(test2,0);
test2.CopyTo(test,0);
WriteFile(#"C:\programs\file.txt", test);

Meanwhile I figure it out:
public void WriteFileResult(string fileName,Google.Protobuf.Collections.RepeatedField<ByteString> data)
{
byte[] bytes = new byte[data.Count];
ByteString[] dataByteString = new ByteString[data.Count];
string path = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);
if (!path.EndsWith(#"\")) path += #"\";
if (File.Exists(Path.Combine(path, fileName)))
File.Delete(Path.Combine(path, fileName));
using (FileStream fs = new FileStream(Path.Combine(path, fileName), FileMode.CreateNew, FileAccess.Write))
{
data.CopyTo(dataByteString, 0);
bytes = dataByteString[0].ToByteArray();
fs.Write(bytes, 0, (int)bytes.Length);
_serviceResponseModel.Json = false;
//fs.Close()
}
}

Splitting of text file not working properly in c#

I have requirement of writing to text file.
If the file size exceeds 700MB, create new file & write to it.
I am currently writing data with “|” delimited from database to file & after that check the file size & splitting into multiple files, but the file splits in middle of the line.
It should write till end of line or start that particular line in new file .
I need to write the column names in the first line in the newly splited file.
I am new to c#, could you please suggest me the solution with the sample code.
Please find below code to splitting the file
private static void ReadWriteToFile(string fileNames)
{
string sourceFileName = fileNames;
string destFileLocation = Path.GetDirectoryName(fileNames);
int index = 0;
long maxFileSize = 700 * 1024 * 1024;
byte[] buffer = new byte[65536];
using (Stream source = File.OpenRead(sourceFileName))
{
while (source.Position < source.Length)
{
index++;
string newFileName = Path.Combine(destFileLocation, Path.GetFileNameWithoutExtension(sourceFileName));
newFileName += index.ToString() + Path.GetExtension(sourceFileName);
using (Stream destination = File.OpenWrite(newFileName))
{
while (destination.Position < maxFileSize)
{
int bytes = source.Read(buffer, 0, (int)Math.Min(maxFileSize, buffer.Length));
destination.Write(buffer, 0, bytes);
if (bytes < Math.Min(maxFileSize, buffer.Length))
{
break;
}
}
}
}
}
}
Thanks in advance.
Could you please let me know if there is any alternative best way to do this

Try this, a rewrite of a line file splitter i wrote in my beginning c# times.
(You only have to add the column header as a string in the beginning of a new file.)
private static void SplitAfterMBytes(int splitAfterMBytes, string filename)
{
// Variable for max. file size.
var maxFileSize = splitAfterMBytes * 1048576;
int fileCount = 0;
long byteCount = 0;
StreamWriter writer = null;
try
{
var inputFile = new FileInfo(filename);
var index = filename.LastIndexOf('.');
//get only the name of the file.
var fileStart = filename.Substring(0, index);
// get the file extension
var fileExtension = inputFile.Extension;
// generate a new file name.
var outputFile = fileStart + '_' + fileCount++ + fileExtension;
// file format is like: QS_201101_321.txt.
writer = new StreamWriter(outputFile);
using (var reader = new StreamReader(filename))
{
for (string str; (str = reader.ReadLine()) != null;)
{
byteCount = byteCount + System.Text.Encoding.Unicode.GetByteCount(str);
if (byteCount >= maxFileSize)
{
// max number of bytes reached
// write into the old file, without Newline,
// so that no extra line is written.
writer.Write(str);
// 1. close the actual file.
writer.Close();
// 2. open a new file with number incresed by 1.
outputFile = fileStart + '_' + fileCount++ + fileExtension;
writer = new StreamWriter(outputFile);
byteCount = 0; //reset the counter.
}
else
{
// Write into the old file.
// Use a Linefeed, because Write works without LF.
// like Java ;)
writer.Write(str);
writer.Write(writer.NewLine);
}
}
}
}
catch (Exception ex)
{
// do something useful, like: Console.WriteLine(ex.Message);
}
finally
{
writer.Dispose();
}
}

Decompressing GZIP stream

I am trying to decompress a GZipped string which is part of response from a webservice. The string that I have is:
"[31,-117,8,0,0,0,0,0,0,0,109,-114,65,11,-62,48,12,-123,-1,75,-50,-61,-42,-127,30,122,21,111,-126,94,60,-119,-108,-72,102,44,-48,-75,-93,-21,100,56,-6,-33,-19,20,20,101,57,37,95,-14,94,-34,4,-63,-5,-72,-73,-44,-110,-117,-96,38,-88,26,-74,38,-112,3,117,-7,25,-82,5,24,-116,56,-97,-44,108,-23,28,24,-44,-85,83,34,-41,97,-88,24,-99,23,36,124,-120,94,99,-120,15,-42,-91,-108,91,45,-11,70,119,60,-110,21,-20,12,-115,-94,111,-80,-93,89,-41,-65,-127,-82,76,41,51,-19,52,90,-5,69,-85,76,-96,-128,64,22,35,-33,-23,-124,-79,-55,-1,-2,-10,-87,0,55,-76,55,10,-57,122,-9,73,42,-45,98,-44,5,-77,101,-3,58,-91,39,38,51,-15,121,21,1,0,0]"
I'm trying to decompress that string using the following method:
public static string UnZip(string value)
{
// Removing brackets from string
value = value.TrimStart('[');
value = value.TrimEnd(']');
//Transform string into byte[]
string[] strArray = value.Split(',');
byte[] byteArray = new byte[strArray.Length];
for (int i = 0; i < strArray.Length; i++)
{
if (strArray[i][0] != '-')
byteArray[i] = Convert.ToByte(strArray[i]);
else
{
int val = Convert.ToInt16(strArray[i]);
byteArray[i] = (byte)(val + 256);
}
}
//Prepare for decompress
System.IO.MemoryStream ms = new System.IO.MemoryStream(byteArray);
System.IO.Compression.GZipStream sr = new System.IO.Compression.GZipStream(ms,
System.IO.Compression.CompressionMode.Decompress);
//Reset variable to collect uncompressed result
byteArray = new byte[byteArray.Length];
//Decompress
int rByte = sr.Read(byteArray, 0, byteArray.Length);
//Transform byte[] unzip data to string
System.Text.StringBuilder sB = new System.Text.StringBuilder(rByte);
//Read the number of bytes GZipStream red and do not a for each bytes in
//resultByteArray;
for (int i = 0; i < rByte; i++)
{
sB.Append((char)byteArray[i]);
}
sr.Close();
ms.Close();
sr.Dispose();
ms.Dispose();
return sB.ToString();
}
The method is a modified version of the one in the following link:
http://www.codeproject.com/Articles/27203/GZipStream-Compress-Decompress-a-string
Sadly, the result of that method is a corrupted string. More specifically, I know that the input string contains a compressed JSON object and the output string has only some of the expected string:
"{\"rootElement\":{\"children\":[{\"children\":[],\"data\":{\"fileUri\":\"file:////Luciano/e/orto_artzi_2006_0_5_pixel/index/shapefiles/index_cd20/shp_all/index_cd2.shp\",\"relativePath\":\"/i"
Any idea what could be the problem and how to solve it?

Try
public static string UnZip(string value)
{
// Removing brackets from string
value = value.TrimStart('[');
value = value.TrimEnd(']');
//Transform string into byte[]
string[] strArray = value.Split(',');
byte[] byteArray = new byte[strArray.Length];
for (int i = 0; i < strArray.Length; i++)
{
byteArray[i] = unchecked((byte)Convert.ToSByte(strArray[i]));
}
//Prepare for decompress
using (System.IO.MemoryStream output = new System.IO.MemoryStream())
{
using (System.IO.MemoryStream ms = new System.IO.MemoryStream(byteArray))
using (System.IO.Compression.GZipStream sr = new System.IO.Compression.GZipStream(ms, System.IO.Compression.CompressionMode.Decompress))
{
sr.CopyTo(output);
}
string str = Encoding.UTF8.GetString(output.GetBuffer(), 0, (int)output.Length);
return str;
}
}
The MemoryBuffer() doesn't "duplicate" the byteArray but is directly backed by it, so you can't reuse the byteArray.
I'll add that I find funny that they "compressed" a json of 277 characters to a stringized byte array of 620 characters.
As a sidenote, the memory occupation of this method is out-of-the-roof... The 620 character string (that in truth is a 277 byte array) to be decompressed causes the creation of strings/arrays for a total size of 4887 bytes (including the 620 initial character string) (disclaimer: the GC can reclaim part of this memory during the execution of the method). This is ok for byte arrays of 277 bytes... But for bigger ones the memory occupation will become quite big.

Following on from Xanatos's answer in C# slightly modified to return a simple byte array. This takes a gzip compressed byte array and returns the inflated gunzipped array.
public static byte[] Decompress(byte[] compressed_data)
{
var outputStream = new MemoryStream();
using (var compressedStream = new MemoryStream(compressed_data))
using (System.IO.Compression.GZipStream sr = new System.IO.Compression.GZipStream(
compressedStream, System.IO.Compression.CompressionMode.Decompress))
{
sr.CopyTo(outputStream);
outputStream.Position = 0;
return outputStream.ToArray();
}
}

serial port readline method reads strange characters

I'm working on a project to transfer files between two COM ports.
First , I'm taking file name and extension and size before I convert the file to a byte array and send it to the second COM.
the problem is that I get strange characters in the beginning of the first readline method where I'm sending file name, like this :
"\0R\0\0\0\0\0\b\0\0\0S\0BAlpha" // file name
".docx" // file extension
"11360" // file size
here is the code I'm using to send the files :
Send sfile = new Send();
string path = System.IO.Path.GetFullPath(op.FileName);
sfile.Bytes = File.ReadAllBytes(path);
int size = sfile.Bytes.Length;
sfile.FileName = System.IO.Path.GetFileNameWithoutExtension(path);
sfile.Extension = System.IO.Path.GetExtension(path);
FileStream fs = new FileStream(path,FileMode.Open);
BinaryReaderbr = new BinaryReader(fs);
serialPort1.WriteLine(sfile.FileName); // sending file name
serialPort1.WriteLine(sfile.Extension);// sending extension
serialPort1.WriteLine(size.ToString());// sending size
byte[] b1 = br.ReadBytes((int)fs.Length);
for (int i = 0; i <= b1.Length; i++)
{
serialPort1.Write(b1, 0, b1.Length);
}
br.Close();
fs.Dispose();
fs.Close();
serialPort1.Close();
and the code below is used to receive data being sent :
string path1 = serialPort1.ReadLine();
string path2 = serialPort1.ReadLine();
string path3 = serialPort1.ReadLine();
int size = Convert.ToInt32(path3);
string path0 = #"C:\";
string fullPath = path0 + path1 + path2;
// File.Create(fullPath);
FileStream fs = new FileStream(fullPath, FileMode.Create);
byte[] b1 = new byte[size];
for (int i = 0; i < b1.Length; i++)
{
serialPort1.Read(b1, 0, b1.Length);
}
fs.Write(b1, 0, b1.Length);
fs.Close();
serialPort1.Close();

You are not writing the bytes correctly. It should be:
byte[] b1 = br.ReadBytes((int)fs.Length);
serialPort1.Write(b1, 0, b1.Length);
The way you read them is completely wrong as well. It should be:
byte[] b1 = new byte[size];
for (int i = 0; i < b1.Length; )
{
i += serialPort1.Read(b1, i, b1.Length - i);
}

Open SQLite3 file as string c#

I am tring to open a SQLite format 3 file as a string in C#
string fileName = #"C:\Users\Test\Downloads\hello.sql";
using (BinaryReader b = new BinaryReader(File.Open(fileName, FileMode.Open),
Encoding.GetEncoding(1252)))
{
int pos = 0;
int length = (int)b.BaseStream.Length;
while (pos < length)
{
int v = b.ReadInt32();
textBox1.Text += (v);
pos += sizeof(int);
}
}
This is the code I have got so far, but when I try open it, it freezes and wont let you do anything, as if its on a constant loop trying to open it.
If you are still confused on what I am trying to do. If you where to right click on the file and open it with notepad or notepad++ you can see the file contents(String) and thats what I want to be displayed in the String or textbox in the program.
P.S Im not wanting to open it in a SQLite format, I want to open it as text

I'm not sure why you want to do this, but this method is efficient:
string fileName = #"C:\Users\Test\Downloads\hello.sql";
textBox1.Text = File.ReadAllText(filename, Encoding.GetEncoding(1252));
or if you want to see the file contents as byte values, you can use:
byte[] buffer = File.ReadAllBytes(filename);
textBox1.Text = BitConverter.ToString(buffer);
Edit:
This code works with winforms textboxes. Instead of just checking for 0x00 I check for all control characters using .NET's char.IsControl function and replace them.
string fileName = #"C:\Users\Test\Downloads\hello.sql";
byte[] buffer = File.ReadAllBytes(fileName);
Encoding enc = Encoding.GetEncoding(1252);
char[] chars = enc.GetChars(buffer);
for (int n = 0; n < chars.Length; n++)
{
if (char.IsControl(chars[n])) chars[n] = '.';
}
textBox1.Text = new string(chars);

Categories

Resources