So. Let's say I were to make a hex editor to edit... oh... let's say a .DLL file. How can I edit a .DLL file's hex by using C# or C++? And for the "fixed part", I want to make it so that I can browse from the program for a specific .DLL, have some pre-coded buttons on the programmed file, and when the button is pressed, it will automatically execute the requested action, meaning the button has been pre-coded to know what to look for in the .DLL and what to change it to. Can anyone help me get started on this?
Also, preferably C#. Thank you!
The basics are very simple.
A DLL, or any file, is a stream of bytes.
Basic file operations allow you to read and write arbitrary portions of a file. The term of art is basically "Random Access Files Operations".
In C, the fundamental operations are read(), write(), and lseek().
read allows you to read a stream of bytes in to a buffer, write allows you to write a buffers of bytes to a file, lseek allows you to position anywhere you want in the file.
Example:
int fd = open("test.dat", O_RDWR);
off_t offset = lseek(fd, 200, SEEK_SET);
if (off_t == -1) {
printf("Boom!\n");
exit(1);
}
char buf[1024];
ssize_t bytes_read = read(fd, buf, 1024);
offset = lseek(fd, 100, SEEK_SET);
ssize_t bytes_written = write(fd, buf, 1024);
flush(fd);
close(fd);
This reads 1024 bytes from a file, starting at the 200th byte of the file, then writes it back to the file at 100 bytes.
Once you can change random bytes in a file, it's a matter of choosing what bytes to change, how to change them, and doing the appropriate reads/lseeks/writes to make the changes.
Note, those are the most primitive I/O operations, there are likely much better ones you can use depending on your language etc. But they're all based on those primitives.
Interpreting the bytes of a file, displaying them, etc. That's an exercise for the reader. But those basic I/O capabilities give you the fundamentals of changing files.
If the idea is to load a hex edit box you can use the following: Be.HexEditor
Editing a file's "hex" is nothing more than changing bytes in it. The part of having pre-programmed changes is going to be that more general type. But for actually viewing, finding and then having the option of changing anything you want, Be.HexEditor is a good option. I used it over a year ago, I would hope that it has some new features that will make your life easier.
Related
I am trying to open and read a bunch of geo-referenced timelog files that are in binary format. They supposedly follow the ISO-11783 (ISOBUS) standard for agricultural machinery, but after reading 100s of pages of the standard I cannot figure out how to read the files either with a hex editor or programmatically with .NET c#. I know the timelog comes in file-pairs: an xml file and a binary file. The binary file, for example, is named TLG00004.bin and in notepad it looks like this (partial):
and when I open that file in Visual Studio 2015 (Community) as a binary file the hex looks like this:
which does not help me. I don't even know how to begin reading this as a byte stream in code (or anything else for that matter).
I know the file is supposed to look like this in human readable form:
(TimeStart, PositionNorth, PositionEast, PositionStatus, # DLV, DLV 0, PDV 0, DLV 1, PDV 1, DLV 2, PDV 2,...) it can have up to 255 DLV-PDV pairs which I believe are 32-bit integers. An example was shown as: (2005-05-02T16:32:00,51.00678,6.03489,1,2,0,10,1,15)
Little hints I have seen in the documentation indicate to me this must be utf-8 and perhaps base64 encoding with little endian and no Byte Order Mark. But I tried opening this in the free version of Hexinator and can't (human) read it using any of the dozens of encodings in that app, including utf-8, 16, 32...
I know this is not normal programming stuff but am throwing it out there to see if I'm lucky enough that someone has done this before and sees this. Any hints or resource-pointing would find me grateful, and I would be very thankful if someone can share any code that reads this kind of file.
Your data seems to follow the ISO 11783-10 standard for "Log data binary file structure" data exchange.
You will need to unpack your binary data into data types according to the specification. For example, the first 32 bits of the data are the milliseconds since midnight stored as a 32 bit unsigned integer. The next 16 bits are the days since 1980-01-01 stored as a 16 bit unsigned integer.
Unpacking binary data is programming language specific and some programming languages have useful libraries to assist in shifting through binary data.
As your question is about the general parsing of ISOBUS and I'm not proficient in your given language (C#), I can only give you an initial pointer.
BinaryReader looks to be the ideal way of unpacking a binary file by reading a number of bits from a stream and advancing the pointer through it:
using (BinaryReader reader = new BinaryReader(File.Open(fileName, FileMode.Open)))
{
milliSecondsSinceMidnight = reader.ReadUInt32();
daysSince1980 = reader.ReadUInt16();
}
If you need further help, you can now ask a specific question about byte parsing in C#.
This is a continuation of my question about downloading files in chunks. The explanation will be quite big, so I'll try to divide it to several parts.
1) What I tried to do?
I was creating a download manager for a Window-Phone application. First, I tried to solve the problem of downloading
large files (the explanation is in the previous question). No I want to add "resumable download" feature.
2) What I've already done.
At the current moment I have a well-working download manager, that allows to outflank the Windows Phone RAM limit.
The plot of this manager, is that it allows to download small chunks of file consequently, using HTTP Range header.
A fast explanation of how it works:
The file is downloaded in chunks of constant size. Let's call this size "delta". After the file chunk was downloaded,
it is saved to local storage (hard disk, on WP it's called Isolated Storage) in Append mode (so, the downloaded byte array is
always added to the end of the file). After downloading a single chunk the statement
if (mediaFileLength >= delta) // mediaFileLength is a length of downloaded chunk
is checked. If it's true, that
means, there's something left for download and this method is invoked recursively. Otherwise it means, that this chunk
was last, and there's nothing left to download.
3) What's the problem?
Until I used this logic at one-time downloads (By one-time I mean, when you start downloading file and wait until the download is finished)
that worked well. However, I decided, that I need "resume download" feature. So, the facts:
3.1) I know, that the file chunk size is a constant.
3.2) I know, when the file is completely downloaded or not. (that's a indirect result of my app logic,
won't weary you by explanation, just suppose, that this is a fact)
On the assumption of these two statements I can prove, that the number of downloaded chunks is equal to
(CurrentFileLength)/delta. Where CurrentFileLenght is a size of already downloaded file in bytes.
To resume downloading file I should simply set the required headers and invoke download method. That seems logic, isn't it? And I tried to implement it:
// Check file size
using (IsolatedStorageFileStream fileStream = isolatedStorageFile.OpenFile("SomewhereInTheIsolatedStorage", FileMode.Open, FileAccess.Read))
{
int currentFileSize = Convert.ToInt32(fileStream.Length);
int currentFileChunkIterator = currentFileSize / delta;
}
And what I see as a result? The downloaded file length is equal to 2432000 bytes (delta is 304160, Total file size is about 4,5 MB, we've downloaded only half of it). So the result is
approximately 7,995. (it's actually has long/int type, so it's 7 and should be 8 instead!) Why is this happening?
Simple math tells us, that the file length should be 2433280, so the given value is very close, but not equal.
Further investigations showed, that all values, given from the fileStream.Length are not accurate, but all are close.
Why is this happening? I don't know precisely, but perhaps, the .Length value is taken somewhere from file metadata.
Perhaps, such rounding is normal for this method. Perhaps, when the download was interrupted, the file wasn't saved totally...(no, that's real fantastic, it can't be)
So the problem is set - it's "How to determine number of the chunks downloaded". Question is how to solve it.
4) My thoughts about solving the problem.
My first thought was about using maths here. Set some epsilon-neiborhood and use it in currentFileChunkIterator = currentFileSize / delta; statement.
But that will demand us to remember about type I and type II errors (or false alarm and miss, if you don't like the statistics terms.) Perhaps, there's nothing left to download.
Also, I didn't checked, if the difference of the provided value and the true value is supposed to grow permanently
or there will be cyclical fluctuations. With the small sizes (about 4-5 MB) I've seen only growth, but that doesn't prove anything.
So, I'm asking for help here, as I don't like my solution.
5) What I would like to hear as answer:
What causes the difference between real value and received value?
Is there a way to receive a true value?
If not, is my solution good for this problem?
Are there other better solutions?
P.S. I won't set a Windows-Phone tag, because I'm not sure that this problem is OS-related. I used the Isolated Storage Tool
to check the size of downloaded file, and it showed me the same as the received value(I'm sorry about Russian language at screenshot):
I'm answering to your update:
This is my understanding so far: The length actually written to the file is more (rounded up to the next 1KiB) than you actually wrote to it. This causes your assumption of "file.Length == amount downloaded" to be wrong.
One solution would be to track this information separately. Create some meta-data structure (which can be persisted using the same storage mechanism) to accurately track which blocks have been downloaded, as well as the entire size of the file:
[DataContract] //< I forgot how serialization on the phone works, please forgive me if the tags differ
struct Metadata
{
[DataMember]
public int Length;
[DataMember]
public int NumBlocksDownloaded;
}
This would be enough to reconstruct which blocks have been downloaded and which have not, assuming that you keep downloading them in a consecutive fashion.
edit
Of course you would have to change your code from a simple append to moving the position of the stream to the correct block, before writing the data to the stream:
file.Position = currentBlock * delta;
file.Write(block, 0, block.Length);
Just as a possible bug. Dont forget to verify if the file was modified during requests. Specialy during long time between ones, that can occor on pause/resume.
The error could be big, like the file being modified to small size and your count getting "erronic", and the file being the same size but with modified contents, this will leave a corrupted file.
Have you heard an anecdote about a noob-programmer and 10 guru-programmers? Guru programmers were trying to find an error in his solution, and noob had already found it, but didn't tell about it, as it was something that stupid, we was afraid to be laughed at.
Why I remembered this? Because the situation is similar.
The explanation of my question was very heavy, and I decided not to mention some small aspects, that I was sure, worked correctly. (And they really worked correctly)
One of this small aspects, was the fact, that the downloaded file was encrypted via AES PKCS7 padding. Well, the decryption worked correctly, I knew it, so why should I mention it? And I didn't.
So, then I tried to find out, what exactly causes the error with the last chunk. The most credible version was about problems with buffering, and I tried to find, where am I leaving the missing bytes. I tested again and again, but I couldn't find them, as every chunk was saving without any losses. And one day I comprehended:
There is no spoon
There is no error.
What's the point of AES PKCS7? Well, the primary one is that it makes the decrypted file smaller. Not much, only at 16 bytes. And it was considered in my decryption method and download method, so there should be no problem, right?
But what happens, when the download process interrupts? The last chunk will save correctly, there will be no errors with buffering or other ones. And then we want to continue download. The number of the downloaded chunks will be equal to currentFileChunkIterator = currentFileSize / delta;
And here I should ask myself: "Why are you trying to do something THAT stupid?"
"Your downloaded one chunk size is not delta. Actually, it's less than delta". (the decryption makes chunk smaller to 16 bytes, remember?)
The delta itself consists of 10 equal parts, that are being decrypted. So we should divide not by delta, but by (delta - 16 * 10) which is (304160 - 160) = 304000.
I sense a rat here. Let's try to find out the number of the downloaded chunks:
2432000 / 304000 = 8. Wait... OH SHI~
So, that's the end of story.
The whole solution logic was right.
The only reason it failed, was my thought, that, for some reason, the downloaded decrypted file size should be the same as the sum of downloaded encrypted chunks.
And, of course, as I didn't mention about the decryption(it's mentioned only in previous question, which is only linked), none of you could give me a correct answer. I'm terribly sorry about that.
In continue to my comment..
The original file size as I understand from your description is 2432000 bytes.
The Chunk size is set to 304160 bytes (or 304160 per "delta").
So, the machine which send the file was able to fill 7 chunks and sent them.
The receiving machine now has 7 x 304160 bytes = 2129120 bytes.
The last chunk will not be filled to the end as there is not enough bytes left to fill to it.. so it will contain: 2432000 - 2129120 = 302880 which is less than 304160
If you add the numbers you will get 7x304160 + 1x302880 = 2432000 bytes
So according to that the original file transferred in full to the destination.
The problem is that you are calculating 8x304160 = 2433280 insisting that even the last chunk must be filled completely - but with what?? and why??
In humble.. are you locked in some kind of math confusion or did I misunderstand your problem?
Please answer, What is the original file size and what size is being received at the other end? (totals!)
C#.NET 4.0
I'm having an interesting problem here with reading a custom file archive format. In C#, I wrote a program that creates an archive header (some overhead info about the archive as a whole, number of files, those kinds of things). It then takes an input file to be stored, reads and bytes, and then writes some overhead about the file (filename, type, size and such) and then the actual file data. I can also extract files from the archive through this program. To test it, I stored a png image and extracted it by reading the filesize from the overhead and then allocating an array of bytes of that size, pulled the filedata into that array, and then wrote it with a streamwriter. No big deal, worked fine. Now, we go to the C++ side...
C++
My C++ program needs to read the filedata in, determine the filetype, and then pass it off to the appropriate processing class. The processing classes were giving errors, which they shouldn't have. So I decided to write the filedata out fro the C++ program after reading it using fwrite(), and the resulting file appears to be damaged? In a nutshell, this is the code being used to read the file...
unsigned char * data = 0;
char temp = 0;
__int64 fileSize = 0;
fread(&fileSize, sizeof(__int64), 1, _fileHandle);
data = new unsigned char[fileSize];
for (__int64 i = 0; i < fileSize; i++)
{
fread(&temp, 1, 1, _fileHandle);
data[i] = temp;
}
(I'm at work right now, so I just wrote this from memory. However, I'm 99% positive it's accurate to my code at home. I'm also not concerned with non MS Standards at the moment, so please bear with the __int64.)
I haven't gone through all 300 something thousand bytes to determine if everything is consistent, but the first 20 or so bytes that I looked at appear to be correct. I don't exactly see why there is a problem. Is there something funny about fread()? I also to double check the file in the archive, removed all the archive overhead and saved just the image data to a new png image with notepad, which worked fine.
Should I be reading this differently? Is there something wrong with using fread() to read in this data?
Given that the first n bytes appear to be correct, did you by chance forget to open the file in binary mode ("rb")? If you didn't then it's helpfully converting any sequences of \r\n into \n for you which would obviously not be what you want.
Since this question is tagged C++ did you consider using the canonical C++ approach of iostreams rather than the somewhat antiquated FILE* streams from C?
I am curious as to how applications generate their own data that is used with the application itself. For example, if you take any kind of PC game's save file or some sort of program that generates binary data like Photoshop's PSD files or .torrent files for BitTorrent applications, I'd assume they are all specific to the corresponding application and that the authors of that application programmed the way this data was created. My first question is: is that true? I am 99% positive that it is binary data because when opening a PSD file or a .torrent file in Notepad++, it's easy to see that it's nothing that can be read by a human...
My second question is: if I wanted to make an application that generates its own data in binary format (no plain-text or anything that's easily manipulated), how would I go about handling this data? I can vaguely picture generating this data and saving it to a file in binary format, but I am really stuck on how I'd handle this data when it's needed by the application again. Since this type of data is not plain text and can't be treated as a string or anything like that, how is it that applications create and handle/parse their own binary data (or any binary data in general)?
I can obviously see that when you open a PSD file, Photoshop opens and it displays whatever the PSD file contained. But how do many applications handle these formats? I am just not seeing how to parse this specific data (or binary data in general) and programmatically do what you want to with it.
Well, as a simple example, let's take bitmaps.
Bitmaps have a standard file structure, which is defined by the info header and file header.
On the wikipedia article (link: http://en.wikipedia.org/wiki/BMP_file_format) you'll see that the info header has a well defined format, as well as the file header.
Each of these is written as binary as is, and is read in as binary as is. Then, the actually bitmap image is written out as binary.
In other applications, the application may choose to do a custom plain text format, in which case it must be written to in a consistent manner or have some support for versioning so you can use newer features in the file.
Look up on serialization though, it's a rather broad topic and there are lots of approaches to this.
Edit: Here is a code sample (not optimal) for reading (or writing, with the right modifications) in bitmaps:
// Tell visual studio to align on 2-byte boundary
// Necessary so if you write to file, it only writes 14 bytes and not 16.
#pragma pack(2)
struct BMIH
{
short bfType;
long bfSize;
short bfReserved0;
short bfReserved1;
long bOffbits;
};
#pragma pack(8)
struct BMFH
{
long biSize;
long biWidth;
long biHeight;
short biPlanes;
short biBitCount;
long biCompression;
long biImageSize;
long biXPelsPerMeter;
long biYPelsPerMeter;
long biClrUsed;
long biClrImportant;
};
BMIH infoheader;
BMFH fileheader;
std::fstream file(filename.c_str(), std::ios::in | std::ios::binary);
// Read in info and file headers
file.read((char *) &infoheader, sizeof(infoheader));
file.read((char *) &fileheader, sizeof(fileheader));
// Calculate size of image
int size = fileheader.biHeight * fileheader.biWidth;
int bytes = size * fileheader.biBitCount / 8;
// Read in the image to a buffer
unsigned char data = new unsigned char[bytes];
file.read((char *) td.data, bytes);
file.close();
That code is actually a drastic simplification and completely ignores all sorts of issues, such as what happens if the file headers or data are corrupt, if the file isn't incomplete, etc. But it's just meant as a proof of concept. The #pragmas are actually visual studio specific for enforcing proper alignment of the headers.
When we write this out to a file, we might not actually say "Okay, now write out this integer". Instead, we want to write it as a binary format. For example, code that you might (but shouldn't) use to write it would look like:
// Assume for arguments sake these data structures came pre-filled
BMFH fileheader;
BMIH infoheader;
unsigned char *data;
int size = fileheader.biHeight * fileheader.biWidth;
int bytes = size * fileheader.biBitCount / 8;
std::fstream file("MyImage.bitmap", std::ios::out | std::ios::binary);
file.write((char *) &infoheader, sizeof(BMIH));
file.write((char *) &fileheader, sizeof(BMFH));
file.write((char *) data, sizeof(unsigned char) * bytes);
Read up on Binary Serialization on MSDN. The .Net Framework goes a long way to helping with this.
Yes, Many applications leverage some sort of application-specific binary formats that can not be easily manipulated. To create your own binary format, there are some options:
Binary Serialization Technique
Using IO classes to manually read and write bytes and actually creating a random access file.
Today i'm cutting video at work (yea me!), and I came across a strange video format, an MOD file format with an companion MOI file.
I found this article online from the wiki, and I wanted to write a file format handler, but I'm not sure how to begin.
I want to write a file format handler to read the information files, has anyone ever done this and how would I begin?
Edit:
Thanks for all the suggestions, I'm going to attempt this tonight, and I'll let you know. The MOI files are not very large, maybe 5KB in size at most (I don't have them in front of me).
You're in luck in that the MOI format at least spells out the file definition. All you need to do is read in the file and interpret the results based on the file definition.
Following the definition, you should be able to create a class that could read and interpret a file which returns all of the file format definitions as properties in their respective types.
Reading the file requires opening the file and generally reading it on a byte-by-byte progression, such as:
using(FileStream fs = File.OpenRead(path-to-your-file)) {
while(true) {
int b = fs.ReadByte();
if(b == -1) {
break;
}
//Interpret byte or bytes here....
}
}
Per the wiki article's referenced PDF, it looks like someone already reverse engineered the format. From the PDF, here's the first entry in the format:
Hex-Address: 0x00
Data Type: 2 Byte ASCII
Value (Hex): "V6"
Meaning: Version
So, a simplistic implementation could pull the first 2 bytes of data from the file stream and convert to ASCII, which would provide a property value for the Version.
Next entry in the format definition:
Hex-Address: 0x02
Data Type: 4 Byte Unsigned Integer
Value (Hex):
Meaning: Total size of MOI-file
Interpreting the next 4 bytes and converting to an unsigned int would provide a property value for the MOI file size.
Hope this helps.
If the files are very large and just need to be streamed in, I would create a new reader object that uses an unmanagedmemorystream to read the information in.
I've done a lot of different file format processing like this. More recently, I've taken to making a lot of my readers more functional where reading tends to use 'yield return' to return read only objects from the file.
However, it all depends on what you want to do. If you are trying to create a general purpose format for use in other applications or create an API, you probably want to conform to an existing standard. If however you just want to get data into your own application, you are free to do it however you want. You could use a binaryreader on the stream and construct the information you need within your app, or get the reader to return objects representing the contents of the file.
The one thing I would recommend. Make sure it implements IDisposable and you wrap it in a using!