I'm writing some code using PacketDotNet and SharpPCap to parse H.225 packets for a VOIP phone system. I've been using Wireshark to look at the structure, but I'm stuck. I've been using This as a reference.
Most of the H.225 packets I see are user information type with an empty message body and the actual information apparently shows up as a list of NonStandardControls in Wireshark. I thought I'd just extract out these controls and parse them later, but I don't really know where they start.
In almost all cases, the items start at the 10th byte of the H.225 data. Each item appears to begin with the length which is recorded as 2 bytes. However, I am getting a packet that has items starting at the 11th byte.
The only difference I see in this packet is something in the message body supposedly called open type length which has a value of 1, whereas the rest all appear to be 0. Would the items start at 10 + open type length? Is there some document that explains what this open type length is for?
Thanks.
H.225 doesn't use a fixed length encoding, it user ASN.1 PER encoding (not BER).
You probably won't find a C# library. OPAL is adding a C API if you are able to use that.
Related
I'm new to Pcapng files. I've read the 40+ page whitepaper and I'm still scratching my head and sweating. I understand that the Pcapng file is:
Made up of a Section Header Block - This is the start of every Pcapng file.
Question 1: How large is this?
It appears that it's BlockType (4 Bytes) + BlockTotalLength (4 bytes) + Byte Order Magic (4 Bytes) + Mahor and Minor Version (4 bytes total, 2 bytes each) + Section Length (4 bytes) + Options (Variable) + Block Total length (again, 4 bytes).
If I'm building a parser, how would I know how many bytes I need to skip to arrive at my first data frame block?
Question 2: Where is the data stored? By data I mean the entire frame that contains Ethernet, IP, and TCP Data, as shown in the picture below (Figure 1).
The documentation states that:
A section includes data delimited by two section header blocks.
When doing a manual inspection (yes, I went byte by byte over a file to see how many bytes lie in between two frames :'( ), I noticed there were 35 bytes in between each message (each message shown on wireshark had 35 bytes in between). Are these bytes related to a pcapng block?
Once I understand how to get to the first tcp frame, and how many bytes I need to skip to get to the next, I can build my parser.
I'm willing to send Bitcoin/Monero to anyone who can help me understand how I can best parse these pcapng messages. Thanks!
If I'm building a parser, how would I know how many bytes I need to skip to arrive at my first data frame block?
That's not how you do it.
If you're building a parser, note that a parser must look at more than just the first data frame block.
First of all, it must look at the Section Header Block (SHB), to determine the byte order of the data in all the subsequent blocks by looking at the Byte-Order Magic field.
After that, you need to look at all subsequent blocks, looking for Interface Description Blocks and Enhanced Packet Blocks (EPBs), Simple Packet Blocks (SPBs), and possibly Packet Blocks (PBs) (those are obsolete, so no program should write them, but programs should be prepared to read them). Each EPB or PB has an interface ID that refers to an IDB, which must have appeared before the EPB or PB in question; an SPB implicitly refers to the first IDB, which, again, must have appeared before the SPB in question.
The format of the packet data in an EPB, SPB, or PB depends on the link-layer type specified by the IDB to which it refers, so you need to have read the IDB in question.
And, as the above indicates, there is no fixed number of bytes between the SHB and the first EPB, SPB, or PB, so there is no simple fixed number of bytes to skip to get to the first data frame block. For one thing, there's a variable number of bytes, which you can only determine by reading all the blocks before the first EPB, SPB, or PB. For another thing, you can't skip them, you have to read them to get enough information to interpret the packet data in them.
Where is the data stored? By data I mean the entire frame that contains Ethernet, IP, and TCP Data, as shown in the picture below (Figure 1).
It's stored in EPBs, SPBs, or PBs. See the descriptions of those three block types; frames are in the "Packet Data" fields of those blocks.
So I'm at my Interface Description Block and the 64 bit number that contains both a Timestamp Resolution of 9 (10^-9, Nanoseconds?) and 6 (10^-6, Microseconds).
As Christopher Maynard indicated, the 9 isn't a timestamp resolution, it's an option type. Pcapng blocks have both fixed information at the beginning and options; an option begins with an option type and option value length, followed by the option data. An IDB if_tsresol option has
2 bytes of option type, with the value 9;
2 bytes of option value length, with the value 1;
1 byte of option value, with the value as specified in the description of that option.
A value of 6 means the time stamp resolution is 1/10^6 of a second, which means 1 microsecond.
I think #tee-zad-awk found an answer that helped over at https://ask.wireshark.org/question/15159/how-can-i-display-as-much-pcapng-information-as-possible/, but for the benefit of anyone else looking for an answer to this question, I've linked it here and have provided my answer below, just in case the link is ever broken someday.
It seems that, after reading the 40 page whitepaper on Pcapng ...
The current PCAP Next Generation (pcapng) Capture File Format draft document is 52 pages, so perhaps you're not looking at the most recent version? Other versions do exist, such as those at https://datatracker.ietf.org/doc/html/draft-tuexen-opswg-pcapng-00, https://pcapng.github.io/pcapng/ or https://www.tcpdump.org/pcap/pcap.html and probably others, but they're all obsolete.
If you're looking for a pcapng parser to help you decipher the file, then look no further than Wireshark itself. If you've loaded a pcapng file into Wireshark, you can use "View -> Reload as File Format/Capture" (Ctrl+Shift+F) to cause Wireshark to load and display the raw file contents itself rather than to load and display the packets from the file. This should cause you to be able to see the various pcapng blocks and be able to drill down into them. For example:
Frame 1: 184 bytes on wire (1472 bits), 184 bytes captured (1472 bits)
MIME file
PCAPNG File Format
Block: Section Header Block 1
Block: Interface Description Block 0
Block: Enhanced Packet Block 1
You can also have a look at the Wireshark source code, such as the epan/dissectors/file-pcapng.c and wiretap/pcapng.c files.
By the way, if you're looking to support all extensions, the Wireshark [PcapNg wiki page] (https://wiki.wireshark.org/Development/PcapNg) has a link to Augmented PCAP Next Generation Dump File Format page that you might also want to take a look at. I don't know how many other extensions may have been implemented but not included in the main pcapng file format specification, but hopefully not many, as this could quickly become problematic with different projects possibly using the same block type for different blocks. That practice should be highly discouraged.
In order to find it out, it is helpful to read the specifications of the protocol of the network device and the package that has been sent. For example, we need to know the frame description of an Ethernet device and the package description of a TCP/IP package in order to understand the raw data. Having studied this, we record some traffic in Wireshark and select a block in the upper window of Wireshark. The middle window will tell you in clear text what Wireshark has received. On clicking on any of the lines in the middle window, Wireshark will mark the bytes of the raw data in the lower window that bear the information of the clicked line. Also, you can click on the raw data and then the clear text is marked. Moreover, the status line also informs you about it. This is very helpful for understanding the data.
I needed to read the TCP / IPv4 packages of Ethernet traffic. The block starts with the identification block type = 0x00000006 and the length of the block. The device was Ethernet so that I had the link type LINKTYPE_ETHERNET. The section length can be taken from byte 16-23. The other entries of the block header can be taken from here.
After the block header or after 28 bytes , the Ethernet frame came with the following entries (see here for a description):
mac address destination, 6 bytes
mac address source, 6 bytes
type: 0x0800 for IPv4, 0x0806 for ARP, 0x86DD for IPv6, 0x8100 for the presence of an IEEE 802.1Q tag.
For an IPv4 package or type = 0x0800, the following bytes are the IPv4 header (see here for a description):
IP version and header length, 1 byte
differentiated services field, 1 byte
total length, 2 bytes
identification, 2 bytes
flags, 1 byte
fragment offset, 1 byte
time to live, 1 byte
protocol with 0x06 for TCP, 1 byte
header checksum, 2 bytes
source IP address
destination IP address
options
The total length is very important: the byte that follows the last byte of the IPv4 + TCP package is located at total length bytes after the entry IP version and header length. However, this entry can be tricky. I head an entry with length 0 though the IP header length already had 20 bytes. In this case, Wireshark was helpful. It reported
[Total Length: 1547 bytes (reported as 0, presumed to be because of "TCP segmentation offload" (TSO))]
A detailed description of this phenomenon can be taken from here. In this case I could compute the payload length by the section length from above minus the length of the Ethernet frame (14 bytes) minus the length of the IP Header minus the length of the TCP header. However, padding problems could arise though I did not have these problems. A padding problem occurs when the package length is extended to a multiple of 4 bytes or something.
If the protocol of the IPv4 header is 0x06, the TCP package follows. The details of an TCP package can be taken from here. Of course, Wireshark also helps you interpreting the TCP package: just click on the lines in the middle window that belong to the TCP package or click on the raw data.
As outlined here, the interpretation of a pcapng file has many ifs and whens.
I have a program that work like a chat.
Client and server are connected with 2 TCP sockets, one for incoming messages another for outgoing messages.
Sometimes the messages can be very big (ex. 2 MByte of text) so I want to compress them before sending over the channel.
The problem is that I don't know how to find the start and end of compressed message.
Now I use two special characters to find start and end of message but with compression there can be errors.
There is maybe a type of compression that don't use some specific bytes?
I use C# to open and manage sockets so I need a compression that work under windows.
Append to start of message it length. After that you just need to read length, and after that get exactly count of bytes what you need.
It will looks like:
|length|data|..|..|length|data|..|..|..|
And more exactly
|3|26|125|36|4|12|45|16|34|
Where 3 and 4 are length.
You just need an escaping scheme.
Send STX (Start of Transmission) at the start.
Send ETX (End of Transmission) at the end.
If an STX or ETX appears in the data, prefix it with ESC (escape).
If an ESC appears in the data, prefix it with ESC.
At the receiver:
The first byte should be STX, otherwise you have a bug. Discard it.
After that, if a byte is ESC, discard it and accept the next byte, whatever it is.
Otherwise, if the next byte is ETX, discard it and stop reading.
The problem with the length-word prefix suggested in another answer is that you can't know the length without doing the compression first, which costs time and space.
I am using a networkstream to pass short strings around the network.
Now, on the receiving side I have encountered an issue:
Normally I would do the reading like this
see if data is available at all
get count of data available
read that many bytes into a buffer
convert buffer content to string.
In code that assumes all offered methods work as probably intended, that would look something like this:
NetworkStream stream = someTcpClient.GetStream();
while(!stream.DataAvailable)
;
byte[] bufferByte;
stream.Read(bufferByte, 0, stream.Lenght);
AsciiEncoding enc = new AsciiEncoding();
string result = enc.GetString(bufferByte);
However, MSDN says that NetworkStream.Length is not really implemented and will always throw an Exception when called.
Since the incoming data are of varying length I cannot hard-code the count of bytes to expect (which would also be a case of the magic-number antipattern).
Question:
If I cannot get an accurate count of the number of bytes available for reading, then how can I read from the stream properly, without risking all sorts of exceptions within NetworkStream.Read?
EDIT:
Although the provided answer leads to a better overall code I still want to share another option that I came across:
TCPClient.Available gives the bytes available to read. I knew there had to be a way to count the bytes in one's own inbox.
There's no guarantee that calls to Read on one side of the connection will match up 1-1 with calls to Write from the other side. If you're dealing with variable length messages, it's up to you to provide the receiving side with this information.
One common way to do this is to first work out the length of the message you're going to send and then send that length information first. On the receiving side, you then obtain the length first and then you know how big a buffer to allocate. You then call Read in a loop until you've read the correct number of bytes. Note that, in your original code, you're currently ignoring the return value from Read, which tells you how many bytes were actually read. In a single call and return, this could be as low as 1, even if you're asking for more than 1 byte.
Another common way is to decide on message "formats" - where e.g. message number 1 is always 32 bytes in length and has X structure, and message number 2 is 51 bytes in length and has Y structure. With this approach, rather than you sending the message length before sending the message, you send the format information instead - first you send "here comes a message of type 1" and then you send the message.
A further common way, if applicable, is to use some form of sentinels - if your messages will never contain, say, a byte with value 0xff then you scan the received bytes until you've received an 0xff byte, and then everything before that byte was the message you wanted to receive.
But, whatever you want to do, whether its one of the above approaches, or something else, it's up to you to have your sending and receiving sides work together to allow the receiver to discover each message.
I forgot to say but a further way to change everything around is - if you want to exchange messages, and don't want to do any of the above fiddling around, then switch to something that works at a higher level - e.g. WCF, or HTTP, or something else, where those systems already take care of message framing and you can, then, just concentrate on what to do with your messages.
You could use StreamReader to read stream to the end
var streamReader = new StreamReader(someTcpClient.GetStream(), Encoding.ASCII);
string result = streamReader.ReadToEnd();
I want to use the stream class to read/write data to/from a serial port. I use the BaseStream to get the stream (link below) but the Length property doesn't work. Does anyone know how can I read the full buffer without knowing how many bytes there are?
http://msdn.microsoft.com/en-us/library/system.io.ports.serialport.basestream.aspx
You can't. That is, you can't guarantee that you've received everything if all you have is the BaseStream.
There are two ways you can know if you've received everything:
Send a length word as the first 2 or 4 bytes of the packet. That says how many bytes will follow. Your reader then reads that length word, reads that many bytes, and knows it's done.
Agree on a record separator. That works great for text. For example you might decide that a null byte or a end-of-line character signals the end of the data. This is somewhat more difficult to do with arbitrary binary data, but possible. See comment.
Or, depending on your application, you can do some kind of timing. That is, if you haven't received anything new for X number of seconds (or milliseconds?), you assume that you've received everything. That has the obvious drawback of not working well if the sender is especially slow.
Maybe you can try SerialPort.BytesToRead property.
Whats the best way to format a message to a server, at moment I'm serilising an object using the binaryformatter and then sending it to the server.
At the server end its listening in an async fashion and then when the buffer size recieved is not 100% it assumes that the transfer has complete.
This is working and the moment, and I can deserialise the object at the other end, I'm just concerned that if I start sending async this method will fail has message's could be blurred.
I know that I need to mark the message somehow as to say that's the end of message one, this other bit belongs to message 2, but I'm unsure of the correct way to do this.
Could anyone point me in the right direction and maybe give me some examples?
Thanks
You could always serialize it to a memory stream, see how big it is, send the length as a 4-byte binary number then send the stream's contents.
On the other side you can just sit and wait for 4 bytes, combine them into an integer then sit and wait for that number of bytes.
When your read function returns (make them blocking reads), you know you have the entire object into a buffer, so you just deserialize it and cast it to your common interface type.
edit: this is the answer to your specific question. This being said you would be better off using a higher-level library than pure tcp.
If your object has a fixed length you can received specified amount of bytes on the other end and then create your object.
Otherwise you can send delimiters (symbol or sequence of symbols that you are not using in your object) between your objects and keep reading received data byte by byte until you see the delimiter.
You might want to take a look at Protocol Buffers (http://code.google.com/p/protobuf), a (implementation language independent) data interchange format/framework. There exist at least two .NET implementations for it (see http://code.google.com/p/protobuf/wiki/ThirdPartyAddOns).