Develop Reference

Raw http request parsing

I am using windivert to capture tcp packet. I am successful to capture the packets by WinDivertSharp.dll. Now i want to parse the packet for only http request. To parse packet i am using the flowing code.
var index = BinaryMatch(messageBody, Encoding.ASCII.GetBytes("\r\n\r\n")) + 4;
var headers = Encoding.ASCII.GetString(messageBody, 0, index);
var memory = new MemoryStream(messageBody);
memory.Position = index;
string body = "";
if (headers.IndexOf("Content-Encoding: gzip") > 0)
{
using (GZipStream decompressionStream = new GZipStream(memory, CompressionMode.Decompress))
using (var decompressedMemory = new MemoryStream())
{
decompressionStream.CopyTo(decompressedMemory);
decompressedMemory.Position = 0;
body = Encoding.UTF8.GetString(decompressedMemory.ToArray());
}
}
else
{
body = Encoding.UTF8.GetString(messageBody, index, messageBody.Length - index);
}
It working good for some webs. But for http request like from https://stackoverflow.com/ it is not working. Please help to decode
http request for all kinds of web.

Might be a long shot, but it might be related to WinDivert not decrypting responses from sites that use SSL/TLS (HTTPS) like https://stackoverflow.com/.
I would suggest you try and compare between sites that use HTTP and HTTPS.
Hope it helps!

C# StreamContent and File.OpenRead() Not Producing HTTP'able multipart content

I have a .Net Core 2.0 application that is sending files to a Web API endpoint, using multipart content. Everywhere I've looked, such as C# HttpClient 4.5 multipart/form-data upload, makes it seem that it should be as easy as passing a FileStream to a StreamContent. However, when I make the post, it looks like the file is attaching as text, not bits.
Actual code:
var request = new HttpRequestMessage()
{
Method = HttpMethod.Post,
RequestUri = new Uri( "http://localhost:10442/filetest" )
};
var multiContent = new MultipartFormDataContent();
var filestream = File.OpenRead( path );
var filename = Path.GetFileName( path );
var streamContent = new StreamContent( filestream );
streamContent.Headers.Add( "Content-Type", "application/octet-stream" );
streamContent.Headers.Add( "Content-Disposition", $"form-data; name=\"file1\"; filename=\"{filename}\"" );
multiContent.Add( streamContent, "file", filename );
request.Content = multiContent;
var response = await new HttpClient().SendAsync( request );
The request looks like this which, as you may notice, is not all on one line (which I think is a/THE problem):
POST http://localhost:10442/filetest HTTP/1.1
Content-Type: multipart/form-data; boundary="c5295887-425d-4ec7-8638-20c6254f9e4b"
Content-Length: 88699
Host: localhost:10442
--c5295887-425d-4ec7-8638-20c6254f9e4b
Content-Type: application/octet-stream
Content-Disposition: form-data; name="file1"; filename="somepdf.pdf"
%PDF-1.7
%
1 0 obj
<</Type/Catalog/Version/1.7/Pages 3 0 R/Outlines 2 0 R/Names 8 0 R/Metadata 31 0 R>>
endobj
Fiddler shows the entire post all the way down to the end boundary, but await Request.Content.ReadAsStringAsync() in the endpoint only shows the first couple dozen bytes (it looks as if the stream wasn't finished, but if Fiddler got it all, shouldn't my endpoint have too?).
I was having similar trouble trying to hit a remote endpoint; I built this endpoint to test locally.
The exception I'm getting is:"Unexpected end of MIME multipart stream. MIME multipart message is not complete." To me, this makes sense both if I'm really only getting part of my stream, or if the line breaks are throwing something off.
I have also tried throwing some of the Idisposables into Usings but, as expected, that closes the streams and I get exceptions that way.
And for completeness's sake, here's the endpoint I'm calling:
public async void ReceiveFiles()
{
// exception happens here:
var mpData = await Request.Content.ReadAsMultipartAsync();
await Task.FromResult( 0 );
}

Try something like this:
static int Main(string[] args)
{
var request = new HttpRequestMessage()
{
Method = HttpMethod.Post,
RequestUri = new Uri("http://localhost:10442/filetest")
};
var path = "c:\\temp\\foo.bak";
using (var filestream = File.OpenRead(path))
{
var length = filestream.Length.ToString();
var streamContent = new StreamContent(filestream);
streamContent.Headers.Add("Content-Type", "application/octet-stream");
streamContent.Headers.Add("Content-Length", length);
request.Content = streamContent;
Console.WriteLine($"Sending {length} bytes");
var response = new HttpClient().SendAsync(request).Result;
Console.WriteLine(response.Content.ReadAsStringAsync().Result);
}
Console.WriteLine("Hit any key to exit");
Console.ReadKey();
return 0;
}
and
[HttpPost]
public async Task<IActionResult> Upload()
{
var buf = new byte[1024 * 64];
long totalBytes = 0;
using (var rs = Request.Body)
{
while (1 == 1)
{
int bytesRead = await rs.ReadAsync(buf, 0, buf.Length);
if (bytesRead == 0) break;
totalBytes += bytesRead;
}
}
var uploadedData = new
{
BytesRead = totalBytes
};
return new JsonResult(uploadedData) ;
}

I'm trying to solve a similar issue, and I'm not 100% to a solution yet, but maybe some of my research can help you.
It was helpful to me to read through the microsoft docs for .NET core file uploads, specifically for large files that use streams and multipart form data:
https://learn.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads?view=aspnetcore-2.1#uploading-large-files-with-streaming
You already referenced it, but there's some relevant useful information in this answer:
C# HttpClient 4.5 multipart/form-data upload
This explains the details of the content-disposition header and how it is used with multipart form data requests: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Disposition#As_a_header_for_a_multipart_body
As to your specific problem of the file being sent as text instead of bits, since http is text-based, it can only be sent as text, but that text can be encoded as you see fit. Perhaps your StreamContent needs a specific encoding to be used, like base64 encoding or similar? I do believe the newlines are significant in the multipart request, so hopefully setting the encoding for the file content as needed would be enough.
Another possibility: could it be that you need to set additional information on the file section's headers or in the definition of the StreamContent to indicate that it should expect to continue, or that the boundary information isn't put in correctly? See Multipart forms from C# client

I use this lib : https://github.com/jgiacomini/Tiny.RestClient
It's make easier to send multiplart file to send multipart files.
Here a sample :
await client.PostRequest("MultiPart/Test").
AsMultiPartFromDataRequest().
AddStream(stream1, "request", "request2.bin").
AddStream(stream2, "request", "request2.bin")
ExecuteAsync();

Webserver response for image in C#

I'm trying to develop a little program in C# as UWP that will be able to send a response over http as HTML page. Actually I'm able to answer with a text stream with this code:
using (var output = args.Socket.OutputStream)
{
using (var response = output.AsStreamForWrite())
{
var html = Encoding.UTF8.GetBytes(htmlResponse);
using (var bodyStream = new MemoryStream(html))
{
var header = $"HTTP/1.1 200 OK\r\nContent-Length: {bodyStream.Length}\r\nConnection: close\r\n\r\n";
var headerArray = Encoding.UTF8.GetBytes(header);
await response.WriteAsync(headerArray, 0, headerArray.Length);
await bodyStream.CopyToAsync(response);
await response.FlushAsync();
}
}
}
And the response is
HTTP/1.1 200 OK
Content-Length: {bodyStream.Length}
Connection: close
But when I try to answer with a jpeg image or png image, the answer is not interpreted from browser. I tryed to convert the image in byte array, stream, Base64 but nothing will do the job.
How can I do that?
Thanx a lot

You need to set response.ContentType="image/jpeg" or response.ContentType="image/png". Otherwise the browser does not know how to render the image

I think you only need to add a header with the mime type: image/jpeg and try to response with the types you listed above.(byte array, stream, Base64)

Passing zip file in httpwebrequest body

I am trying to implement the following cURL call with c# code:
curl -H "Content-Type: application/zip" -u admin:admin -X POST --data-binary #<path to .zip> http://localhost:606060/content/test.json
I tried the following code, but the server is returning a 400 bad request code.
___________________________________________________
MemoryStream postDataStream = new MemoryStream();
StreamWriter postDataWriter = new StreamWriter(postDataStream);
postDataWriter.Write("\r\n--" + boundary + "\r\n");
postDataWriter.Write("Content-Disposition: form-data; name=\"{0}\"\r\n\r\n{1}",
"myFileDescription",
"A sample file description");
// Include the file in the post data
postDataWriter.Write("\r\n--" + boundary + "\r\n");
// Include the file in the post data
postDataWriter.Write("Content-Disposition: form-data;"
+ "name=\"{0}\";"
+ "filename=\"{1}\""
+ "\r\nContent-Type: {2}\r\n\r\n",
"myFile",
Path.GetFileName(filePath),
"application/zip");
postDataWriter.Flush();
// Read the file
FileStream fileStream = new FileStream(filePath, FileMode.Open, FileAccess.Read);
byte[] buffer = new byte[1024];
int bytesRead = 0;
while ((bytesRead = fileStream.Read(buffer, 0, buffer.Length)) != 0)
{
postDataStream.Write(buffer, 0, bytesRead);
}
fileStream.Close();
postDataWriter.Write("\r\n--" + boundary + "--\r\n");
postDataWriter.Flush();
// Set the http request body content length
request.ContentLength = postDataStream.Length;
// Dump the post data from the memory stream to the request stream
using (Stream s = request.GetRequestStream())
{
postDataStream.WriteTo(s);
}
postDataStream.Close();
try
{
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
// I am getting exception on this line
{
.......................................................................
Actually the code I used here is for passing a file as a form parameter. In my case, there is no form and I am specifying the files path explicitly. I think the way I am making the request is wrong. Is there any better way to make an httpwebrequest with C# corresponding to the cURL request I provided?
The documentation only says:
The request body must contain the zip file .

The curl command you use send the entire zip file contents in the body, plain and simple.
The source code you provide is a completely different beast as it implements multipart formpost content with headers and boundaries and all. It looks like you're overdoing it.
Use curl's --trace-ascii option and you can see exactly what it sends and you may realize the differences better.

Use WebClient, DownloadData method and Custom Header to Post to a web page

** Update I have a solution below, but it's rather manual... if there is an easier way, I'd prefer that answer.
I want to record execution and download time of a webpage, and I am trying to use the WebClient.DownloadDataAsync method to post data to a page and get a result.
I know there are numerous ways to post using WebClient or WebRequest objects, but none of them dispatch the DownloadProgressChanged event. This event is key, as I want to track progress of the download, which is in the area of 30 megs+.
I figure the best way to do this would be to construct the header manually, but I'm hitting a dead-ends. Mainly, I cannot add the data to the request.
WebClient client = new WebClient();
//!!!!*****no idea how to add the following data to the request
string data = "test1=value";
client.Headers.Add( "POST", "/About.aspx HTTP/1.1" );
client.Headers.Add( "Host", "http://localhost:12065" );
client.Headers.Add( "Content-Type", "application/x-www-form-urlencoded" );
client.Headers.Add( "Content-length", data.Length.ToString() );
client.DownloadProgressChanged += OnDownloadProgressChanged;
client.DownloadDataCompleted += OnDownloadDataCompleted;
client.DownloadDataAsync( new Uri( "http://localhost:12065/About.aspx" ) );
I am open to other solutions, or getting the header in this example to work as if the client is sending a POST.

OK... well I figured out a way to do this using WebRequest. It's a little more complicated, but it does the trick. I am surprised that WebClient doesn't do this already, but it seems as though WebClient has a number of unexpected issues.
//Using System.Diagnostics, set up a Stopwatch to time the execution
Stopwatch stopWatch = new Stopwatch();
stopWatch.Start();
//convert your post data into a byte array... we're dealing with streams here
byte[] postData = Encoding.ASCII.GetBytes( postData );
//We're using the WebRequest object found in System.Net to do our work for us
WebRequest request = WebRequest.Create( url );
//Set all your headers
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "POST";
request.ContentLength = postData.Length;
//set up your request stream and write out your post data
Stream stream = request.GetRequestStream();
stream.Write( postData, 0, postData.Length );
stream.Close();
//Send the data and wait for a response. [blocking operation]
WebResponse response = request.GetResponse();
//Once you reach this line, the server has finished doing it's work,
//and downloading has commenced
Console.WriteLine( "First Response: {0}", stopWatch.Elapsed );
//Start timing the download, wouldn't it be nice is there was a restart method? (.Net 4.0)
stopWatch.Reset();
stopWatch.Start();
//We want to receive the data. so ask for the response stream
Stream reponseStream = response.GetResponseStream();
//In my case, there are megabytes of information, so the console is no good,
//I'll store it in a file.
FileStream fileStream = File.Open( "result.txt", FileMode.OpenOrCreate );
//create a buffer to collect data as it is downloaded
byte[] buffer = new byte[1024];
//This loop will run as long as the responseStream can read data. (i.e. downloading data)
//Once it reads 0... the downloading has completed
int resultLength;
while( ( resultLength = reponseStream.Read( buffer, 0, buffer.Length ) ) != 0 )
{
//You could further measure progress here... but I don't care.
fileStream.Write( buffer, 0, resultLength );
}
//Everything is done... how long did it take to download?
Console.WriteLine( "Download Complete: {0}", stopWatch.Elapsed );
//housekeeping
fileStream.Flush();
fileStream.Close();
stopWatch.Stop();