I'm trying to implement HMAC security for an API. Everything works fine until I try to post a file.
The HMAC solution can be found here - https://github.com/gavinharriss/WebAPI.HMAC - it's a fork from the original to allow GET requests as well as POST requests.
The code to attach a file:
var requestContent = new MultipartFormDataContent();
var fileContent = new ByteArrayContent(file);
requestContent.Add(fileContent, "file", filename);
if I immediately call HttpContent.ReadAsByteArrayAsync() there is no issue, the byte array is available.
However, the HMAC HttpClient (HMACHttpClient) implements a DelegatingHandler (HMACDelegatingHandler) in order to attach the HMAC header to requests.
In the HMACDelegatingHandler the request is passed along as a HttpRequestMessage from which the HttpRequestMessage.Content property is used in a helper to build the HMAC signature.
When building the signature, the following code is called from a helper class:
private static async Task<byte[]> ComputeHash(HttpContent httpContent)
{
using (var md5 = MD5.Create())
{
byte[] hash = null;
if (httpContent != null)
{
var content = await httpContent.ReadAsByteArrayAsync(); // <-- Fails here
if (content.Length != 0)
{
hash = md5.ComputeHash(content);
}
}
return hash;
}
}
When stepping through the code the var content = await httpContent.ReadAsByteArrayAsync() line is hit, then nothing, no error. The requests just seems to go poof but everything is still running and the HttpClient request never gets sent.
Any ideas what's going on?
Having tested this with various sizes of file, I found the issue arose when files got around the 50,000 byte mark.
This post provided a solution: HttpContent.ReadAsStringAsync causes request to hang (or other strange behaviours).
If you replace erroring line in HMACHelper (line 66):
var content = await httpContent.ReadAsByteArrayAsync();
with this:
var ms = new MemoryStream();
await httpContent.CopyToAsync(ms);
ms.Seek(0, SeekOrigin.Begin);
var content = ms.ToArray();
It should stop hanging.
Related
I am trying to make a call from my Python client to my C#.NET api endpoint.
The following two code blocks work correctly:
# Python Client
endpoint = "http://localhost:12345/api/myController/hmacTestPut"
data = {"type": "my_type", "state": "my_state"}
headers = calc_hmac(...)
r = requests.put(url=endpoint, json=data, headers=headers)
// C# Endpoint
[HttpPut]
[Route("api/myController/hmacTestPut")]
//[HMACAuthentication]
public IHttpActionResult HMACTestPut(MyDTO obj)
{
return Ok(obj)
}
However, when I include the HMACAuthentication attribute on the C# endpoint, the MyDTO obj is null.
Since the endpoint is hit in both scenarios, I believe my HMACAuthenticationAttribute is correct.
Including [FromBody] does not seem to have an effect either way.
Is there a reason why adding HMACAuthentication to an endpoint would stop the body content from being received?
In case anybody else runs into this issue, this problem was (obviously) HMAC related.
In my HMACAuthenticationAttribute, in my ComputeHash() function, which is where I calculate the hash for the body of the request (as in POST or PUT body).
What was breaking is below, where reading the HttpContent to a MemoryStream left the HttpContent of the request empty for the actual endpoint to read nothing:
var ms = new MemoryStream();
await httpContent.copyToAsync(ms);
ms.Seek(0, SeekOrigin.Begin);
var content = ms.ToArray();
byte[] hash = md5.ComputeHash(content)
The solution was to change how we read in the content of the HttpRequest:
var content = await httpContent.ReadAsByteArrayAsync().ConfigureAwait(false);
byte[] hash = md5.ComputeHash(content)
Full simplified function below, in hopes that it helps somebody.
private static async Task<byte[]> ComputeHash(HttpContent httpContent)
{
using (MD5 md5 = MD5.Create())
{
byte[] hash = null;
var content = await httpContent.ReadAsByteArrayAsync().ConfigureAwait(false);
if (content.Length != 0)
{
hash = md5.ComputeHash(content);
}
return hash;
}
}
I have a .Net Core 2.0 application that is sending files to a Web API endpoint, using multipart content. Everywhere I've looked, such as C# HttpClient 4.5 multipart/form-data upload, makes it seem that it should be as easy as passing a FileStream to a StreamContent. However, when I make the post, it looks like the file is attaching as text, not bits.
Actual code:
var request = new HttpRequestMessage()
{
Method = HttpMethod.Post,
RequestUri = new Uri( "http://localhost:10442/filetest" )
};
var multiContent = new MultipartFormDataContent();
var filestream = File.OpenRead( path );
var filename = Path.GetFileName( path );
var streamContent = new StreamContent( filestream );
streamContent.Headers.Add( "Content-Type", "application/octet-stream" );
streamContent.Headers.Add( "Content-Disposition", $"form-data; name=\"file1\"; filename=\"{filename}\"" );
multiContent.Add( streamContent, "file", filename );
request.Content = multiContent;
var response = await new HttpClient().SendAsync( request );
The request looks like this which, as you may notice, is not all on one line (which I think is a/THE problem):
POST http://localhost:10442/filetest HTTP/1.1
Content-Type: multipart/form-data; boundary="c5295887-425d-4ec7-8638-20c6254f9e4b"
Content-Length: 88699
Host: localhost:10442
--c5295887-425d-4ec7-8638-20c6254f9e4b
Content-Type: application/octet-stream
Content-Disposition: form-data; name="file1"; filename="somepdf.pdf"
%PDF-1.7
%
1 0 obj
<</Type/Catalog/Version/1.7/Pages 3 0 R/Outlines 2 0 R/Names 8 0 R/Metadata 31 0 R>>
endobj
Fiddler shows the entire post all the way down to the end boundary, but await Request.Content.ReadAsStringAsync() in the endpoint only shows the first couple dozen bytes (it looks as if the stream wasn't finished, but if Fiddler got it all, shouldn't my endpoint have too?).
I was having similar trouble trying to hit a remote endpoint; I built this endpoint to test locally.
The exception I'm getting is:"Unexpected end of MIME multipart stream. MIME multipart message is not complete." To me, this makes sense both if I'm really only getting part of my stream, or if the line breaks are throwing something off.
I have also tried throwing some of the Idisposables into Usings but, as expected, that closes the streams and I get exceptions that way.
And for completeness's sake, here's the endpoint I'm calling:
public async void ReceiveFiles()
{
// exception happens here:
var mpData = await Request.Content.ReadAsMultipartAsync();
await Task.FromResult( 0 );
}
Try something like this:
static int Main(string[] args)
{
var request = new HttpRequestMessage()
{
Method = HttpMethod.Post,
RequestUri = new Uri("http://localhost:10442/filetest")
};
var path = "c:\\temp\\foo.bak";
using (var filestream = File.OpenRead(path))
{
var length = filestream.Length.ToString();
var streamContent = new StreamContent(filestream);
streamContent.Headers.Add("Content-Type", "application/octet-stream");
streamContent.Headers.Add("Content-Length", length);
request.Content = streamContent;
Console.WriteLine($"Sending {length} bytes");
var response = new HttpClient().SendAsync(request).Result;
Console.WriteLine(response.Content.ReadAsStringAsync().Result);
}
Console.WriteLine("Hit any key to exit");
Console.ReadKey();
return 0;
}
and
[HttpPost]
public async Task<IActionResult> Upload()
{
var buf = new byte[1024 * 64];
long totalBytes = 0;
using (var rs = Request.Body)
{
while (1 == 1)
{
int bytesRead = await rs.ReadAsync(buf, 0, buf.Length);
if (bytesRead == 0) break;
totalBytes += bytesRead;
}
}
var uploadedData = new
{
BytesRead = totalBytes
};
return new JsonResult(uploadedData) ;
}
I'm trying to solve a similar issue, and I'm not 100% to a solution yet, but maybe some of my research can help you.
It was helpful to me to read through the microsoft docs for .NET core file uploads, specifically for large files that use streams and multipart form data:
https://learn.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads?view=aspnetcore-2.1#uploading-large-files-with-streaming
You already referenced it, but there's some relevant useful information in this answer:
C# HttpClient 4.5 multipart/form-data upload
This explains the details of the content-disposition header and how it is used with multipart form data requests: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Disposition#As_a_header_for_a_multipart_body
As to your specific problem of the file being sent as text instead of bits, since http is text-based, it can only be sent as text, but that text can be encoded as you see fit. Perhaps your StreamContent needs a specific encoding to be used, like base64 encoding or similar? I do believe the newlines are significant in the multipart request, so hopefully setting the encoding for the file content as needed would be enough.
Another possibility: could it be that you need to set additional information on the file section's headers or in the definition of the StreamContent to indicate that it should expect to continue, or that the boundary information isn't put in correctly? See Multipart forms from C# client
I use this lib : https://github.com/jgiacomini/Tiny.RestClient
It's make easier to send multiplart file to send multipart files.
Here a sample :
await client.PostRequest("MultiPart/Test").
AsMultiPartFromDataRequest().
AddStream(stream1, "request", "request2.bin").
AddStream(stream2, "request", "request2.bin")
ExecuteAsync();
I have tried to create a simple console application.
We have a call system from 8x8 that provide a web streaming API but their documentation is very limited and nothing in C#.
The api service streams call statuses in near real time and I would like to get that 'stream' and be able to read and process it in realtime if possible. The response or Content Type is 'text/html'. But the actual body of the response can be declared as json - sample below:
{"Interaction":{"attachedData":{"attachedDatum":[{"attachedDataKey":"#pri","attachedDataValue":100},{"attachedDataKey":"callingName","attachedDataValue":999999999999},{"attachedDataKey":"cha","attachedDataValue":99999999999},{"attachedDataKey":"cnt","attachedDataValue":0},{"attachedDataKey":"con","attachedDataValue":0},{"attachedDataKey":"med","attachedDataValue":"T"},{"attachedDataKey":"pho","attachedDataValue":9999999999},{"attachedDataKey":"phoneNum","attachedDataValue":9999999999},{"attachedDataKey":"tok","attachedDataValue":999999999}]},"event":"InteractionCreated","inboundChannelid":9999999999,"interactionEventTS":9999999,"interactionGUID":"int-15b875d0da2-DJOJkDhDsrh3AIaFP8VkICv9t-phone-01-testist","resourceType":0}}
I have seen several posts concerning httpClient and the GetAsync methods but none of these appear to work as they appear to be for calls when a response is made, not something that constantly has a response.
Using fiddler for the call it does not appear to close so the stream is constantly running, so fiddler does not display any data until a separate user or instance connects.
When I use a browser the content is 'streamed' to the page and updates automatically and shows all the content (as above).
The api contains authentication so when another client connects and retrieves data the connected client closes and finally I am able to see the data that was gathering.
This is the code so and does return the big stream when another client connects but ideally I want a real time response and appears to just get stuck in the GETASYNC method:
var response = await client.GetAsync(address, HttpCompletionOption.ResponseHeadersRead);
if (response.IsSuccessStatusCode)
{
var responseContent = response.Content;
string responseString = await responseContent.ReadAsStringAsync();
Console.WriteLine(responseString);
}
Hopefully that's enough information for one of you clever people to help me in my predicament.
I was also having an issue consuming their streaming API and the examples I found that worked with the Twitter and CouchBase streaming API's did not work with 8x8. Both Twitter and CouchBase send line terminators in their pushes so the solution relied on ReadLine to pull in the feed. Since 8x8 does not send terminators you'll need to use ReadBlock or better ReadBlockAsync.
The following code shows how to connect using credentials and consume their feed:
private static async Task StreamAsync(string url, string username, string password)
{
var handler = new HttpClientHandler()
{
Credentials = new NetworkCredential {UserName = username, Password = password},
PreAuthenticate = true
};
// Client can also be singleton
using (var client = new HttpClient(handler))
{
client.Timeout = TimeSpan.FromMilliseconds(Timeout.Infinite);
var request = new HttpRequestMessage(HttpMethod.Get, url);
request.Headers.Connection.Add("keep-alive");
using (var response = await client.SendAsync(
request,
HttpCompletionOption.ResponseHeadersRead))
{
using (var body = await response.Content.ReadAsStreamAsync())
{
using (var reader = new StreamReader(body))
{
while (!reader.EndOfStream)
{
var buffer = new char[1024];
await reader.ReadBlockAsync(buffer, 0, buffer.Length);
Console.WriteLine(new string(buffer));
}
}
}
}
}
}
I'm trying to implement HMAC security for an API. Everything works fine until I try to post data values alongside a file in a MultipartFormDataContent.
The HttpClient DelegatingHandler fails silently when the async line of code to read bytes is hit.
Here's the code building the request:
private FileOutputViewModel GetApiOutput(Uri apiResource, string filename, byte[] file, IDictionary<string, string> extraParameters)
{
FileOutputViewModel result = new FileOutputViewModel();
if (file != null)
{
using (var content = new MultipartFormDataContent())
{
if (extraParameters != null)
{
foreach (var param in extraParameters)
{
content.Add(new StringContent(param.Value), param.Key); // <- If I don't have this, everything works fine
}
}
var fileContent = new ByteArrayContent(file);
fileContent.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment")
{
FileName = filename
};
content.Add(fileContent);
var response = HttpClient.PostAsync(apiResource.ToString(), content).Result;
result.Output = JsonConvert.DeserializeObject(response.Content.ReadAsStringAsync().Result);
result.Filename = Path.GetFileName(filename);
}
}
return result;
}
If I don't use the DelegatingHandler everything works fine, but the HMAC security isn't implemented for the request so is rejected on the API's end.
If I don't add the data values using StringContent items alongside the file then there's no problem reading the bytes. But I'm left with an incomplete request as I need to pass more info along with the file.
The line of code that fails in the DelegatingHandler is indicated below:
private static async Task<byte[]> ComputeHash(HttpContent httpContent)
{
using (var md5 = MD5.Create())
{
byte[] hash = null;
if (httpContent != null)
{
var ms = new MemoryStream();
await httpContent.CopyToAsync(ms); // <- Fails here
ms.Seek(0, SeekOrigin.Begin);
var content = ms.ToArray();
if (content.Length != 0)
{
hash = md5.ComputeHash(content);
}
}
return hash;
}
}
Originally the failing line was:
var content = await httpContent.ReadAsByteArrayAsync();
but this failed with even just the file on its own (previous Stackoverflow question). Using a MemoryStream was one step forward but hasn't got me all the way.
Any ideas how I might be able to work around this issue?
Seems this was caused by having an async signature for the System.Net.Http.DelegatingHandler.SendAsync method. Originally the delegate override was:
protected async override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
when I adapted the code so I could change it to:
protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
everything started to work as expected.
Seems there must be threading issues with this part of the .NET framework. There's some other workarounds described here if you need to try other work-arounds: https://social.msdn.microsoft.com/Forums/vstudio/en-US/55f5571d-fe94-4b68-b1d4-bfb91fd721dd/reading-httpcontent-bytes-fails-inside-delegatinghandler-when-multiple-content-types-present?forum=wcf
I'm trying to figure out how to calculate the correct checksum when passing data to the Softlayer Object Storage.
I know the ETag is the issue because if I remove it form the request it works, however I'd prefer to use it to verify the uploads are not corrupt.
This is my method:
public bool SaveFile(byte[] file, eFetchStorageContainers container, string internalFileName, string fileName = "", bool overPublicNetwork = false)
{
Authenticate(overPublicNetwork);
client = new RestClient(storage_url);
var resourcePath = string.Format("/{0}/{1}", container, internalFileName);
var req = new RestRequest(resourcePath, RestSharp.Method.PUT);
req.AddHeader("X-Auth-Token", auth_token);
req.AddFile(internalFileName, file, fileName);
var md5Checksum = BitConverter.ToString(MD5.Create().ComputeHash(file)).Replace("-", string.Empty).ToLower();
req.AddHeader("ETag", md5Checksum);
var resp = client.Execute(req);
return false;
}
Here is how the ETag is defined:
I believe the problem lies in the fact that i'm getting the checksum for the file and not the request body.
I want to verify that I should be getting the checksum of the Request Body and NOT the file alone.
If the above is true I'm not even sure how to get the checksum for the body - would love some guidance...
Well I did not use C#, but it works using curl fine for me. I get the checksum for the file and it is working fine.
just in case here some examples about this https://community.runabove.com/kb/en/object-storage/how-to-check-file-consistency-using-etag-and-md5.html
Make sure that your request is similar to examples of the link above.
This is the curl I used:
curl -X PUT -T "C:\Users\ncabero\Downloads\picture.jpg" -H "X-Auth-Token: AUTH_XXXXXXX" -H "Etag: a43bf68dd35599a7873c12128f71b1f4" https://dal05.objectstorage.softlayer.net/v1/AUTH_d684780d-aafe-4772-bcbb-0f07d5f6edf3/rcvtest/picture.jpg
I actually figured this out, I was using RestSharp however its impossible to get the request body.
I moved over to HttpClient and was able to access the request body to create a checksum.
var httpClient = new HttpClient();
httpClient.DefaultRequestHeaders.Add("X-Auth-Token", auth_token);
var bytes = new ByteArrayContent(file);
var formData = new MultipartFormDataContent();
formData.Add(bytes, internalFileName, internalFileName);
// this creates a checksum to send over for verification of non corrupted transfers
// this is also prevents us from using RestSharp due to its inability to create a checksum of the request body prior to sending
var md5Checksum = BitConverter.ToString(MD5.Create().ComputeHash(formData.ReadAsByteArrayAsync().Result)).Replace("-", string.Empty).ToLower();
httpClient.DefaultRequestHeaders.Add("ETag", md5Checksum);
var url = string.Format("{0}/{1}{2}/{3}", storage_url, containerName, folderId, internalFileName);
var resp = httpClient.PutAsync(url, formData).Result;
httpClient.Dispose();