I've got a WebApi method that will receive multipart data. Current implementation uses a MultipartMemoryStreamProvider to receive the contents.
I'd like to ensure that certain parts are received before others (aka, file hash before file contents). Unfortunately, from what I can tell of the framework, the HttpContentMultipartExtensions instantiate a MimeMultipartParser to the HttpContent's response stream... the MimeMultipartParser uses the provided StreamProvider to instantiate new streams as the data arrives... but there appears to be no notification / eventing as the parser switches from the previous stream to the next.
By using events, I can queue the file's hash (before the file contents arrive), pump the file contents onto a file, and be confirming the hash while the next file's hash/contents are arriving.
Unfortunately, every example and bit of code I see, suggests that I can only access the content streams after they are complete. (I do see that MimeMultipartBodyPartParser's ParseBuffer yields its returned MimeBodyParts, I just didn't see any way to access it, since it only appeared to be called from the private MultipartReadAsyncComplete)
Am I missing something? Is there a better way?
I would not try to solve MITM attacks in the MIME parsing code itself, as the MITM is an attack inserted in part of the architecture.
There are topologies and proven solutions such as multifactor or SSL that will allow your client to determine the validity of the session and sender of the content.
Like Evk states, if your architecture is susceptible to MITM, then you need to look at the security of the connection end to end, not a validation of the MIME data on receipt.
If you are concerned about the security of the data, then you need to consider encryption of the data, and using non-opaque encryption techniques (not signing), which would prevent manipulation in transit, and allow the client to decrypt the message on receipt.
If securing the connection between client and server to avoid MITM does not work for you, maybe you could explain the constraints on your architecture so that we can provide better answers.
Related
I want to use gRPC to share very large file (more than 6GB) between endpoints and a server.
The project where I'm currently working require a central server where endpoints can upload and download files. One of the constraint is that endpoints don't know each others, but they can receive and send messages each others from a common bus.
To implement this server and its communication with endpoints, I'm evaluating to use gRPC.
Do you think is the best solution for file stream? what alternatives do I have?
thanks in advance.
gRPC with client/server streaming is capable of handling upload/download of files.
However, there's a discussion here on the performance of gRPC vs HTTP for file upload/download, which says HTTP is any day going to be faster to upload/download because this is just reading/writing incoming bytes, while gRPC performs additional serialization/deserialization for each message in the stream adding significant overhead.
There is another blog doing some benchmark on the same - https://ops.tips/blog/sending-files-via-grpc/ .
If you are looking to implement something that has to handle scale, you can do some more research.
If you really want to do this over gRPC, then the key thing is to make the response "server streaming", so that instead of returning 6GiB in one chunk, it returns multiple chunks of whatever size you need, for example maybe 128kiB at a time (or whatever); you can so this with something like:
syntax = "proto3";
message FileRequest {
string id = 1; // or whatever
}
message FileResponse {
bytes chunk = 1; // some segment of the file
}
service SearchService {
rpc GetFile(FileRequest) returns (stream FileResponse);
}
but nothing is automatic: it is now your job to write the multiple segments back.
I suspect a vanilla http download-style response may be simpler!
Last few days I've been building a web server application in C# that uses HttpListener. I've learned quite a lot on the way, and still am. Currently I got it all working, setting headers here and there depending on certain situations.
In most cases things are working fine, however at times a exception error is thrown. This happens on a few occasions. Most if not all of them is closing a connection before all data is send. Then the error occurs. But some of them are really caused by browsers as far as I can tell.
Like let's take Chrome. Whenever I go to a MP3 file directly, it sends 2 GET requests. And one of them is causing the error, the other is working and receiving part of the content. After this, I can listen the MP3 and there are no issues. Streaming works.
But back to the request that gives me the error, there is nothing in the headers that I could use in my code to not output data, like I do already with HEAD requests. So I'm quite puzzled here.
IE also has this problem with both opening MP3 files directly, and streaming via HTML5 audio tag. It also varies from time to time. Sometimes I open the page, and only 2 requests are made. The HTML page, and the MP3. No error there. Sometimes tho, there are 3 requests. It connects to the MP3 twice. Now sometimes one of those connections is being aborted straight after I open the page, and sometimes 2 requests to the MP3 file, doesn't even accept data. In both request headers, they want end of the file. So bytes: 123-123/124.
I've also tested it on w3school's audio element. IE also makes twice connections there, one aborted, other loading the MP3 file.
So my question is, is it possible to make the web server exception/error-proof, or maybe better question, is it bad that these exceptions are thrown? Or do you perhaps know how to fix these errors?
The error I'm getting is: I/O Operation has been aborted by either a thread exit or an application request.
The way I write to the client is:
using (Stream Output = _CResponse.OutputStream)
{
Output.Write(_FileOutput, rangeBegin, rangeLength);
}
I am not sure if there's another (better) way. This is what I came across in many topics, tutorials and pages while researching.
About headers: Default headers: Content Length, Content Type, Status Code. In some cases, like MP3 files and video's, I add a Accept-Ranges: Bytes header. In case the request header has Range in it, I add Content-Range header, and PartialContent status code.
From the server's point of view any client can disconnect at any time. This is part of the normal operation of a server. Detect this specific case, log it and swallow the exception (because it has been handled). It's not a server bug.
I am using the new PushStreamContent entity in MVC4 to stream notifications from my web server back to multiple listening iOS clients (they are using NSURLConnection). The messages being sent are JSON. When I send messages that are less than 1024 bytes, the message sends as expected. Sending messages larger than this size however causes the client to receive the message in multiple chunks, each being 1024 bytes.
I am wondering what is the best way for my iOS clients to consume these multiple messages coming back? Is there a way to have NSURLConnection aggregate the results for me, or do I need to implement something that gets a result, checks if it's valid json, if not wait for the next result and append the previous, and continue until it is valid? What is a better way of doing this?
I found that you are able to adjust the size of the buffer that writes data to the stream that PushStreamContent uses. However, chunking the data is the correct thing for it to do and keeping this small has several advantages. I ended up writing my own method to aggregate the data flowing in on the client side. See the following question for more details:
How to handle chunking while streaming JSON data to NSURLConnection
I have a WCF client (running on Win7) pointing to a WebSphere service.
All is good from a test harness (a little test fixture outside my web app) but when my calls to the service originate from my web project one of the calls (and only that one) is extremely slow to deserialize (it takes minutes VS seconds) and not just the first time.
I can see from fiddler that the response comes back quickly but then the WCF client hangs on the response itself for more than a minute before the next line of code is hit by the debugger, almost if the client was having trouble deserializing. This happens only if in the response I have a given pdf string (the operation generates a pdf), base64 encoded chunked. If for example the service raises a fault (thus the pdf string is not there) then the response is deserialized immediately.
Again, If I send the exact same envelope through Soap-UI or from outside the web project all is good.
I am at loss - What should I be looking for and is there some config setting that might do the trick?
Any help appreciated!
EDIT:
I coded a stub against the same service contract. Using the exact same basicHttpBinding and returning the exact same pdf string there is no delay registered. I think this rules out the string and the binding as a possible cause. What's left?
Changing transferMode="Buffered" into transferMode="Streamed" on the binding did the trick!
So the payload was apparently being chunked in small bits the size of the buffer.
I thought the same could have been achieved by increasing the buffersize (maxBufferSize="1000000") but I had that in place already and it did not help.
I have had this bite me many times. Check in your WCF client configuration that you are not trying to use the windows web proxy, that step to check on the proxy (even if there is not one configured) will eat up a lot of time during your connection.
If the tips of the other users don't help, you might want to Enable WCF Tracing and open it in the Service Trace Viewer. The information is detailed, but it has enabled me to fix a number of hard-to-identity problems in the past.
More information on WCF Tracing can be found on MSDN.
Two thing you can try:
Adjust the readerQoutas settings for your client. See http://msdn.microsoft.com/en-us/library/ms731325.aspx
Disable "Just My Code" in debugging options. Tools -> Options -> Debugging -> General "Enable Just My Code (Managed only)" and see if you can catch interal WCF exceptions.
//huusom
I had the very same issue... The problem of WCF, IMO, is in the deserialization of the base64 string returned by the service into a byte[] client side.
The easiest to solve this if you may not change your service configuration (Ex.: use a transferMode="Streamed") is to adapt your DataContract/ServiceContract client side. Replace the type "byte[]" with "string" in the Response DataContract.
Next simply decode the returned string yourself with a piece of code such as:
byte[] file = Convert.FromBase64String(pdfBase64String);
To download a PDF of 70KB, it used to required ~6 sec. With the suggested change here above, it takes now < 1 sec.
V.
PS.: Regarding the transfer mode, I did try to only change the client side (transferMode="StreamedResponse") but without improvement...
First things to check:
Is the config the same in the web project and the test project?
When you test from SOAP UI are you doing it from the same server and in the same security context as when the code is running from the web project.
-Is there any spike in the memory when the pdf comes back?
Edit
From your comments the 1 min wait, appears that it is waiting for a timeout. You also mention transactions.
I am wondering if the problem is somewhere else. The call to the WCF service goes OK, but the call is inside a transaction, and there is no complete or dispose on the transaction (I am guessing here), then the transaction / code will hang for 1 min while it waits to timeout.
Edit 2
Next things to check:
Is there any difference in the code is the test and in the web project, on how the service is being called.
Is there any differnce in the framework version, for example is one 3.0 and the other 3.5
Can it be that client side is trying to analyse what type of content is coming from server side? Try to specify mime type of the service response explicitly on the server side, e.g. Response.ContentType = "application/pdf" EDIT: By client side I mean any possible mediator like a firewall or a security suite.
I am using the WebClient.UploadFile() method to post files to a service for processing. The file contains an XML document with a compressed, b64-encoded content element in it. For some files (currently 1), the UploadFile throws an exception, indicating that the underlying connection was closed. The innermost exception on socket level gives the message 'An existing connection was forcibly closed by the remote host'.
Questions:
Has anyone necountered the same problem?
Why does it throw an exception for some files, and not for all?
Should I set some additional parameter for files with binary content?
Is there a workaround?
This functionality does work fine in a VPN situation, but obviously we want to use it to work in standard Internet situations.
Thanks, Rine
Sounds like a firewall or other security software sitting in between you and the server may be rejecting the request as a potential attack. I've run into this before where firewalls were rejecting requests that contained a specific file extension-- even if that file extension was encoded in a query string parameter!
If I were you, I'd take the problematic file and start trimming XML out of it. You may be able to find a specific chunk of XML which triggers the issue. Once you've identified the culprit, you can figure out how to get around the issue (e.g. by encoding those characters using their Unicode values instead of as text before sending the files). If, however, any change to the file causes the problem to go away (it's not caused by a specific piece of worrisome text), then I'm stumped.
Any chance it's a size issue and the problematic file is above a certain size and all the working files are below it? The server closing the connection when it hits a max accepted request size matches your symptom. You mentioned it worked in VPN so it's admittedly a stretch, but maybe the VPN case was a different server that's configured differently (or the max request is different for some other reason).
Are there non-WebClient methods for uploading the file to the same service from the same machine and if so, do they work?