I have a .NET web app (MVC4, Web API) that needs to receive large uploads (videos >500MB) via Multipart POST requests. I am trying to improve user experience by relaying the incoming stream of the video part directly to the server that will process and store the video (Kaltura). This is to be an improvement over the current process which stores the video in a file on the filesystem which is later retrieved by the video server.
I've done this by creating a subclass of MultipartStreamProvider called RelayingStreamProvider. I pass it the request stream of a WebRequest I've created to the Kaltura server and it uses this for writing. When I construct the RelayingStreamProvider, I wrap the request stream in a BufferedStream to handle any latency or interruptions on either the reading or writing side of this process.
I've turned off Request buffering in my Web API controller as well.
My questions:
Am I overlooking anything that will likely bite me later with this approach? I'm trying to constrain for efficiency by keeping the filesystem out of the process. On the upside, multiple simultaneous 500 MB uploads are consuming essentially no memory in my test implementation. Downsides?
I'm using a 32KB buffer for my BufferedStream. Is this a reasonable figure? What are the reasons to make this larger/smaller?
Finally, I am indeed aware that there is a widget and mobile APIs I can use to take myself out of the middleman role for Kaltura. And we are going to be implementing these. Since the widget requires Flash, we are using this approach as a fallback for users who don't have Flash.
Related
I’m using a service reference to communicate between my clients (C#) and my server.
I Frequently want to transfer big amounts of data (> 2 MB) over this web service, this works perfectly but with a few customers who has a very small upload this process is very slow and complete fills the upload for a limited time.
I’m looking for a way to limits the bandwidth used by the client to make sure there is some upload available for the customer itself.
I found various solutions the throttle streams but is it possible to use throttle in a web service call?
Maybe using a custom binding?
I'm trying to use WCF web service in C# to send the bitmap data of an image to the client, I plan to send this as an object. I want to understand the nature of web service.
My question is how does this work with a large file? such as 10MB? Does web service work in one single request/response to send all 10MB worth of bitmap data? What if a network error occurs? will the client have to reissue the request?
The reason I ask this is I have been working with Socket and often when it comes to large file, I break it down to trunk of manageable size (such as 4kb) and then send it, if one succeed then I'm sending the next 4kb until all the data are transferred.
Thanks again.
The same principle can be applied here. You can send it in chunks. Related post you might find usefull - wcf upload/download large files (i.e. Img, mp3) in chunks with windows service
How can I send the service a ref of a big file on the cliet's computer by a stream object and then start download piece by piece it from the client's computer (I decide how much MB I transfer every sec)? Do I have any limitations when I use it?
IIS doesn't support streaming - it buffers the whole request.
CodeProject article: WCF 4.5 fixes this
Until then, if you use IIS, the whole file will be stored in server memory before it is passed to your service.
The solution for now is to send the file in chunks - each chunk sent in a separate service call.
This would also help with your bandwidth throttling. This is not built into WCF - you have to do it yourself. You can throttle each chunk either on the client or on the server.
i have implemented a fairly simple wcf service which handles the file transfers from my clients to the server the problem is when a client sends a file request.
all of the bandwidth is allocated to that single client and others have to wait until the requested file transfer is completed.
So my question is how to make the service more efficient and let the users share the bandwidth
[ServiceBehavior(IncludeExceptionDetailInFaults = true, InstanceContextMode =InstanceContextMode.PerCall,
ConcurrencyMode=ConcurrencyMode.Multiple)]
I set the InstanceContextMode attribute to PerCall but that didn't do the trick
UPDATE : This Project is similar to mine
http://www.codeproject.com/Articles/33825/WCF-TCP-based-File-Server
WCF does not have proper load balancing, you will have to develop one yourself.
If you are transferring files, lets assume download, you should send packets of data rather than the complete file at once. When doing this, add 'delays/sleeps' to the process to limit the amount of bytes the server sends on each time window, this will make room for other requests.
It's questionable that it's desirable to serve up files through a WCF endpoint. The reasons against doing this are pretty much exactly the problems you have been having. It works for a few clients at a time - but scaling out requires hosting new instances of the service behind a load balancer.
It would be worth considering hosting your files with some kind of storage service and have your WCF service simply return a link or handle to the file. Then the file can be retrieved offline. Microsoft have created Azure Blob Storage for this exact purpose.
Appreciate this does not address your original question, and understand the scope of your requirement may not accommodate a large reworking.
Another option is to use chunking channel if you are transferring large files. Examples: MSDN, codeplex.
Although I agree with #hugh position.
I have quite some trouble creating a WCF service that supports downloads of very large files. I have read a lot of guides on how to set transferMode attribute to Streamed, increase all messageSize and bufferSize attributes to Int32.MaxValue, and still I have no luck. (I am also returning the stream as the message body via the MessageBodyMember attribute, and metadata is sent via the headers using the MessageHeader attributes).
If I set all these attributes, I can download smaller files fine, but when I try to download 1-2GB files I simply get a 400 bad request error which makes it rather hard to debug...
My service should ideally support file sizes of at least 8GB. Is this even doable with WCF? The various messageSize attributes of the web.config file seem to be limited to Int32.MaxValue which equals a maximum file size of 2GB.
From my studies I have found that it seems I will have to use WebClient.DownloadFile instead.
Files should only be available for download to users who have the required rights. With WCF my download method could take a token-parameter that the server could check and return the stream only if the user had rights to download the requested file. This does not seem straight forward using the WebClient approach. If anyone has some guidelines on how to do this (via the WebClient), I would very much appreciate it.
Ideally my WCF service should administer and provide user tokens and somehow bind to every individual file what tokens are currently legal (tokens should be usable only once). Download should then happen via the WebClient.
Thanks in advance for any clues.
You can do this in WCF. Many moons ago I built a service that did this (we didn't have a web server as part of our configuration). We used WCF streaming:
http://msdn.microsoft.com/en-us/library/ms733742.aspx
The strategy to deal with large payloads is streaming. While messages,
especially those expressed in XML, are commonly thought of as being
relatively compact data packages, a message might be multiple
gigabytes in size and resemble a continuous data stream more than a
data package. When data is transferred in streaming mode instead of
buffered mode, the sender makes the contents of the message body
available to the recipient in the form of a stream and the message
infrastructure continuously forwards the data from sender to receiver
as it becomes available.