I am quite new with .NET, C# and WCF and I am trying to create a service that will expose methods to allow upload and download of a large amount objects (array of objects). I have seen a lot of posts regarding large file transfers in WCF, but I can't seem to find anything that focuses on just uploading and downloading large amounts of serializable objects.
I have been able to "hack up" the web.config file for the allowed bytes and timeout limit etc, but I am wondering if there are actually better ways to configure the WCF for better speed and memory usage to allow for such transmission. Because I have configured the web.settings based on my testing results (if time out/byte limit exceeded, increase the limit with some crazy large numbers etc) I doubt my configuration is making sense at all.
Second, I have seen some implementation options like having binding TransferMode = Streaming or MTOM, yet I don't know if they will apply to my scenario at all. Can someone please point me to the correct direction?
Sorry I might not have constructed my question well enough, but ideas will be much appreciated.
Below is my web.config settings:
<system.web>
<httpRuntime maxRequestLength="409600000" executionTimeout="360000"/>
<compilation debug="true" targetFramework="4.0" />
</system.web>
<system.serviceModel>
<bindings>
<basicHttpBinding>
<binding name="BasicHTTP" closeTimeout="00:01:00" receiveTimeout="01:00:00"
sendTimeout="01:00:00" maxBufferSize="655360000" maxReceivedMessageSize="655360000"
messageEncoding="Text" />
</basicHttpBinding>
</bindings>
<services>
<service behaviorConfiguration="ServiceBehavior" name="WebService.WebService">
<endpoint address="" behaviorConfiguration="BasicEndPoint" binding="basicHttpBinding"
bindingConfiguration="BasicHTTP" name="BasicHTTP" contract="WebService.IWebService" />
</service>
</services>
<behaviors>
<endpointBehaviors>
<behavior name="BasicEndPoint">
<dataContractSerializer maxItemsInObjectGraph="65536000" />
</behavior>
</endpointBehaviors>
<serviceBehaviors>
<behavior name="ServiceBehavior">
<serviceMetadata httpGetEnabled="true" />
<serviceDebug includeExceptionDetailInFaults="true" />
</behavior>
</serviceBehaviors>
</behaviors>
<serviceHostingEnvironment multipleSiteBindingsEnabled="true" />
</system.serviceModel>
<system.webServer>
<security>
<requestFiltering>
<requestLimits maxAllowedContentLength="204800000" />
</requestFiltering>
</security>
<modules runAllManagedModulesForAllRequests="true" />
</system.webServer>
I would recommend to use Stream mode only for tasks like video/audio streaming or similar, when you really have stream on host and you really need receive stream on client.
For any other tasks i would use Buffer mode because it is really easier to use and becase many useful wcf features rely on buffering. Using streaming with enabled SOAP message-level security (for example), can probably eliminate speed benefits of streaming mode (if you ever had some).
In your case i would recommend you to perform some workaround. Check these steps:
Try to use compression. It can greatly improve speed and resource usage, but it can not guarantee that your amount of data always will be the same. Who knows if your array will be 100x larger next year? So you may return to this point one year later.
Replace your method returning one giant array with method returning only one element or a small range of them. Then replace one GetAllStuff() method call with one call of GetItemsCountForDownload() and appropriate number of calls of GetElementAtIndex(int index) (or GetElementsRange(int startIndex,int endIndex)). Anyway, at this poin you will have to find some balance between message size (for each call) and number of calls performed.
If your data can not be easily splitted into small equal parts (e.g. first element of array is 10kb, second is 15 kb and third is 338mb), then try to combine both methods together: serialize your array to disk, compress it whith splittig to some number of parts of acceptable size and then trasfer them one after another.
Try to make parameters of your algorithm ajustable. Put them in your config file, so it will be possible to tune splitting process and compression level on every deployment machine according to avaliable resources.
Another benefit of splitting data and downloading chunk by chunk easiness to build some error-handling layer. If connection is unstable and transfer fails for some reason you can try to redownload only one chuk instead of all the data.
Some tips on architecture
Find appropriate crossplatform compression algorithm. Definitely you can find one that meets your needs. Look at BZip2 for C# <-> Python and GZip for C# <-> Java.
Try to make your architecture clear enough for other programmers. You can make different methods for compressed data transfer and for uncopressed (GetElementsRange, GetElementsRangeGZip, GetElementsRangeBZip2). Or you can make one method with compression type paramether (GetElementsRange(int startIndex,int endIndex,string compressionType). Anyway, other programmer must be able understand what data he is receiving and how to control compression mode.
You can move data splitting parameter from config file to method definition, so client will be able to define it by himself.
You can go further and implement two step architecture. Step one: remote client defines parameters of request (all of them, including compression mode, splitting mode and all other). Step two: receiving data. How about tokens? Methods should look like:
string GetTokenForDataRequest(string compressionMode, int maxChunkSize); //additional parameters like dates range, account number an other are defined in this method only
int GetChunkCount(string token);
byte[] GetDataChunkAtIndex(string token, int index);
Here compressionMode may be "None","BZip2" or "GZip"; If maxChunkSize is 0 then disable splitting (all data will be sent in one chunk), else split data in chunks with size equal to maxChunkSize (the last one will be smaller than other). So approximate scenario would be like:
Remote client sends a data request with desired parameters.
You generate session id (a token), save parameters for this session and prepare data according to parameters. "Prepare data" means loading data according to request, creating temp folder, serializind data, creating chunk files and persisting paths for further usage.
Client recieves a token, that is used for all other methods for retrieving data.
Every time remote client request data chunk throu your methods you will know where it stored on hard disk, how many chunks left and so on - thanks to provided token and persisted info.
You can easily handle simultaneous data transfer session with several clients at one time with no efforts (just make shure you store data chunks in different temp files and folders)
Client is able to redownload any chunk and you will not have to load data from database (or wherever it comes from).
When transfer is complete, you can wipe temp data and free some resources.
Do not look at this as at only possible solution. Just google around a little bit and you will find your very own way to solve your task!
You can use streaming message transfer, but it has some limitations.
If the streaming isn't suitable for you, you can implement paging of data (or try existing implementations, like in WCF Data Services).
Related
I keep getting a SystemOutOfMemoryException when processing large files like 1.6GB in size. I was just wondering if my configuration in web.config is correct. Below are the lines of code I modified to support large files that is more than 1GB in size
<httpRuntime requestValidationMode="2.0" enableVersionHeader="false" maxRequestLength="457286400" executionTimeout="3600" targetFramework="4.5" fcnMode="Single" />
<security>
<requestFiltering>
<requestLimits maxAllowedContentLength="3294967295" />
</requestFiltering>
</security>
I'm not sure if this is correct. I also changed the IIS server to run in x64 via Tools-> Options-> Project and Solutions-> Web Projects -> "Use the 65bit version of IIS"
I'm not sure what I'm missing but I keep getting the system out of memory exception even though I already increased the memory size
UPDATE:
I'm processing a large files like .mp4 files or just anytype of file regardless of type as long as the client requires it.
Also I'm not uploading the file using a frontend. What we do is upload the large file via Filezilla and if the file is uploaded to the server we use the Umbraco MediaService to process this file to have an entry in the Media Page. We are also using Hangfire to trigger our background service to process the MediaService
Accepting large files like this, is not a best practice. Eg this gives the possibility to perform a Ddos attack. Also you will need a lot of memory as the file will be handled in memory before persisting it to disk, hence the issue you have with memory.
The better solution would be to chunk them with javascript and send over small packets, and the stitch them back together server side. There is a pretty good explanation on how to build this yourselves : https://www.dotnetcurry.com/aspnet-mvc/893/upload-big-files-aspnet-mvc-azure-storage
(depending on how you handle your sessions, adding it to the session might not resolve your memory issue. I guess you'll need to save it to disk)
Unrelated to the memroy exception, but make sure you wrap this in a <location path="xxx/yyy"> your web.config stuff </location> so that only this URL will accept large files.
Managed to solve this issue by adding a 2nd and 3rd parameter in my Umbraco MediaService. Initially I was using Service.MediaService.Save(iMediaFile) but it seems that when using large file it needed to know who uploaded the file hence it needs the 2nd parameter which is the user_id and I added the third parameter to suppress any error message. As I was encountering an out of memory exception where in fact the error was I didn't pass the user_id in the second paramter. Hence my new code became Service.MediaService.Save(iMediaFile, user_id, false)
Again the solution was not really a configuration as my configuration was enough to process the file. The error was due to missing 2nd parameter
I have a standard Web Service that processes JSON requests via an webHttpBinding. I am trying to find out whether there is a limit on how many concurrent connections it can handle and how to control it. Can't find anything. I am missing something simple or some setting?
Here is a skeleton of my service:
[ServiceContract]
public interface IMyService {...}
[ServiceErrorBehavior(typeof(ErrorHandler))]
[AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
public class MyService : IMyService {...}
I have some suggestion for this, but this will not be direct config related change. Hold on and read this.
In WCF, there is InstanceContextMode and ConcurrencyMode properties defined for ServiceBehavior. These parameters are configurable only within the service code, and not in XML configuration, because they relate to the runtime behavior of the service and the service developer should be aware of their values. The InstanceContextMode parameter determines how many instances of the service have to be created by the WCF runtime. The possible values are:
PerCall: a new InstanceContext object is created for each call.
PerSession: a new InstanceContext object is created for each session. If the channel does not create a session this value behaves as if it were PerCall.This is the default value.
Single: only one InstanceContext object is used for all incoming calls and is not recycled subsequent to the calls. If a service object does not exist, one is created.
More helpful Blog
Use WCF Service Throttling to control the service.
<behaviors>
<serviceBehaviors>
<behavior name="Throttled">
<serviceThrottling
maxConcurrentCalls="1"
maxConcurrentSessions="1"
maxConcurrentInstances="1"
/>
</behavior>
</serviceBehaviors>
</behaviors>
Always remember that service behaviors do not override limits defined in the host (or binding). For example when using webHttpBinding the default IIS Maximum Concurrent Connections would likely need to be changed for large concurrency values.
Turns out it has more to do with threading than anything else. The default ASP.NET settings are pretty conservative, so you have to hike them up. Once I did that, the concurrent connections bottleneck completely disappeared.
Make sure you have the following in the appropriate machine.config (not web.config):
<configuration>
<system.net>
<connectionManagement>
<add address="*" maxconnection="100" />
</connectionManagement>
</system.net>
<system.web>
<processModel
autoConfig="true"
maxWorkerThreads = "100"
maxIoThreads = "100"
minWorkerThreads = "50"
minIoThreads = "50"
/>
<httpRuntime
minFreeThreads="176"
minLocalRequestFreeThreads="152"
/>
</system.web>
</configuration>
I took all this info from Tuning IIS article by Stuart Brierley. The only thing I changed significantly from his recommendations is maxConnection value.
I have an application written in c# on .NET 4.0 which needs to make multiple web service requests. The web service requests vary in nature but are mostly requesting information.
The Types involved is a derivative of System.ServiceModel.ClientBase
The connection is setup in code and uses types such as BasicHttpBinding, EndpointAddress, and CustomBinding to name a few.
How can I determine the max number of concurrent requests that can be made on the derivative of the ClientBase?
I've not been able to find any property that pertains to MaxConnections but I do come across things like NetTcpBinding.MaxConnections and ConnectionManagementElement.MaxConnection but neither of these seem compatible with my leveraged APIs. Either I'm missing how to use them, this isn't available or I don't know where to look.
WCF is an abstraction on core networking concepts. For HTTP bindings, it falls under the ServicePoint configuration which determines things like your HTTP concurrent connection limits.
You want ServicePointManager.DefaultConnectionLimit for HTTP:
http://msdn.microsoft.com/en-us/library/system.net.servicepointmanager.defaultconnectionlimit.aspx
You can also do this via your config file:
http://msdn.microsoft.com/en-us/library/fb6y0fyc.aspx
This would be in the binding configuration section of the service host's .config file. Depending on the binding being used, you can set things like maxConcurrentCalls and maxConcurrentSessions, there are usually default limits for them imposed by WCF.
Real life example:
<system.serviceModel>
<behaviors>
<serviceBehaviors>
<behavior name="ServiceBehaviorBasicHttp">
<serviceThrottling maxConcurrentCalls="1000" maxConcurrentSessions="1000" maxConcurrentInstances="1000"/>
<serviceMetadata />
</behavior>
</system.serviceModel>
Or in code behind, something like this:
ServiceHost host = new ServiceHost(typeof(MyService));
ServiceThrottlingBehavior throttleBehavior = new ServiceThrottlingBehavior
{
MaxConcurrentCalls = 40,
MaxConcurrentInstances = 20,
MaxConcurrentSessions = 20,
};
host.Description.Behaviors.Add(throttleBehavior);
host.Open();
Taken from here: WCF: How do I add a ServiceThrottlingBehavior to a WCF Service?
I'm trying to create a web service using Azure.
For the time being, everything is being run locally. The web service and Azure work fine, a simple string Test() method which returns "Hello world" works without problems, as you'd expect. ;)
Now, I've created two methods which add rows to Azure Data Tables. The first sends (using a special DataContract) a single row of data, and this works fine.
The second is for sending larger amount of data, and sends an IEnumerable. So, to test the service, I've created a client application which creates a number of random data to send. If I create up to 42 rows and send that, all goes well.
Above that, I get a 400 Bad request error.
The problem is that there's no inner message to work with (or rather, that WAS the inner message). I strongly suspect it has to do with the size of the request however.
Note, that if I put a breakpoint on the service's method, it doesn't even get that far. I've read quite a few various forum posts regarding similar issues, but those seemed to deal with ordinary WCF services, not Azure ones, and so the Web.config file doesn't contain definitions for bindings nor endpoints, which would be something I could work with.
Please help.
PS. I realise I may have posted very little information. If something else is needed, please ask, and I'll do my best to include it.
Adding the following lines to the Web.config file (under system.serviceModel) in the Azure service project (NOT the Web.config in the client application) resolved the issue:
<bindings>
<basicHttpBinding>
<!--The basicHttpBinding is used for clients which use the generated code to transmit data; the following
settings make it possible to send larger amounts to the service-->
<binding maxReceivedMessageSize="10000000" receiveTimeout="01:00:00">
<readerQuotas maxStringContentLength="10000000" />
</binding>
</basicHttpBinding>
</bindings>
I am looking into using WCF for a project which would require the ability for people to upload large files (64MB-1GB) to my server. How would I handle this with WCF, possibly with the ability to resume uploads.
In order to handle a larger client base, I wanted to test out JSON via WCF. How would this affect the file upload? Can it be done from JSON, or would they need to switch to REST for the upload portion?
If you want to upload large files, you'll definitely need to look into WCF Streaming Mode.
Basically, you can change the transfer mode on your binding; by default, it's buffered, i.e. the whole message needs to be buffered on the sender, serialized, and then transmitted as a whole.
With Streaming, you can define either one-way streaming (for uploads only, for downloads only) or bidirectional streaming. This is done by setting the transferMode of your binding to StreamedRequest, StreamedResponse, or just plain Streamed.
<bindings>
<basicHttpBinding>
<binding name="HttpStreaming"
maxReceivedMessageSize="2000000"
transferMode="StreamedRequest"/>
</basicHttpBinding>
</bindings>
Then you need to have a service contract which either receives a parameter of type Stream (for uploads), or returns a value of type Stream (for downloads).
[ServiceContract]
public interface IFileUpload
{
[OperationContract]
bool UploadFile(Stream stream);
}
That should do it!
MTOM is optimized to handle large binary data.
You can use webHttpBinding with TransferMode streamed and a single Stream parameter or Stream response (as appropriate) for large file up/downloads, but you'd have to send any request metadata via URLs and/or headers, unless you're going to devise your own framing on the Stream. You'll have to build a custom non-HTML client (like Silverlight, Flash, etc) though, since browsers don't support random access to local files, and the normal file upload will be a form post, not JSON.