I want to stream data from an IHttpHandler class. I'm loading a large number of rows from the DB, serializing, and compressing them, then sending them down the wire. On the other end, I want my client to be able decompress, and deserialize the data before the server is even done serializing all the objects.
I'm using context.Response.OutputSteam.Write to write my data, but it still seems like the output data is being put into a buffer before being sent to the client. Is there a way to avoid this buffering?
The Response.Flush method should send it down the wire; however, there are some exceptions. If IIS is using Dynamic Compression, that is it's configured to compress dynamic content, then IIS will not flush the stream. Then there is the whole 'chunked' transfer encoding. If you have not specified Content-Length then the recieving end does not know how large the response body will be. This is accomplished with the chunked transfer encoding. Some HTTP servers require that the client uses an Accept-Encoding request header containing the chunked keyword. Others just default to chunked when you begin writing bytes before the full length is specified; however, they do not do this if you have specified your own Transfer-Encoding response header.
With IIS 7 and compression disabled, Response.Flush should then always do the trick, right? Not really. IIS 7 can have many modules that intercept and interact with the request and response. I don't know if any that are installed/enabled by default, but you should still be aware that they can effect your desired result.
... I'm loading a large number of rows from the DB, serializing, and compressing them, then sending them down the wire...
Curious that you are compressing this content. If you are using GZIP then you will not be in control of when and how much data is sent by calling flush. Additionally using GZIP content means that the receiving end may also be unable to start reading data right away.
You may want to break the records into smaller, digestible chucks of 10, 50, or 100 rows. Compress that and send it, then work on the next set of rows. Of course now you will need to write something to the client so they know how big each compressed set of rows is, and when they have reached the end. see http://en.wikipedia.org/wiki/Chunked_transfer_encoding for an example of how the chunked transfer works.
You can use context.Response.Flush() or context.Response.OutputSteam.Flush() to force buffered content to be written immediately.
Related
I looked online and couldn't find a proper explanation.
Link I checked: Difference
This link says:
If you are trying to upload large files (like images or documents) you
need to be aware that you may need to adjust your maxRequestLength.
Then if files are really big you may need to adjust the
maxAllowedContentLength.
But both sentence mean the same and I am confused.
Another link: Difference
This says
The maxRequestLength indicates the maximum file upload size supported
by ASP.NET, the maxAllowedContentLength specifies the maximum length
of content in a request supported by IIS. Hence, we need to set both
maxRequestLength and maxAllowedContentLength values to upload large
files.
My Question is: If I have a file upload of 10GB. Is my content 10GB or is my FileSize 10GB? I don't understand what the difference is between file size being uploaded and the content size?
Bottom Line: Please tell me in layman terms if I have a file upload of 10GB how these two parameters come into picture.
Request consist of headers and body (that provides encoded content of the file in your case). So request length is total size of the request, content length is size in bytes of the body (which is likely more than size of data you are sending).
Fake sample:
User-agent: Bob the builder the 4th
Authorization: hereIcome
Content-length: 4
Content-Encoding: Base64
BEEF
So request length here is about 100, content length is just 4 (length of "BEEF") but actual data is 3 bytes (FromBase64String("BEEF") - 0x04 0x41 0x05).
For case of huge files size of headers can be ignored and both maxRequestLength and maxAllowedContentLength set to the same very high value. Depending on encoding used to send files the values need to be some multiplier of max size of the file.
These settings differ both in semantics as well as usage.
maxAllowedContentLength
This is a IIS specific setting. Any request you send is going to be handled by IIS first, irrespective of the fact whether it is going to be handled by your application or any other. So, if you imagine the web server as a building, this is going to be your entry gate to the building. And as mentioned by #Alexei, this considers only the content or payload size and is measured in bytes. If you send a request whose payload size exceeds this limit, you are going to get a Http 404.13 error response (http response 404 with a subcode of 13. You can check the different IIS status codes in this link).
maxRequestLength
In comparison, maxRequestLength is an ASP.Net specific setting, which defines the buffering threshold of the input stream. So, in the building example, this is the door of an apartment and hence is apartment specific. So, your request has to fit through both the building door and the apartment door. And this considers the entire request length, not just the payload, and is measured in kbs. If your request passes the IIS setting and fails to pass here due to size, you will get a Http 500 error.
I'm currently trying to program my own HttpWebRequest class. I've already written the code that sends the header and the body of the request to the server and awaits a response. However, I am unsure which charset I should use for the header.
I've also been wondering what would be a good way of processing the response (header + body). Should I try to decode all received data into a string, or should I do it differently? I was thinking of splitting the header from the body using the two line feeds/carriage returns that separate those two parts. Then I would be able to decode the header and leave the body for later, when I know its charset.
So my questions in short:
What charset does HTTP use for its headers?
What's a good way of processing the response?
First I would recommend that you become intimately familiar with RFC-2616 which is the RFC for the HTTP 1.1 protocol.
From the above RFC you will find the following statement
The TEXT rule is only used for descriptive field contents and values
that are not intended to be interpreted by the message parser. Words
of *TEXT MAY contain characters from character sets other than ISO-
8859-1 [22] only when encoded according to the rules of RFC 2047
[14].
The headers should use ISO-8859-1 encoding unless encoded using the MIME encoding outlined in RFC-2047.
As for the parsing of the response, that really depends on the message. Personally, I would processes the response based on the BNF defined for HTTP as I identify tokens that I recognize I would update the state of the parser to process the rest of the response accordingly. For example as the response data is processed you might find that the response is a JPG image and the the content length is X, so you can setup the appropriate memory stream to read the content into and then create an Image etc.
I tried send some data(variable 1 - 4 MB) by http headers,but returned the following error in ajax.response:
HTTP Error 400. The size of the request headers is too long.
there something that I can do or the method single is compressing the data? if yes,how I do this?
any help is appreciated. Thanks in advance!
If you're sending around that much data, put it in the body of the request (e.g, in a HTTP POST), not in the headers. Increasing the header size limit (as cwallenpoole suggests) will still cause problems with users who are behind web proxies.
Most http server accept about 8-16KB for the header. Therefore, if your data is too large, just use POST method to send it.
Personally, I would just up the size of the acceptable header. MS suggests the same and gives instructions on how to raise it to 16 MB if necessary (see MaxRequestBytes).
I am trying to figure out how to get a file that I have stored in my Database, onto the filesystem. I have looked everywhere, but all I can find is like for images pulling them out and serving them via Response. Can anyone point me in the direction of a tutorial or provide me with some input on which way to go?
From the perspective of serving the file through the web, the process is exactly the same for images as it is for any file type. The only difference would be the content type in the response header. (For a PDF, it would be application/pdf.)
Understand that over HTTP there is absolutely no concept of "files." All you have are requests and responses, each of which has headers and data. Whether it's an image, a PDF, a web page, anything... the structure of the response being delivered to the client (browser) is exactly the same. Headers tell the browser what kind of data it is and various other meta-details about it, and the data is just the raw data of the "file" being sent. The browser can choose what to do with the data.
In the case of web pages, the browser generally just renders it. In the case of other or unknown content types, the browser generally prompts the user to save it as a file. (Note that a file name can also be suggested in the header of the response.)
Edit: In response to your comment on #Pete M's answer...
The process is still very similar. "Files" as email attachments work in much the same way as "files" over HTTP. Essentially, all you need is the data stream and the content type. When building the MailMessage object, take a look at the Attachment object. As you pull your PDF file from the database (just like when pulling images from the database to serve to a browser client), set the data stream to the ContentStream on the attachment object. Then also set the ContentType (again, application/pdf) and the Name on the attachment object and you should be able to send it as part of the MailMessage.
You may need to set more options on the Attachment as well. Testing will indicate how explicit you need to be with things like encoding. If you can share some code as you attempt this, we can help you with that code.
It's no different than writing out any other kind of file. If you find a good "read/write text file" tutorial it's going to be the exact same process, you'll just get your initial stream from the database.
Can I ask why you are doing this? If it's for caching so you can return it to users who ask for it I would consider other options first. Requiring a dip into disk will often cause a bottle neck in performance.
I have a device that uses a custom scripting language. This language provides the capability to set http request headers then perform a post to a URL. The server that the device communicates with is running .net, and I would like the handler on the server to retrieve data from the device by simply pulling it from the Request.Files[] collection. To do this, does the device need to support multipart posting or is it possible for the uploaded content to show up in the Files collection by simply setting a number of headers then sending the data?
I'm not a linux guy, but I did a search and I found this. Does it help at all? (see section 4.3)
http://curl.haxx.se/docs/httpscripting.html
If you mean that at the server end you want the file(s) to be readable via HttpRequest.Files, then I believe so.
I think the format that this expects is based on the File Upload RFC and therefore if you follow the standard procedure for submitting a file upload then it'll work.
As to how you go about formatting the request from your device - I'm afraid I can't help on that one. Assuming you can format an Http Post, writing any headers you need and then formatting the Post body with the file content, it'll work, though.
In order for .Net to be able to read a file posted to a header it must be in the multipart posting format. Otherwise it will just be text noise in the header and .Net won't know what to do with it.