Develop Reference

How to efficiently set the number of bytes to download for HttpWebRequest?

I'm currently working on a file downloader project. The application is designed so as to support resumable downloads. All downloaded data and its metadata(download ranges) are stored on the disk immediately per call to ReadBytes. Let's say that I used the following code snippet :-
var reader = new BinaryReader(response.GetResponseStream());
var buffr = reader.ReadBytes(_speedBuffer);
DownloadSpeed += buffr.Length;//used for reporting speed and zeroed every second
Here _speedBuffer is the number of bytes to download which is set to a default value.
I have tested the application by two methods. First is by downloading a file which is hosted on a local IIS server. The speed is great. Secondly, I tried to download the same file's copy(from where it was actually downloaded) from the internet. My net speed is real slow. Now, what I observed that if I increase the _speedBuffer then the downloading speed from the local server is good but for the internet copy, speed reporting is slow. Whereas if I decrease the value of _speedBuffer, the downloading speed(reporting) for the file's internet copy is good but not for the local server. So I thought, why shouldn't I change the _speedBuffer at runtime. But all the custom algorithms(for changing the value) I came up with were in-efficient. Means the download speed was still slow as compared other downloaders.
Is this approach OK?
Am I doing it the wrong way?
Should I stick with default value for _speedBuffer(byte count)?

The problem with ReadBytes in this case is that it attempts to read exactly that number of bytes, or it returns when there is no more data to read.
So you receive a packet containing 99 bytes of data, then calling ReadBytes(100) will wait for the next packet to include that missing byte.
I wouldn't use a BinaryReader at all:
byte[] buffer = new byte[bufferSize];
using (Stream responseStream = response.GetResponseStream())
{
int bytes;
while ((bytes = responseStream.Read(buffer, 0, buffer.Length)) > 0)
{
DownloadSpeed += bytes;//used for reporting speed and zeroed every second
// on each iteration, "bytes" bytes of the buffer have been filled, store these to disk
}
// bytes was 0: end of stream
}

After transmission of text files blank lines appears in them

EDIT: Filezilla caused the problem, when i download files back from server it added new lines. I'm sorry for confusion.
This method upload files to ftp server and it's work fine, but in text files uploaded to server blank lines appear after every line("cr lf" appear), for example:
File:
First line
Second line
Third line
Uploaded file:
First line
Second line
Third line
Origin and uploaded files accordingly have different sizes, non-text files are the same.
Code:
private void sendFile(string In, string Out)
{
FtpWebRequest request = (FtpWebRequest) WebRequest.Create("ftp://domain//" + Out);
request.Method = WebRequestMethods.Ftp.UploadFile;
request.Credentials = new NetworkCredential("username", "password");
FileStream sourceStream = new FileStream(In, FileMode.Open, FileAccess.Read, FileShare.Read);
byte[] fileContents = new byte[sourceStream.Length];
sourceStream.Read(fileContents, 0, (int) sourceStream.Length);
sorceStream.Close();
request.ContentLength = fileContents.Length;
Stream requestStream = request.GetRequestStream();
requestStream.Write(fileContents, 0, fileContents.Length);
requestStream.Close();
}
How can i fix this?

EDIT: As the answer below doesn't seem to have helped (but I'm leaving it there for posterity as it shows better code) here are the next diagnostics steps I'd check:
How are you viewing the files? If at all possible, get onto the server directly rather than fetching the files again via a web browser or whatever.
What's the type of FTP server you're connecting to? Maybe there's a known issue.
Have you tried looking at what's actually being sent via Wireshark?
Have you tried sending the same files via a normal FTP client?
You should set FtpWebRequest.UseBinary to true in order to preserve the exact file contents. Otherwise the two systems will try to figure out line endings themselves, changing line terminators as they see fit. I very rarely think that's a good idea. (EDIT: UseBinary is actually true by default, but this sounds like the kind of problem introduced by using text mode... it certainly does no harm to make this explicit.)
Additionally:
You should be disposing of your FileStream via a using statement
You should be disposing of the request stream via a using statement
You should be taking note of the result of Stream.Read - it needn't always read the whole of the requested data in one go
You can either use File.ReadAllBytes to simply read the complete file data in one go, or use Stream.CopyTo (if you're using .NET 4) to copy the FileStream to the request stream (which won't set the content length, of course; I don't know whether this is a problem)
You're never calling GetResponse; it's unclear exactly what happens if you never fetch the response of an FtpWebRequest
Your parameter names don't match .NET naming conventions, and aren't very descriptive
So I would probably use:
private void SendFile(string inputFile, string outputPath)
{
FtpWebRequest request = (FtpWebRequest) WebRequest.Create
("ftp://domain//" + outputPath);
request.Method = WebRequestMethods.Ftp.UploadFile;
request.UseBinary = true;
request.Credentials = new NetworkCredential("username", "password");
byte[] fileContents = File.ReadAllBytes(inputFile);
request.ContentLength = fileContents.Length;
using (Stream requestStream = request.GetRequestStream())
{
requestStream.Write(fileContents, 0, fileContents.Length);
}
// This *may* be necessary in order to validate that everything has happened
using (WebResponse response = request.GetResponse())
{
}
}

Its strange. I face the same problem and I was unable to fix it until I did not provide an extension in file. For Example if my file name was
abcfile
then I make it abcfile.dat and after that it shows me the uploaded file as actual file. I again upload file with abcfile.txt but this time again empty line problem appear in my uploaded file.
I suggest that you must provide extension to your file any except .txt.

The system that you're sending to uses different line endings to what your system uses. I can assume, because you get an extra line, that you're on Windows, and it uses CRLF endings. The system you're sending to recognises CR and LF as separate endings, so you get the extra lines.
For text, truncate the LF or the CR, see what happens. I have no clue about the differing file sizes.

In the top menu of FileZilla, set:
Transfer menu > Transfer type > binary

In the top menu of FileZilla, set:
Transfer menu > Transfer type > binary
It's working for me.

Record HTTP Audio stream to file

In C# I want to record an audio stream
I am doing something along the lines of:
HttpWebRequest req;
req = (HttpWebRequest)WebRequest.Create("http://url.com/stream");
Webresponse resp = req.GetResponse();
Stream s = resp.GetResponseStream();
fs = File.Exists(fileName)
? new FileStream(fileName, FileMode.Append)
: new FileStream(fileName, FileMode.Create);
byte[] buffer = new byte[4096];
while (s.CanRead)
{
Array.Clear(buffer, 0, buffer.Length);
total += s.Read(buffer, 0, buffer.Length);
fs.Write(buffer, 0, buffer.Length);
}
and the file size grows but can't be played back by VLC or any other program.
This is not my exact code I do a lot of error checking etc, but this gives the general idea.

Array.Clear(buffer, 0, buffer.Length);
total += s.Read(buffer, 0, buffer.Length);
fs.Write(buffer, 0, buffer.Length);
You do not have to clear the whole array before you read - there's no point in doing this. But you have to check how many bytes you actually read, there's no guarantee the whole array is filled every time (and it probably won't):
int bytesRead = s.Read(buffer, 0, buffer.Length);
fs.Write(buffer, 0, bytesRead);
total+=bytesRead;
Also whether the file plays (even when it is not corrupted anymore once you fix the file writing code) back depends on the format that you are downloading - what codec / file type is it?

THe problem is the streamed bits don't have context. When you stream to an application, there is a tacit agreement that you are dealing with file type X and the streaming program then tries to play the bits.
WHen you stream to a file, you have to add the context. One of the most important bits is the header identifying the type of file and other information.
If you can add the header, you can play the file from the file system. The header will not be part of the stream, as the server and client have agreed on what type fo file it it is already.
If you create a streaming player, you can possibly play back the bits you saved, as you negotiate the type. BUt to have it automagically work from file, you have to add the header.

Trying to save streamed MP3 audio to disk is essentially impossible without a detailed understanding of both the stream format and the file format for MP3. What you're getting from the stream is a series of "windowed" chunks of audio converted to frequency domain; the player receiving the stream converts the chunks back into time-domain audio on the fly and plays them one after the other.
To make an MP3 file, you would have to first write out a header containing the format information and then write each chunk of data. But most likely the format for storing these chunks in a file is different from the way in which they're compacted into a stream.
Sorry, but I would seriously advise you to give this up. One major reason that music services stream instead of offering file downloads is specifically because it's so difficult to save an MP3-type stream to disk (it would be a trivial matter to save an uncompressed audio stream to a WAV file).

How to read the response stream before the Http response completes

When making a request using HttpWebRequest object, I need to call the method GetResponse() to send the request and get the response back.
The problem with this method is that it doesn't return the response object until all data has been received. Say I am downloading a 100 MB file, I won't be able to read it until the response finish and all the 100 MB is downloaded.
What I want is to be able to read the response stream bytes as soon as they arrive, without waiting for the response to complete.
I know I can use the Range Http header, but it won't work on my situation.

I think this is very close to what #Zachary suggests. And it (seems to) work(s); actually I think using using as #Zachary does is even "nicer".
My main point being I cannot see the blocking behaviour of GetResponse() you (seem to) describe.
In addition the following code only roughly shows how everything works; it will not read the stream to the end for example (unless by coincidence :)). But it should work if you copy-n-paste it into an empty "Console Application"-project in Visual Studio.
You can try using some "shorter" URL for a test. The example here starts downloading an ISO of the debian distribution (a bit more than 600 MByte). Sorry debian, I did not mean to steal your bandwidth. -> Btw: is there something sensible one can use to test such a scenario?
The Code is strongly inspired by C# - How to read a continuous stream of XML over HTTP.
namespace StreamReadWebRequest
{
using System;
using System.Collections.Generic;
using System.Text;
using System.Net;
using System.IO;
class Program
{
static void Main(string[] args)
{
HttpWebRequest req;
HttpWebResponse res = null;
try
{
req = (HttpWebRequest)WebRequest.Create(
"http://cdimage.debian.org/debian-cd/5.0.4/i386/iso-cd/debian-504-i386-CD-1.iso");
res = (HttpWebResponse)req.GetResponse();
Stream stream = res.GetResponseStream();
byte[] data = new byte[4096];
int read;
while ((read = stream.Read(data, 0, data.Length)) > 0)
{
Process(data, read);
}
}
finally
{
if (res != null)
res.Close();
}
Console.In.Read();
}
private static void Process(byte[] data, int read)
{
Console.Out.Write(ASCIIEncoding.ASCII.GetString(data));
}
}
}

I was looking for the same thing: server streams chunked XML data and I needed a C# client that could access this data while server is streaming. I tried many different ways to access the source (WebChannelFactory, WebClient, HttpWebRequest/Response, TcpClient) but failed so far. Finding this thread I focused on HttpWebRequest/Response where I have the same problem that following line is blocking:
HttpWebResponse resp = (HttpWebResponse)request.GetResponse();
As Artiom Chilaru stated, if it's blocking: something is wrong, because it should not. Now focusing on trying to replicate default behavior with downloading large .ISO files I found out that Fiddler was blocking the GetResponse() method!
However there is no problem to open Fiddler once the stream has been set up (i.e. GetResponse() has already been called), but during the HTTP GET if you find GetResponse() is blocking try to close Fiddler and see if your application now continuous it's normal flow (i.e. reading the stream).

If you set the buffer size on your read, you can read in the data in chunks... example...
// Get the response stream
using(Stream resStream = response.GetResponseStream())
{
string parseString = null;
int count = 0;
do
{
// Read a chunk of data
count = resStream.Read(buf, 0, buf.Length);
if (count != 0)
{
// Convert to ASCII
parseString = Encoding.ASCII.GetString(buf, 0, count);
// Append string to results
sb.Append(tempString);
}
}
while (count > 0);
}

I'm not sure what you have on your side, but I know for a fact (and I'm sure many people will agree here) that GetResponse() will NOT download the whole file back. It will send the request, wait for the response, and get the response headers.
After you have the response, you can easily get the response stream with GetResponseStream(), which is the actual data stream that's downloading from the server. And you can easily access the response stream BEFORE the whole file is downloaded. This is 100% true and tested.
If you're not getting the same behaviour (which is really strange, and shouldn't happen) could you add a code example that is not working as I explained above?
Also, do test the example posted by scherand. It just proves once again that it works just fine, without any special hacks.

Why the buffer size changes my stream output?

I'm trying to stream a pdf file. Most of the files open without any problems but sometimes it fails. When it fails, it also looks like file size is smaller than the original one. For example, I was trying to open a 47K file but when the streamed output to the browser it's only 44.5K. When check the size of the stream (result.FileStream), it's 47K like it supposed to be.
I'm using Stream.Read to output the file to the browser. When I had a problem, I was using buffer size of 10000 bytes. However, when I changed the buffer size from 10000 to 1000 the problem disappear and I was able to the file. I cannot explain why the change in the buffer size makes the streaming behave differently.
Here's the code I'm using result.FileStream is of type Stream:
using (result.FileStream)
{
int length;
const int byteSize = 1000;
var buffer = new byte[byteSize];
while ((length = result.FileStream.Read(buffer, 0, byteSize)) > 0 && Response.IsClientConnected)
{
Response.OutputStream.Write(buffer, 0, length);
Response.Flush();
}
}
Response.Close();
Please enlighten me because I definitely don't understand something.

You're using Response.Close(), which seems to be much more evil then the documentation would make you believe.
http://forums.iis.net/t/1152058.aspx