When making a request using HttpWebRequest object, I need to call the method GetResponse() to send the request and get the response back.
The problem with this method is that it doesn't return the response object until all data has been received. Say I am downloading a 100 MB file, I won't be able to read it until the response finish and all the 100 MB is downloaded.
What I want is to be able to read the response stream bytes as soon as they arrive, without waiting for the response to complete.
I know I can use the Range Http header, but it won't work on my situation.
I think this is very close to what #Zachary suggests. And it (seems to) work(s); actually I think using using as #Zachary does is even "nicer".
My main point being I cannot see the blocking behaviour of GetResponse() you (seem to) describe.
In addition the following code only roughly shows how everything works; it will not read the stream to the end for example (unless by coincidence :)). But it should work if you copy-n-paste it into an empty "Console Application"-project in Visual Studio.
You can try using some "shorter" URL for a test. The example here starts downloading an ISO of the debian distribution (a bit more than 600 MByte). Sorry debian, I did not mean to steal your bandwidth. -> Btw: is there something sensible one can use to test such a scenario?
The Code is strongly inspired by C# - How to read a continuous stream of XML over HTTP.
namespace StreamReadWebRequest
{
using System;
using System.Collections.Generic;
using System.Text;
using System.Net;
using System.IO;
class Program
{
static void Main(string[] args)
{
HttpWebRequest req;
HttpWebResponse res = null;
try
{
req = (HttpWebRequest)WebRequest.Create(
"http://cdimage.debian.org/debian-cd/5.0.4/i386/iso-cd/debian-504-i386-CD-1.iso");
res = (HttpWebResponse)req.GetResponse();
Stream stream = res.GetResponseStream();
byte[] data = new byte[4096];
int read;
while ((read = stream.Read(data, 0, data.Length)) > 0)
{
Process(data, read);
}
}
finally
{
if (res != null)
res.Close();
}
Console.In.Read();
}
private static void Process(byte[] data, int read)
{
Console.Out.Write(ASCIIEncoding.ASCII.GetString(data));
}
}
}
I was looking for the same thing: server streams chunked XML data and I needed a C# client that could access this data while server is streaming. I tried many different ways to access the source (WebChannelFactory, WebClient, HttpWebRequest/Response, TcpClient) but failed so far. Finding this thread I focused on HttpWebRequest/Response where I have the same problem that following line is blocking:
HttpWebResponse resp = (HttpWebResponse)request.GetResponse();
As Artiom Chilaru stated, if it's blocking: something is wrong, because it should not. Now focusing on trying to replicate default behavior with downloading large .ISO files I found out that Fiddler was blocking the GetResponse() method!
However there is no problem to open Fiddler once the stream has been set up (i.e. GetResponse() has already been called), but during the HTTP GET if you find GetResponse() is blocking try to close Fiddler and see if your application now continuous it's normal flow (i.e. reading the stream).
If you set the buffer size on your read, you can read in the data in chunks... example...
// Get the response stream
using(Stream resStream = response.GetResponseStream())
{
string parseString = null;
int count = 0;
do
{
// Read a chunk of data
count = resStream.Read(buf, 0, buf.Length);
if (count != 0)
{
// Convert to ASCII
parseString = Encoding.ASCII.GetString(buf, 0, count);
// Append string to results
sb.Append(tempString);
}
}
while (count > 0);
}
I'm not sure what you have on your side, but I know for a fact (and I'm sure many people will agree here) that GetResponse() will NOT download the whole file back. It will send the request, wait for the response, and get the response headers.
After you have the response, you can easily get the response stream with GetResponseStream(), which is the actual data stream that's downloading from the server. And you can easily access the response stream BEFORE the whole file is downloaded. This is 100% true and tested.
If you're not getting the same behaviour (which is really strange, and shouldn't happen) could you add a code example that is not working as I explained above?
Also, do test the example posted by scherand. It just proves once again that it works just fine, without any special hacks.
Related
I am trying to send a continuous stream, from a C# application, to an ASP Core REST API.
I define a continuous stream as for example someone talking into a microphone and the sound being sent directly, without being saved to a local file) to the Rest API to be saved to file.
I have been searching a lot on Google for something like that and so far could not find anything really useful.
I have been trying to emulate it by sending a large file (297MB).
This is what I have so far for the client side:
string TARGETURL = "http://localhost:58000/api/file/";
string filePath = #"G:\Voice\Samples\The Monkey's Paw.wav";
byte[] fileContent = File.ReadAllBytes(filePath);
var dummyStream = new MemoryStream(fileContent);
var inputData = new StreamContent(dummyStream);
HttpResponseMessage response = this._httpClient.PostAsync(TARGETURL, inputData).Result;
HttpContent result = response.Content;
if (response.IsSuccessStatusCode)
{
string contents = result.ReadAsStringAsync().Result;
}
else
{
// do something
}
And for the server side:
[Route("")]
[HttpPost]
public async Task<JsonResult> Post()
{
Dictionary<string, object> rv = new Dictionary<string, object>();
try
{
string file = Path.Combine(#"G:\Voice\Samples\dummy.txt");
using (FileStream fs = new FileStream(file, FileMode.Create, FileAccess.Write,
FileShare.None, 4096, useAsync: true))
{
await Request.Body.CopyToAsync(fs);
}
// complete the transaction
rv.Add("success", true);
rv.Add("error", "");
}
catch(Exception ex)
{
}
return Json(rv);
}
When I am sending the file, the server throw the following exception:
The request's Content-Length 304137380 is larger than the request body size limit 30000000.
I know that I could increase the body size limit, but that's not a longer term solution as the stream length could get longer that any limit I set.
That's why I am trying to find a solution that send the stream by chunks for the server to rebuild and write to a file.
What you probably want to do is use a different network stack. A web application is always going to try and fit everything into HTTP. This is a very specific kind of way to communicate. And REST is built on top of these ideas as well. Things are generally though of as a document with references on the Internet, and REST is an extension to this idea.
It does however sit on top of some other great technologies that might suit your need better.
There's nothing to stop you using the internet, but maybe you need to look at possibly a UDP or TCP level implementation. Be aware that you will still be sending information in packets. There is no such thing as a constant stream of bits on the internet. A sound wave in the real world is an infinite thing, but computers are rubbish at that.
Maybe start by taking a look at using sockets and a library like NAudio.
I have a external server URL to which I pass credentials and and id to get the audio file like http://myexternalserver.com/?u=xxx&p=xxx&id=xxx
In order to avoid doing this from javascript and exposing the credentials to user, I was attempting to call the url from backend and stream it to the UI request(on my server)
using (Stream mystream = httpResponse2.GetResponseStream())
{
using (BinaryReader reader = new BinaryReader(mystream))
{
int length = 2048;
byte[] buffer = new byte[length];
System.Web.HttpResponse response = System.Web.HttpContext.Current.Response;
response.BufferOutput = true;
response.ContentType = "audio/wav";
while((bytesRead = reader.Read(buffer, 0, buffer.Length)) > 0)
{
response.OutputStream.Write(buffer, 0, bytesRead);
}
response.End();
}
}
Using this approach, I am successfully able to play the stream in an <audio> element.
Below are issues which I'm facing:
While the stream is playing, the seek control bar is always stuck at 0 as audio length is Infinity.Due to this I am unable to use control slider to seek to buffer areas
When the stream ends, $("audio")[0].duration returns 9188187664790.943 (or some huge number for 20 - 30 seconds audio) and audio's display time shows -596523:-14:-8 (while playing this was a number going from 00:01 to 00:24 and then suddenly to a negative number).
I'm unable to find a solution which will allow seeking into an unbuffered area.
I'm also not quite sure if this is a correct/best approach, so suggestions on approach would also be very helpful.
I was able to solve the issue.
What I did was I examined what my HTTP server responded to requests for regular mp3 files stored on the server statically. I noticed that the server was setting two headers which I missed. Those were Accept-Ranges: bytes and Content-Length: xxx.
When I set those headers all the issues from the question disappeared.
Hope this will help somebody.
I'm trying creating a program that allows users to upload JAR Files for some third-party code they've written to an online server and then receive a String message back in response.
The online server is coded using Java with standard TCP Socket networking. The client uses a piece of additional software, which means I have to use C# for the File uploader. The code I've written is included below. In this program the File uploader works fine, but for some reason the client hangs when it reaches input.ReadLine() where it is supposed to receive the String message response from the server.
public static string sendFile(string filepath) {
String response = "";
// Get the file
Stream fileStream = File.OpenRead(filepath);
byte[] buffer = new byte[fileStream.Length];
fileStream.Read(buffer, 0, buffer.Length);
try {
// Create the connection
TCPClient client = new TCPClient("127.0.0.1", 21000);
NetworkStream stream = client.GetStream();
// Send the file to the server
stream.Write(buffer, 0, buffer.Length);
stream.Flush();
// Receive a single string response from the server
if (stream.CanRead) {
StreamReader input = new StreamReader(stream);
inputString = input.ReadLine();
}
input.Close();
stream.Close();
client.Close();
}
catch (IOException e) {
print ("Error: " + e);
}
// Return the response message string
return inputString;
}
I have also tried implementing the code above using a StreamWriter rather than writing directly from the NetworkStream itself. Unfortunately the StreamWriter class doesn't have a method for sending a byte array, (only a char array). I'm wondering whether the problem is being caused by the fact I'm calling the Write method of the NetworkStream directly rather than using a StreamWriter.
If anybody has any idea why the code above isn't working then please let me know. Alternatively, if you have a different solution that would allow me to send a file (byte array) and receive back a string message using the same TCPClient connection then please also feel free to mention it.
Regards,
Midavi.
The readline hangs because it will only return when it has successfully read a line from the server, this is the disadvantage of using blocking sockets. Please make sure your server is accually sending a line( string ending with "\n"
Is your stream terminated with an end of line?
Readline will block until the stream ends or receives the end of line character. If your uploader doesn't terminate the string it could act like you're saying.
Checkout networkComms.net, an open source C# network communication library. A short example demonstrating the most basic functionality here, hopefully not overly complex! Most of the problems you might come across will already have been solved and it might save you some time.
Readline wait for end of line \r\n but in java the end of line is by default \n only. This can be the difference that can block your comunication, but to be sure you need to snoop the network activity or you can use a different method to read data.
You can try to use the read method:
byte[] myReadBuffer = new byte[512];
int numberOfBytesRead = 0;
// Incoming message may be larger than the buffer size.
do{
numberOfBytesRead = stream.Read(myReadBuffer, 0, myReadBuffer.Length);
// do something with data in myReadBuffer
}
while(stream.DataAvailable);
EDIT: Filezilla caused the problem, when i download files back from server it added new lines. I'm sorry for confusion.
This method upload files to ftp server and it's work fine, but in text files uploaded to server blank lines appear after every line("cr lf" appear), for example:
File:
First line
Second line
Third line
Uploaded file:
First line
Second line
Third line
Origin and uploaded files accordingly have different sizes, non-text files are the same.
Code:
private void sendFile(string In, string Out)
{
FtpWebRequest request = (FtpWebRequest) WebRequest.Create("ftp://domain//" + Out);
request.Method = WebRequestMethods.Ftp.UploadFile;
request.Credentials = new NetworkCredential("username", "password");
FileStream sourceStream = new FileStream(In, FileMode.Open, FileAccess.Read, FileShare.Read);
byte[] fileContents = new byte[sourceStream.Length];
sourceStream.Read(fileContents, 0, (int) sourceStream.Length);
sorceStream.Close();
request.ContentLength = fileContents.Length;
Stream requestStream = request.GetRequestStream();
requestStream.Write(fileContents, 0, fileContents.Length);
requestStream.Close();
}
How can i fix this?
EDIT: As the answer below doesn't seem to have helped (but I'm leaving it there for posterity as it shows better code) here are the next diagnostics steps I'd check:
How are you viewing the files? If at all possible, get onto the server directly rather than fetching the files again via a web browser or whatever.
What's the type of FTP server you're connecting to? Maybe there's a known issue.
Have you tried looking at what's actually being sent via Wireshark?
Have you tried sending the same files via a normal FTP client?
You should set FtpWebRequest.UseBinary to true in order to preserve the exact file contents. Otherwise the two systems will try to figure out line endings themselves, changing line terminators as they see fit. I very rarely think that's a good idea. (EDIT: UseBinary is actually true by default, but this sounds like the kind of problem introduced by using text mode... it certainly does no harm to make this explicit.)
Additionally:
You should be disposing of your FileStream via a using statement
You should be disposing of the request stream via a using statement
You should be taking note of the result of Stream.Read - it needn't always read the whole of the requested data in one go
You can either use File.ReadAllBytes to simply read the complete file data in one go, or use Stream.CopyTo (if you're using .NET 4) to copy the FileStream to the request stream (which won't set the content length, of course; I don't know whether this is a problem)
You're never calling GetResponse; it's unclear exactly what happens if you never fetch the response of an FtpWebRequest
Your parameter names don't match .NET naming conventions, and aren't very descriptive
So I would probably use:
private void SendFile(string inputFile, string outputPath)
{
FtpWebRequest request = (FtpWebRequest) WebRequest.Create
("ftp://domain//" + outputPath);
request.Method = WebRequestMethods.Ftp.UploadFile;
request.UseBinary = true;
request.Credentials = new NetworkCredential("username", "password");
byte[] fileContents = File.ReadAllBytes(inputFile);
request.ContentLength = fileContents.Length;
using (Stream requestStream = request.GetRequestStream())
{
requestStream.Write(fileContents, 0, fileContents.Length);
}
// This *may* be necessary in order to validate that everything has happened
using (WebResponse response = request.GetResponse())
{
}
}
Its strange. I face the same problem and I was unable to fix it until I did not provide an extension in file. For Example if my file name was
abcfile
then I make it abcfile.dat and after that it shows me the uploaded file as actual file. I again upload file with abcfile.txt but this time again empty line problem appear in my uploaded file.
I suggest that you must provide extension to your file any except .txt.
The system that you're sending to uses different line endings to what your system uses. I can assume, because you get an extra line, that you're on Windows, and it uses CRLF endings. The system you're sending to recognises CR and LF as separate endings, so you get the extra lines.
For text, truncate the LF or the CR, see what happens. I have no clue about the differing file sizes.
In the top menu of FileZilla, set:
Transfer menu > Transfer type > binary
In the top menu of FileZilla, set:
Transfer menu > Transfer type > binary
It's working for me.
I have to interface with a slightly archaic system that doesn't use webservices. In order to send data to this system, I need to post an XML document into a form on the other system's website. This XML document can get very large so I would like to compress it.
The other system sits on IIS and I use C# my end. I could of course implement something that compresses the data before posting it, but that requires the other system to change so it can decompress the data. I would like to avoid changing the other system as I don't own it.
I have heard vague things about enabling compression / http 1.1 in IIS and the browser but I have no idea how to translate that to my program. Basically, is there some property I can set in my program that will make my program automatically compress the data that it is sending to IIS and for IIS to seamlessly decompress it so the receiving app doesn't even know the difference?
Here is some sample code to show roughly what I am doing;
private static void demo()
{
Stream myRequestStream = null;
Stream myResponseStream = null;
HttpWebRequest myWebRequest = (HttpWebRequest)System.Net
.WebRequest.Create("http://example.com");
byte[] bytMessage = null;
bytMessage = Encoding.ASCII.GetBytes("data=xyz");
myWebRequest.ContentLength = bytMessage.Length;
myWebRequest.Method = "POST";
// Set the content type as form so that the data
// will be posted as form
myWebRequest.ContentType = "application/x-www-form-urlencoded";
//Get Stream object
myRequestStream = myWebRequest.GetRequestStream();
//Writes a sequence of bytes to the current stream
myRequestStream.Write(bytMessage, 0, bytMessage.Length);
//Close stream
myRequestStream.Close();
WebResponse myWebResponse = myWebRequest.GetResponse();
myResponseStream = myWebResponse.GetResponseStream();
}
"data=xyz" will actually be "data=[a several MB XML document]".
I am aware that this question may ultimately fall under the non-programming banner if this is achievable through non-programmatic means so apologies in advance.
I see no way to compress the data on one side and receiving them uncompressed on the other side without actively uncompressing the data..
No idea if this will work since all of the examples I could find were for download, but you could try using gzip to compress the data, then set the Content-Encoding header on the outgoing message to gzip. I believe that the Length should be the length of the zipped message, although you may want to play with making it the length of the unencoded message if that doesn't work.
Good luck.
EDIT I think the issue is whether the ISAPI filter that supports compression is ever/always/configurably invoked on upload. I couldn't find an answer to that so I suspect that the answer is never, but you won't know until you try (or find the answer that eluded me).