Develop Reference

Force an async WebRequest to flush its buffer c#

I have the following connection code:
_request = (HttpWebRequest)WebRequest.Create(complianceUrl);
_request.Method = "GET";
var authInfo = string.Format("{0}:{1}", _username, _password);
authInfo = Convert.ToBase64String(Encoding.Default.GetBytes(authInfo));
_request.Headers.Add("Authorization", "Basic " + authInfo);
// set stream parameters
_request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
_request.Headers.Add("Accept-Encoding", "gzip");
_request.Accept = "application/json";
_request.ContentType = "application/json";
_request.ReadWriteTimeout = 30000;
_request.AllowReadStreamBuffering = false;
_request.Timeout = 30; //seconds, sends 15-second heartbeat.
_asyncCallback = HandleResult; //Setting handleResult as Callback method...
_request.BeginGetResponse(_asyncCallback, _request); //Calling BeginGetResponse on
This works fine, and the buffer fills with data and while I have large volumes of data this is fine. But with low volumes of data it takes a while for the buffer to fill up and I want to periodically flush the buffer if I have not had any activity in a while.
I tried to do that this way:
_request.GetRequestStream().FlushAsync();
But this is wrong as it tells me I am causing a ProtocolViolationException I guess as this is a GET Verb?
Can anyone tell me how to forcibly cause the connection to dump the buffer to the client?
Handling response code added:
private void HandleResult(IAsyncResult result)
{
using (var response = (HttpWebResponse) _request.EndGetResponse(result))
using (var stream = response.GetResponseStream())
using (var memory = new MemoryStream())
{
var compressedBuffer = new byte[BlockSize];
while (stream != null && stream.CanRead)
{
var readCount = stream.Read(compressedBuffer, 0, compressedBuffer.Length);
// if readCount is 0, then the stream must have disconnected. Process and abort!
if (readCount == 0)
{
}
}
}
}

It is not possible to make the server side of an HTTP call send you data in a particular way. Neither HTTP nor TCP have provisions for that. You have to take what you get, or talk to the vendor.
Read does not block until a buffer is full. It gives you what arrives immediately. This behavior is well known, this is not a guess of mine.
The 3rd party service send 15 second heartbeats CRLF which will also eventually fill up the buffer.
This sounds like the service is doing what you want. If that is true, the problem must be in your code. But not in the code shown in the question. Try this:
while (true)
{
var readCount = stream.Read(compressedBuffer, 0, compressedBuffer.Length);
Console.WriteLine(readCount);
}
This should show 2 bytes every 15 seconds. If not, the vendor is at fault. If yes, you are.
while (stream != null && stream.CanRead) this is weird because none of these conditions can become false. And even if they become false, what do you do about it?! This should be while (true) plus break on stream depletion.
// if readCount is 0, then the stream must have disconnected.
This condition means that the remote side has orderly finished their sending of the HTTP response.

For anyone interested, you can read and deflate manually doing the following:
using (var response = (HttpWebResponse) _request.EndGetResponse(result))
using (var stream = response.GetResponseStream())
using (var compressedMemory = new MemoryStream())
using (var uncompressedMemory = new MemoryStream()) // Added new memory stream
using (var gzipStream = new GZipStream(compressedMemory, CompressionMode.Decompress))
{
var compressedBuffer = new byte[BlockSize];
while (stream != null && stream.CanRead)
{
var readCount = stream.Read(compressedBuffer, 0, compressedBuffer.Length);
compressedMemory.Write(compressedBuffer.Take(readCount).ToArray(), 0, readCount);
compressedMemory.Position = 0;
gzipStream.CopyTo(uncompressedMemory); // use copy to rather than trying to read
var outputString = Encoding.UTF8.GetString(uncompressedMemory.ToArray());
Debug.WriteLine(outputString);
uncompressedMemory.Position = 0;
uncompressedMemory.SetLength(0);
compressedMemory.Position = 0;
compressedMemory.SetLength(0); // reset length
}
}

Possible Infinite Loop when streaming response

We are trying to identify high CPU usage in services we have, and we believe there are a few potential areas that may be causing infinite loops. Below is code that we believe may potentially be causing an infinite loop. Is there anything specific that sticks out that may cause the while loop to run indefinitely?
WebRequest request = WebRequest.Create(Url);
request.ContentLength = formDataLength;
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "POST";
using (Stream rs = request.GetRequestStream())
{
ASCIIEncoding encoding = new ASCIIEncoding();
var postData = encoding.GetBytes(formData);
rs.Write(postData, 0, postData.Length);
string str = string.Empty;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (Stream sm = response.GetResponseStream())
{
int totalBytesRead = 0;
long responseBytesToRead = 1024;
byte[] buffer = new byte[responseBytesToRead];
int bytesRead;
do
{
bytesRead = sm.Read(buffer, totalBytesRead, (int)(responseBytesToRead - totalBytesRead));
totalBytesRead += bytesRead;
} while (totalBytesRead < bytesRead);
request.Abort();
str = Encoding.Default.GetString(buffer);
}
}
return str;
}

From the MSDN documentation:
Return Value
Type: System.Int32 The total number of bytes read into
the buffer. This can be less than the number of bytes requested if
that many bytes are not currently available, or zero (0) if the end of
the stream has been reached.
0 indicates the end of the stream. The condition for reaching end of stream has already been defined. The condition you rely on could be unreliable and is unnecessary.
Try
while(bytesRead != 0)

It is better to use StreamReader
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
...
reader.ReadToEnd();
// or
while (!reader.EndOfStream)
{
// do read.
}

C# Read webpage content Streamreader

I need to read the content of a webpage in streamreader like
www.example.com
<test>
<sample></sample>
</test>
i got this:
System.IO.StreamReader StreamReader1 =
new System.IO.StreamReader("www.example.com");
string test = StreamReader1.ReadToEnd();
but i then i get this error code
Attempt to access the method failed:
System.IO.StreamReader..ctor(System.String)

Try a WebClient, it's easier and you don't have to worry about streams and rivers:
using (var client = new WebClient())
{
string result = client.DownloadString("http://www.example.com");
// TODO: do something with the downloaded result from the remote
// web site
}

If you want to use the StreamReader, here is the code I am using:
const int Buffer_Size = 100 * 1024;
WebRequest request = CreateWebRequest(uri);
WebResponse response = request.GetResponse();
result = GetPageHtml(response);
...
private string GetPageHtml(WebResponse response) {
char[] buffer = new char[Buffer_Size];
Stream responseStream = response.GetResponseStream();
using(StreamReader reader = new StreamReader(responseStream)) {
int index = 0;
int readByte = 0;
do {
readByte = reader.Read(buffer, index, 256);
index += readByte;
}
while (readByte != 0);
response.Close();
}
string result = new string(buffer);
result = result.TrimEnd(new char[] {'\0'});
return result;
}

Sending image with HttpListener only working for some images

I'm trying create a small http proxy service. This is not working so well. It is able to serve HTML okayish, however it chokes up on images. That is, some images.
Sending in a url through my proxy yields 19.4 kb in the response (according to firebug)
Visiting that url directly also yields 19.4 kb in the response, again according to firebug. The difference is, it doesn't show up when I put it through my proxy, but it does when I browse directly.
A completely different url works just fine. Does anyone have any idea?
private void DoProxy()
{
var http = listener.GetContext();
string url = http.Request.QueryString["url"];
WebRequest request = HttpWebRequest.Create(url);
WebResponse response = request.GetResponse();
http.Response.ContentType = response.ContentType;
byte[] content;
using (Stream responseStream = response.GetResponseStream())
content = ReadAll(responseStream);
http.Response.ContentLength64 = content.Length;
http.Response.OutputStream.Write(content, 0, content.Length);
http.Response.Close();
}
private byte[] ReadAll(Stream stream)
{
IList<byte> array = new List<byte>();
int b;
while ((b = stream.ReadByte()) != -1)
array.Add(Convert.ToByte(b));
return array.ToArray();
}

I would try and flush/close the OutputStream before you close the response.
Also as a second suggestion have a look at the HTTP traffic from the original site and then through your proxy site using an HTTP debugger like Fiddler - there must be a difference when using your proxy.
Also to make the ReadAll method more effective, in general I would avoid to load the full content into memory, because this will blow up on huge files - just stream them directly from the input stream to the output stream. If you still want to use byte arrays consider the following (untested but should work):
private byte[] ReadAll(Stream stream)
{
byte[] buffer = new byte[8192];
int bytesRead = 1;
List<byte> arrayList = new List<byte>();
while (bytesRead > 0)
{
bytesRead = stream.Read(buffer, 0, buffer.Length);
arrayList.AddRange(new ArraySegment<byte>(buffer, 0, bytesRead).Array);
}
return arrayList.ToArray();
}

You can try to replace
http.Response.Close();
with
http.Response.Flush();
http.Response.End();

A problem could be that you don't specify the MIME type of the response. Browsersthese days are very forgiving, but maybe there is a circumstance where the browser doesn't know how to handle whatever you are sticking through its throat.
I have written the most smallish file-based http server, presented here, which as far as I can remember can serve images without much problem.

Just separate the text response and image response, and write the outputs separately. I did like below and it worked for me.
static void Main(string[] args)
{
HttpListener server = new HttpListener();
server.Prefixes.Add("http://localhost:9020/");
server.Start();
Console.WriteLine("Listening...");
while (true)
{
try
{
HttpListenerContext context = server.GetContext();
HttpListenerResponse response = context.Response;
String localpath = context.Request.Url.LocalPath;
string page = Directory.GetCurrentDirectory() + localpath;
string msg = "";
bool imgtest = false;
if (localpath == "/")
page = "index.html";
Console.WriteLine(localpath);
if (!page.Contains("jpg") && !page.Contains("png"))//Separates image request
{
TextReader tr = new StreamReader(page);
msg = tr.ReadToEnd();
tr.Dispose();
}
else
{
byte[] output = File.ReadAllBytes(page);
response.ContentLength64 = output.Length;
Stream st1 = response.OutputStream;
st1.Write(output, 0, output.Length);
imgtest = true;
}
if (imgtest==false)
{
byte[] buffer = Encoding.UTF8.GetBytes(msg);
response.ContentLength64 = buffer.Length;
Stream st = response.OutputStream;
st.Write(buffer, 0, buffer.Length);
context.Response.Close();
}
}
catch (Exception ex)
{
Console.WriteLine("Error: "+ex);
Console.ReadKey();
}
}

How to use WebRequest to POST some data and read response?

Need to have the server make a POST to an API, how do I add POST values to a WebRequest object and how do I send it and get the response (it will be a string) out?
I need to POST TWO values, and sometimes more, I see in these examples where it says string postData = "a string to post"; but how do I let the thing I am POSTing to know that there is multiple form values?

From MSDN
// Create a request using a URL that can receive a post.
WebRequest request = WebRequest.Create ("http://contoso.com/PostAccepter.aspx ");
// Set the Method property of the request to POST.
request.Method = "POST";
// Create POST data and convert it to a byte array.
string postData = "This is a test that posts this string to a Web server.";
byte[] byteArray = Encoding.UTF8.GetBytes (postData);
// Set the ContentType property of the WebRequest.
request.ContentType = "application/x-www-form-urlencoded";
// Set the ContentLength property of the WebRequest.
request.ContentLength = byteArray.Length;
// Get the request stream.
Stream dataStream = request.GetRequestStream ();
// Write the data to the request stream.
dataStream.Write (byteArray, 0, byteArray.Length);
// Close the Stream object.
dataStream.Close ();
// Get the response.
WebResponse response = request.GetResponse ();
// Display the status.
Console.WriteLine (((HttpWebResponse)response).StatusDescription);
// Get the stream containing content returned by the server.
dataStream = response.GetResponseStream ();
// Open the stream using a StreamReader for easy access.
StreamReader reader = new StreamReader (dataStream);
// Read the content.
string responseFromServer = reader.ReadToEnd ();
// Display the content.
Console.WriteLine (responseFromServer);
// Clean up the streams.
reader.Close ();
dataStream.Close ();
response.Close ();
Take into account that the information must be sent in the format key1=value1&key2=value2

Here's what works for me. I'm sure it can be improved, so feel free to make suggestions or edit to make it better.
const string WEBSERVICE_URL = "http://localhost/projectname/ServiceName.svc/ServiceMethod";
//This string is untested, but I think it's ok.
string jsonData = "{ \"key1\" : \"value1\", \"key2\":\"value2\" }";
try
{
var webRequest = System.Net.WebRequest.Create(WEBSERVICE_URL);
if (webRequest != null)
{
webRequest.Method = "POST";
webRequest.Timeout = 20000;
webRequest.ContentType = "application/json";
using (System.IO.Stream s = webRequest.GetRequestStream())
{
using (System.IO.StreamWriter sw = new System.IO.StreamWriter(s))
sw.Write(jsonData);
}
using (System.IO.Stream s = webRequest.GetResponse().GetResponseStream())
{
using (System.IO.StreamReader sr = new System.IO.StreamReader(s))
{
var jsonResponse = sr.ReadToEnd();
System.Diagnostics.Debug.WriteLine(String.Format("Response: {0}", jsonResponse));
}
}
}
}
catch (Exception ex)
{
System.Diagnostics.Debug.WriteLine(ex.ToString());
}

Here's an example of posting to a web service using the HttpWebRequest and HttpWebResponse objects.
StringBuilder sb = new StringBuilder();
string query = "?q=" + latitude + "%2C" + longitude + "&format=xml&key=xxxxxxxxxxxxxxxxxxxxxxxx";
string weatherservice = "http://api.worldweatheronline.com/free/v1/marine.ashx" + query;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(weatherservice);
request.Referer = "http://www.yourdomain.com";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream stream = response.GetResponseStream();
StreamReader reader = new StreamReader(stream);
Char[] readBuffer = new Char[256];
int count = reader.Read(readBuffer, 0, 256);
while (count > 0)
{
String output = new String(readBuffer, 0, count);
sb.Append(output);
count = reader.Read(readBuffer, 0, 256);
}
string xml = sb.ToString();

A more powerful and flexible example can be found here: C# File Upload with form fields, cookies and headers

Below is the code that read the data from the text file and sends it to the handler for processing and receive the response data from the handler and read it and store the data in the string builder class
//Get the data from text file that needs to be sent.
FileStream fileStream = new FileStream(#"G:\Papertest.txt", FileMode.OpenOrCreate, FileAccess.ReadWrite);
byte[] buffer = new byte[fileStream.Length];
int count = fileStream.Read(buffer, 0, buffer.Length);
//This is a handler would recieve the data and process it and sends back response.
WebRequest myWebRequest = WebRequest.Create(#"http://localhost/Provider/ProcessorHandler.ashx");
myWebRequest.ContentLength = buffer.Length;
myWebRequest.ContentType = "application/octet-stream";
myWebRequest.Method = "POST";
// get the stream object that holds request stream.
Stream stream = myWebRequest.GetRequestStream();
stream.Write(buffer, 0, buffer.Length);
stream.Close();
//Sends a web request and wait for response.
try
{
WebResponse webResponse = myWebRequest.GetResponse();
//get Stream Data from the response
Stream respData = webResponse.GetResponseStream();
//read the response from stream.
StreamReader streamReader = new StreamReader(respData);
string name;
StringBuilder str = new StringBuilder();
while ((name = streamReader.ReadLine()) != null)
{
str.Append(name); // Add to stringbuider when response contains multple lines data
}
}
catch (Exception ex)
{
throw ex;
}