Develop Reference

GetRequestStream method and hanging thread

Assume I have the following code:
private string PostData(string functionName, string parsedContent)
{
string url = // some url;
var http = (HttpWebRequest)WebRequest.Create(new Uri(url));
http.Accept = "application/json";
http.ContentType = "application/json";
http.Method = "POST";
http.Timeout = 15000; // 15 seconds
Byte[] bytes = Encoding.UTF8.GetBytes(parsedContent);
using (Stream newStream = http.GetRequestStream())
{
newStream.Write(bytes, 0, bytes.Length);
}
using (WebResponse response = http.GetResponse())
{
using (var stream = response.GetResponseStream())
{
var sr = new StreamReader(stream);
var content = sr.ReadToEnd();
return content;
}
}
}
I set up a breakpoint over this line of code:
using (Stream newStream = http.GetRequestStream())
before http.GetRequestStream() gets executed. Here is a screenshot of my active threads:
This whole method is running in background thread with ThreadId = 3 as you can see.
After pressing F10 we get http.GetRequestStream() method executed. And here is an updated screenshot of active threads:
As you can see, now we have one extra active thread that is in state of waiting. Probably the method http.GetRequestStream() spawns it. Everything is fine, but.. this thread keeps hanging like that for the whole app lifecycle, which seems not to be the intended behaviour.
Am I misusing GetRequestStream somehow?

If I use ilspy it looks like the request is send asynchronously. That would explain the extra thread.
Looking a little bit deeper the HttpWebRequest creates a static TimerQueue with one thread and a never ending loop, that has a Monitor.WaitAny in it. Every webrequest in the appdomain will register a timer callback for timeout handling and all those callbacks are handled by that thread. Due to it being static that instance will never get garbage collected and therefore it will keep hold of the thread.
It did register for the AppDomain.Unload event so if that fires it will clean up it's resources including any threads.
Do notice that these are all internal classes and those implementation details might change at any time.

A method for making HTTP requests on Unity iOS?

I need to send HTTP requests with all the standard RESTful methods and access to the body of the request in order to send/receive JSON with it. I've looked into,
WebRequest.HttpWebRequest
This works almost perfectly, but there are cases where, for example, if the server is down the function GetResponse can take several seconds to return- since it is a synchronous method- freezing the application for that period. The asynchronous version of this method, BeginGetResponse, does not seem to work asynchronously (in Unity anyway) as it still freezes the application for that period.
UnityEngine.WWW#
Only supports POST and GET requests for some reason- but I also need PUT and DELETE (standard RESTful methods) so I didn't bother looking into it any further.
System.Threading
In order to run WebRequest.HttpWebRequest.GetResponse without freezing the application I looked into using threads. Threads seem to work in the editor (but seem extremely volatile- if you don't stop a thread when the application exits it keeps running in the editor forever even when you stop it), and when built to an iOS device crash it as soon as I try to start a thread (I forgot to write down the error and I don't have access to it right now).
Run threads in a native iOS app with a bridge to the Unity app
Ridiculous, not even going to attempt this.
UniWeb
This. I would like to know how they managed it.
Here is an example of the WebRequest.BeginGetResponse method I am trying,
// The RequestState class passes data across async calls.
public class RequestState
{
const int BufferSize = 1024;
public StringBuilder RequestData;
public byte[] BufferRead;
public WebRequest Request;
public Stream ResponseStream;
// Create Decoder for appropriate enconding type.
public Decoder StreamDecode = Encoding.UTF8.GetDecoder();
public RequestState()
{
BufferRead = new byte[BufferSize];
RequestData = new StringBuilder(String.Empty);
Request = null;
ResponseStream = null;
}
}
public class WebRequester
{
private void ExecuteRequest()
{
RequestState requestState = new RequestState();
WebRequest request = WebRequest.Create("mysite");
request.BeginGetResponse(new AsyncCallback(Callback), requestState);
}
private void Callback(IAsyncResult ar)
{
// Get the RequestState object from the async result.
RequestState rs = (RequestState) ar.AsyncState;
// Get the WebRequest from RequestState.
WebRequest req = rs.Request;
// Call EndGetResponse, which produces the WebResponse object
// that came from the request issued above.
WebResponse resp = req.EndGetResponse(ar);
}
}
... based on this: http://msdn.microsoft.com/en-us/library/86wf6409(v=vs.71).aspx

Ok, I finally managed to write my own solution. We basically need a RequestState, a Callback Method and a TimeOut Thread. Here I'll just copy what was done in UnifyCommunity (now called unity3d wiki). This is outdated code, but smaller than what's there, so more convenient to show something here. Now I've removed (in the unit3d wiki) System.Action and static for performance and simplicity:
Usage
static public ThisClass Instance;
void Awake () {
Instance = GetComponent<ThisClass>();
}
static private IEnumerator CheckAvailabilityNow () {
bool foundURL;
string checkThisURL = "http://www.example.com/index.html";
yield return Instance.StartCoroutine(
WebAsync.CheckForMissingURL(checkThisURL, value => foundURL = !value)
);
Debug.Log("Does "+ checkThisURL +" exist? "+ foundURL);
}
WebAsync.cs
using System;
using System.IO;
using System.Net;
using System.Threading;
using System.Collections;
using UnityEngine;
/// <summary>
/// The RequestState class passes data across async calls.
/// </summary>
public class RequestState
{
public WebRequest webRequest;
public string errorMessage;
public RequestState ()
{
webRequest = null;
errorMessage = null;
}
}
public class WebAsync {
const int TIMEOUT = 10; // seconds
/// <summary>
/// If the URLs returns 404 or connection is broken, it's missing. Else, we suppose it's fine.
/// </summary>
/// <param name='url'>
/// A fully formated URL.
/// </param>
/// <param name='result'>
/// This will bring 'true' if 404 or connection broken and 'false' for everything else.
/// Use it as this, where "value" is a System sintaxe:
/// value => your-bool-var = value
/// </param>
static public IEnumerator CheckForMissingURL (string url, System.Action<bool> result) {
result(false);
Uri httpSite = new Uri(url);
WebRequest webRequest = WebRequest.Create(httpSite);
// We need no more than HTTP's head
webRequest.Method = "HEAD";
RequestState requestState = new RequestState();
// Put the request into the state object so it can be passed around
requestState.webRequest = webRequest;
// Do the actual async call here
IAsyncResult asyncResult = (IAsyncResult) webRequest.BeginGetResponse(
new AsyncCallback(RespCallback), requestState);
// WebRequest timeout won't work in async calls, so we need this instead
ThreadPool.RegisterWaitForSingleObject(
asyncResult.AsyncWaitHandle,
new WaitOrTimerCallback(ScanTimeoutCallback),
requestState,
(TIMEOUT *1000), // obviously because this is in miliseconds
true
);
// Wait until the the call is completed
while (!asyncResult.IsCompleted) { yield return null; }
// Deal up with the results
if (requestState.errorMessage != null) {
if ( requestState.errorMessage.Contains("404") || requestState.errorMessage.Contains("NameResolutionFailure") ) {
result(true);
} else {
Debug.LogWarning("[WebAsync] Error trying to verify if URL '"+ url +"' exists: "+ requestState.errorMessage);
}
}
}
static private void RespCallback (IAsyncResult asyncResult) {
RequestState requestState = (RequestState) asyncResult.AsyncState;
WebRequest webRequest = requestState.webRequest;
try {
webRequest.EndGetResponse(asyncResult);
} catch (WebException webException) {
requestState.errorMessage = webException.Message;
}
}
static private void ScanTimeoutCallback (object state, bool timedOut) {
if (timedOut) {
RequestState requestState = (RequestState)state;
if (requestState != null)
requestState.webRequest.Abort();
} else {
RegisteredWaitHandle registeredWaitHandle = (RegisteredWaitHandle)state;
if (registeredWaitHandle != null)
registeredWaitHandle.Unregister(null);
}
}
}

I got threading to work on iOS- I believe it was crashing due to ghost threads or something. Restarting the device seems to have fixed the crashing so I'll just use WebRequest.HttpWebRequest with threads.

There is a way of doing this asynchronously, without using IEnumerator and yield return stuff. Check out the eDriven framework.
HttpConnector class: https://github.com/dkozar/eDriven/blob/master/eDriven.Networking/Rpc/Core/HttpConnector.cs
I've been using JsonFX with HttpConnector all the time, for instance in this WebPlayer demo: http://edrivenunity.com/load-images
Not having PUT and DELETE is not a big issue, since all of it could be done using GET and POST. For instance I'm successfully communicating with Drupal CMS using its REST service.

// javascript in the web player not ios, android or desktop you could just run the following code:
var jscall:String;
jscall="var reqScript = document.createElement('script');";
jscall+="reqScript.src = 'synchmanager_secure2.jsp?userid="+uid+"&token="+access_token+"&rnd='+Math.random()*777;";
jscall+="document.body.appendChild(reqScript);";
Application.ExternalEval(jscall);
// cs
string jscall;
jscall="var reqScript = document.createElement('script');";
jscall+="reqScript.src = 'synchmanager_secure2.jsp?userid="+uid+"&token="+access_token+"&rnd='+Math.random()*777;";
jscall+="document.body.appendChild(reqScript);";
Application.ExternalEval(jscall);
// then update your object using the your return in a function like this
// json return object always asynch
function sendMyReturn(args){
var unity=getUnity();
unity.SendMessage("object", "function", args );
}
sendMyReturn(args);
or you can send it via a AJAX function prewritten custom headers for security purposes
with this you would need signed headers and a signed request back from the server
I prefer md5 signatures comparatively they are not so big

WebClient does not support concurrent I/O operations

How can I get this error from with in the DownloadStringCompleted Event? Doesn't that mean, it's finished? Is there another event I can fire this from?
I get this error extremely rarely, but once in a while it will happen on my WP7 phone. I have a web client that I fire over and over, and I fire it again from the completed event. Is this happening because there is still some stale connection open? Is there a way to prevent this 100%?
I have checked to see if there is a chance for the thread to walk over itself, but it is only fired from within the completed event.
How can I be sure, when the complete event is fired, the client is no longer isBusy? One suggestion was to add a while with a thread sleep while the client is busy.
Some pseudo code.
var client = new WebClient("URL 1");
client.CompletedEvent += CompletedEvent;
client.downloadasync();
void CompletedEvent(){
Dosomestuff;
client.downloadasync(); //This is where we break.
}

The WebClient only supports a single operations, it cannot download multiple files. You haven't shown your code, but my guess is that you are somehow firing a new request before the old is completed. My bet is that WebClient.IsBusy is true when you attempt to perform another fetch.
See the following thread:
wb.DownloadFileAsync throw "WebClient does not support concurrent I/O operations." exception

The only answer is to create a new webclient within the scope of the Completed Event. You can't set it to new since webclient is readonly. Creating a new client is the only solution. This allows the old client to complete in the background. This does have slight memory implications since you are creating a new instance instead of reusing an old. But the garbage collector should keep it clean if your scope is setup right.

Instead of using WebClient use HttpClient to do parallel HTTP calls. Below code shows how to download files.
HttpClient httpClient = new HttpClient();
var documentList=_documentManager.GetAllDocuments();
documentList.AsParallel().ForAll(doc =>
{
var responseResult= httpClient.GetAsync(doc.FileURLPath);
using (var memStream = responseResult.Result.Content.ReadAsStreamAsync().Result)
{
using (var fileStream =File.Create($"{filePath}\\{doc.FileName}"))
{
memStream.CopyTo(fileStream);
}
}
});

The solution, I found is to use multiple WebClient objects, so to modify your pseudocode example; try
var client = new WebClient("URL 1");
client.CompletedEvent += CompletedEvent;
client.downloadasync();
void CompletedEvent(){
Dosomestuff;
var client2 = new WebClient();
client2.downloadasync();
}

Create a new Web Client for each new request. Don't reuse an existing Web Client instance.
This allows the first request to complete before starting the new one. This is a standard way of creating new requests.

private async Void SyncParcelStatus(List<string> Urls)
{
try
{
foreach (var URL in WebhookUrls)
{
Task.Factory.StartNew(() => AsyncDownLoad(URL));
}
}
catch (Exception ex)
{
//log Exception
}
}
private async void AsyncDownLoad(string URL)
{
using (WebClient myWebClient = new WebClient())
{
try
{
Uri StringToUri = new Uri(URL);
myWebClient.DownloadStringAsync(StringToUri);
}
catch (Exception ex)
{
//log Exception
}
}
}

C# and running of HttpListener in background

I created simple HttpListener that listens to port 9090 and depending on a request's URL writes some info to the console.
But I'm stuck :( I thought of multithreading, event based system, but I dind't manage to do anything with it.
Here is the code of my listener that I launched as a separate console app:
string urlTemplate = String.Format("/prefix/{0}/suffix", id);
string prefix = String.Format("http://localhost:9090/");
HttpListener listener = new HttpListener();
listener.Prefixes.Add(prefix);
listener.Start();
Console.WriteLine("Listening to {0}...", prefix);
while (true)
{
HttpListenerContext context = listener.GetContext();
HttpListenerRequest request = context.Request;
//Response object
HttpListenerResponse response = context.Response;
//Construct response
if (request.RawUrl.Contains(urlTemplate) && request.HttpMethod == "POST")
{
string requestBody;
Stream iStream = request.InputStream;
Encoding encoding = request.ContentEncoding;
StreamReader reader = new StreamReader(iStream, encoding);
requestBody = reader.ReadToEnd();
Console.WriteLine("POST request on {0} with body = [{1}]", request.RawUrl, requestBody);
response.StatusCode = (int)HttpStatusCode.OK;
//Return a response
using (Stream stream = response.OutputStream) { }
}
else
{
response.StatusCode = (int)HttpStatusCode.BadRequest;
Console.WriteLine("Invalid HTTP request: [{0}] {1}", request.HttpMethod, request.Url);
using (Stream stream = response.OutputStream) { }
}
}
I decided to use it as an utility for unit tests (maybe somewhere else). So when test starts I need to configure the listener, run it, then make some requests and receive the info (which listener wrote earlier to Console), and at the end of the test stop the listener.
My main idea was to incapsulate this listener to separate class MyHttpListener which has methods: StartListener(), StopListener().
But when I call StartListener() my test freezes because of infinite while loop. I tried to create separate background thread or event based system, but my lack of experience with them, prevents me from doing it. I've already spent a lot of time trying to find the solution, but all for nothing.
Hope you can help me finding the solution for such trivial task.
Thanks in advance.

One of the responder's variant (it seems he deleted his post) looked good, but it didn't work for me. I tried to fix things in order it started working, but at the end that variant gave me an idea how to solve the problem - with multithreading and events :)
Here's what I have now and it works:
public delegate void HttpListenerRequestHandler(object sender, HttpListenerEventArgs e);
public event HttpListenerRequestHandler OnCorrectRequest;
...
if(OnCorrectRequest != null)
OnCorrectRequest(this, new HttpListenerEventArgs(response));
lock (threadLock)
{
Console.WriteLine("POST request on {0} with body = [{1}]", request.RawUrl, requestBody);
}
...
public class HttpListenerEventArgs : EventArgs
{
public readonly HttpListenerResponse response;
public HttpListenerEventArgs(HttpListenerResponse httpResponse)
{
response = httpResponse;
}
}
In order to receive detailed responses from HttpListener in main thread, I use the following:
private HttpListenerResponse response;
public void HttpCallbackRequestCorrect(object sender, HttpListenerEventArgs e)
{
response = e.response;
Console.WriteLine("{0} sent: {1}", sender, e.response.StatusCode);
}
Unfortunately I have the following exception which I don't know how to handle:
System.Net.HttpListenerException: The I/O operation has been aborted because of either a thread exit or an application request
at System.Net.HttpListener.GetContext()

About HttpWebRequest's EndGetResponse

I'm using asynchronous calls for communication to my server. I written some component to collect all unauthorized requests and to resend them after user logs in. I written some test to produce 10 threads that are sending some requests without first being authorized. Than I wait for 20 seconds and do authorization and after that I wait for request to successfully finish. But problem appeared at EndGetResponse method which I call in my callback method. I done that this way:
public void InternalCallback(IAsyncResult result)
{
try
{
RequestState state = (RequestState)result.AsyncState;
IHttpWebRequest request = state.Request;
using (IHttpWebResponse response = responseGetter.GetResponse(request, result))
{
// ...
}
}
// ...
}
So, I made some custom class RequestState which has some higher level callbacks I need and it has request which I'll use to call EndGetResponse method. But this way I got error:
IAsyncResult object was not returned from the corresponding asynchronous method.
I changed this so I now have Request field in my callback class which I set before calling BeginGetResponse and I use that Request field when calling EndGetResponse in my callback.
public void InternalCallback(IAsyncResult result)
{
try
{
using (IHttpWebResponse response = responseGetter.GetResponse(this.Request, result))
{
// ...
}
}
// ...
}
Is this new solution valid one? Can you suggest is this good way to do this or how should I do this?