Multiple HttpWebRequest in loop? - c#

I have to call web-service multiple time (in loop) the problem is that my code always return empty object (Image Description) and doesn't run properly when I tested it separately (out of loop) it worked normally
here is my portion of code
HttpWebRequest httpReq = (HttpWebRequest)HttpWebRequest.Create(new Uri(imageCollection[i].ImageTag));
httpReq.BeginGetResponse(new AsyncCallback((iar) =>
{
try
{
string strResponse = "";
var response = (HttpWebResponse)((HttpWebRequest)iar.AsyncState).EndGetResponse(iar);
Stream stream = response.GetResponseStream();
StreamReader reader = new StreamReader(stream);
strResponse = reader.ReadToEnd();
HtmlDocument htmlDocument = new HtmlDocument();
htmlDocument.OptionFixNestedTags = true;
htmlDocument.LoadHtml(strResponse);
HtmlAgilityPack.HtmlNode titleNode = htmlDocument.DocumentNode.SelectSingleNode("//meta[#property='og:description']");
if (titleNode != null)
{
string desc = titleNode.GetAttributeValue("content", "");
imageCollection[i].ImageDescription = desc;
}
}
catch (Exception ex)
{
throw ex;
}
}), httpReq);
httpReq.Abort();

I got the answer from another post in stack-overflow modified to adapt my solution here
Getting the Response of a Asynchronous HttpWebRequest
I made specific class called Request to transform my logic to newly async and await here it's
public Task<string> MakeAsyncRequest()
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(URL);
Task<WebResponse> task = Task.Factory.FromAsync(
request.BeginGetResponse,
asyncResult => request.EndGetResponse(asyncResult),
(object)null);
return task.ContinueWith(t => ReadStreamFromResponse(t.Result));
}
private string ReadStreamFromResponse(WebResponse response)
{
string desc = "";
try
{
using (Stream responseStream = response.GetResponseStream())
using (StreamReader sr = new StreamReader(responseStream))
{
//Need to return this response
string strContent = sr.ReadToEnd();
HtmlDocument htmlDocument = new HtmlDocument();
htmlDocument.OptionFixNestedTags = true;
htmlDocument.LoadHtml(strContent);
HtmlAgilityPack.HtmlNode titleNode = htmlDocument.DocumentNode.SelectSingleNode("//meta[#property='og:description']");
if (titleNode != null)
{
desc = titleNode.GetAttributeValue("content", "");
}
imageDesc = desc;
return desc;
}
}
catch (Exception ex)
{ return desc; }
}
public string imageDesc { get; private set; }
}
Then I made a Queue of Request
queueWebRequest = new Queue<Request>();
for (int i = 0; i < imageCollection.Count; i++)
{
queueWebRequest.Enqueue(new Request(imageCollection[i].ImageTag));
}
for (int i = 0; i < imageCollection.Count; i++)
{
if (queueWebRequest.Count > 0)
{
Request currentRequest = queueWebRequest.Dequeue();
await currentRequest.MakeAsyncRequest();
imageCollection[i].ImageDescription = currentRequest.imageDesc;
}
else
break;
}

Related

Open link in browser with HttpClient.GetAsync

Is it possible to open a link in browser with HttpClient.GetAsync?
I got a report from a user of my app that the app opened a link in the browser which is only used in with HttpClient.GetAsync.
Some one else had the issue too: why HttpClient.GetAsync causes opening link in browser?
private JObject Get(string url)
{
JObject getResponse = new JObject();
getResponse["error"] = (int)Error.None;
getResponse["message"] = "";
try
{
using (HttpClient client = new HttpClient())
{
Task<HttpResponseMessage> res1 = client.GetAsync(url + GetRequestRandomizer());
HttpResponseMessage res = res1.WaitAndUnwrapException();
if (res.IsSuccessStatusCode && res.StatusCode == HttpStatusCode.OK)
{
var responseContent = res.Content;
getResponse["error"] = (int)Error.None;
getResponse["message"] = responseContent.ReadAsStringAsync().Result;
}
else
{
getResponse["error"] = (int)Error.RequestError;
getResponse["message"] = "";
}
}
}
catch (HttpRequestException exception)
{
getResponse["error"] = (int)Error.NetworkError;
getResponse["message"] = "";
}
return getResponse;
}

strava login via httpwebrequest failed

Hi I'm trying to login via https://www.strava.com/session with HttpWebrequest but it doesn't log me in. It gives me an response of 302 which is good but it never redirect me to https://www.strava.com/dashboard.
this is the code that I'm using
Httpclient:
public class HttpClient
{
private const string UserAgent = "Mozilla/5.0";
public CookieCollection CookieCollection;
public HttpWebRequest WebRequest;
public HttpWebResponse WebResponse;
public int code { get; set; }
public string location { get; set; }
public string PostData(string url, string postData, string refer = "")
{
WebRequest = (HttpWebRequest)System.Net.WebRequest.Create(url);
WebRequest.UserAgent = UserAgent;
WebRequest.Referer = refer;
WebRequest.AllowAutoRedirect =false;
WebRequest.Timeout = 10000;
WebRequest.KeepAlive = true;
WebRequest.CookieContainer = new CookieContainer();
if (CookieCollection != null && CookieCollection.Count > 0)
{
WebRequest.CookieContainer.Add(CookieCollection);
}
WebRequest.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
WebRequest.Method = "POST";
try
{
var postBytes = Encoding.UTF8.GetBytes(postData);
WebRequest.ContentLength = postBytes.Length;
var postDataStream = WebRequest.GetRequestStream();
postDataStream.Write(postBytes, 0, postBytes.Length);
postDataStream.Close();
try
{
WebResponse = (HttpWebResponse)WebRequest.GetResponse();
this.code = (int)WebResponse.StatusCode;
this.location = WebResponse.Headers["Location"];
if (WebResponse.StatusCode == HttpStatusCode.OK ||WebResponse.StatusCode == HttpStatusCode.Redirect)
{
WebResponse.Cookies = WebRequest.CookieContainer.GetCookies(WebRequest.RequestUri);
if (WebResponse.Cookies.Count > 0)
{
if (CookieCollection == null)
{
CookieCollection = WebResponse.Cookies;
}
else
{
foreach (Cookie oRespCookie in WebResponse.Cookies)
{
var bMatch = false;
foreach (
var oReqCookie in
CookieCollection.Cast<Cookie>()
.Where(oReqCookie => oReqCookie.Name == oRespCookie.Name))
{
oReqCookie.Value = oRespCookie.Value;
bMatch = true;
break;
}
if (!bMatch)
CookieCollection.Add(oRespCookie);
}
}
}
var reader = new StreamReader(WebResponse.GetResponseStream());
var responseString = reader.ReadToEnd();
reader.Close();
return responseString;
}
}
catch (WebException wex)
{
if (wex.Response != null)
{
using (var errorResponse = (HttpWebResponse)wex.Response)
{
using (var reader = new StreamReader(errorResponse.GetResponseStream()))
{
var error = reader.ReadToEnd();
return error;
}
}
}
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
return "Error in posting data" ;
}
public string GetData(string url, string post = "")
{
var responseStr = string.Empty;
WebRequest = (HttpWebRequest)System.Net.WebRequest.Create(url);
WebRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8";
WebRequest.Method = "GET";
WebRequest.KeepAlive = true;
WebRequest.Credentials = CredentialCache.DefaultCredentials;
WebRequest.UserAgent = UserAgent;
WebRequest.CookieContainer = new CookieContainer();
if (CookieCollection != null && CookieCollection.Count > 0)
{
WebRequest.CookieContainer.Add(CookieCollection);
}
if (!string.IsNullOrEmpty(post))
{
var postBytes = Encoding.UTF8.GetBytes(post);
WebRequest.ContentLength = postBytes.Length;
var postDataStream = WebRequest.GetRequestStream();
postDataStream.Write(postBytes, 0, postBytes.Length);
postDataStream.Close();
}
WebResponse wresp = null;
try
{
wresp = WebRequest.GetResponse();
var downStream = wresp.GetResponseStream();
if (downStream != null)
{
using (var downReader = new StreamReader(downStream))
{
responseStr = downReader.ReadToEnd();
}
}
return responseStr;
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
if (wresp != null)
{
wresp.Close();
wresp = null;
}
}
finally
{
WebRequest = null;
}
return responseStr;
}
}
Getting crfs-token:
private string GetToken()
{
string token = "";
String sourcestring = hc.GetData(loginURL);
Regex metaTag = new Regex(#"<meta[\s]+[^>]*?name[\s]?=[\s""']+(.*?)[\s""']+content[\s]?=[\s""']+(.*?)[""']+.*?>");
foreach (Match m in metaTag.Matches(sourcestring))
{
if (m.Groups[2].Value.Contains("token"))
{
continue;
}
token = m.Groups[2].Value;
}
return token;
}
Custom keyvaluepair
private string PostParam(Dictionary<string, string> data)
{
var sb = new StringBuilder();
var p = new List<string>();
foreach (KeyValuePair<string, string> pair in data)
{
sb.Clear();
sb.Append(pair.Key).Append("=").Append(pair.Value);
p.Add(sb.ToString());
}
var pp = string.Join("&", p);
return pp;
}
Login:
private HttpClient hc = new HttpClient();
Dictionary data = new Dictionary();
data.Add("utf8", "✓");
data.Add("authenticity_token", GetToken());
data.Add("plan", "");
data.Add("email", "email");
data.Add("password", "password");
hc.PostData(sessionURL,WebUtility.UrlEncode(PostParam(data)), loginURL);
Can someone tell me what I'm doing wrong? if I look the request header when trying to login to strava in browser its the same but still it doesn't log me.
I found the problem.
You need to encode only the token (and the UTF8 character), not the full post data.
This works for me (for some reason I need to run the code two times)
// First time says "logged out"
Dictionary<string, string> data = new Dictionary<string, string>();
data.Add("utf8", WebUtility.UrlEncode("✓"));
string token = GetToken();
string tokenEncoded = WebUtility.UrlEncode(token);
data.Add("authenticity_token", tokenEncoded);
data.Add("plan", "");
data.Add("email", "youremail");
data.Add("password", "yourpwd");
data.Add("remember_me", "on");
string parameters = PostParam(data);
hc.PostData(sessionURL, parameters, loginURL);
// Second time logs in
Dictionary<string, string> data = new Dictionary<string, string>();
data.Add("utf8", WebUtility.UrlEncode("✓"));
string token = GetToken();
string tokenEncoded = WebUtility.UrlEncode(token);
data.Add("authenticity_token", tokenEncoded);
data.Add("plan", "");
data.Add("email", "youremail");
data.Add("password", "yourpwd");
data.Add("remember_me", "on");
string parameters = PostParam(data);
hc.PostData(sessionURL, parameters, loginURL);
Note:
// Keep this on default value "true"
//WebRequest.AllowAutoRedirect = false;
remark: you can use this code (see my previous post) to change activity's status (privacy) after logging in:
Dictionary<string, string> data2 = new Dictionary<string, string>();
data2.Add("utf8", WebUtility.UrlEncode("✓"));
string token2 = GetToken();
string tokenEncoded2 = WebUtility.UrlEncode(token2);
data2.Add("_method", "patch");
data2.Add("authenticity_token", tokenEncoded2);
data2.Add("activity%5Bvisibility%5D", "only_me"); // or "followers_only"
string parameters2 = PostParam(data2);
hc.PostData("https://www.strava.com/activities/youractivityID", parameters2, loginURL);

Set the Authentication Token Profile Header

I have an OpenIdConnect Server I'm connecting to an I would like to forward token data the first time logging in to be stored on the server. Currently I'm doing this to forward the access token
var xhr = new XMLHttpRequest();
xhr.open("GET", url);
xhr.onload = function () {
log(xhr.status, JSON.parse(xhr.responseText));
}
xhr.setRequestHeader("Authorization", "Bearer " + user.access_token);
xhr.send();
I want to send the Profile Data as well but I don't know the proper header.
How can I do something like this:
xhr.setRequestHeader("Authorization-Profile", "Bearer " + user.profile);
Does anyone know the proper header so I can add these claims to the access token.
Here is an example of what we did in one of our project:
Created a common API response class as below:
public class ApiCommonResponse
{
public object Object { get; set; }
public int httpStatus { get; set; }
public string httpErrorMessage { get; set; }
}
And a generic method to call GET and POST API endpoints. This method will map the response to the supplied data model and will return you the object.
public static ApiCommonResponse GetApiData<T>(string token, T dataModel, string apiEndPoint = null)
{
var responseText = "";
var apiCommonResponse = new ApiCommonResponse();
if (apiEndPoint != null)
{
var request = (HttpWebRequest)WebRequest.Create(apiEndPoint);
request.Method = "GET";
request.ContentType = "application/json";
request.Headers.Add("Authorization", "Bearer " + token);
request.Headers.Add("X-Api-Version", "");
try
{
var httpResponse = (HttpWebResponse)request.GetResponse();
var stream = httpResponse.GetResponseStream();
if (stream != null)
{
using (var streamReader = new StreamReader(stream))
{
responseText = streamReader.ReadToEnd();
}
}
}
catch (WebException we)
{
var stream = we.Response.GetResponseStream();
if (stream != null)
{
var resp = new StreamReader(stream).ReadToEnd();
dynamic obj = JsonConvert.DeserializeObject(resp);
throw new Exception(obj.ToString());
}
}
}
var jsonSettings = new JsonSerializerSettings { MissingMemberHandling = MissingMemberHandling.Ignore };
apiCommonResponse.Object = JsonConvert.DeserializeObject<T>(responseText, jsonSettings);
apiCommonResponse.httpStatus = 0;
return apiCommonResponse;
}
public static ApiCommonResponse PostApiData<T>(string username, string token, T dataModel, string apiEndPoint = null)
{
var apiCommonResponse = new ApiCommonResponse();
if (apiEndPoint == null) return null;
var webRequest = WebRequest.Create(apiEndPoint);
webRequest.Method = "POST";
webRequest.Timeout = 20000;
webRequest.ContentType = "application/json";
request.Headers.Add("Authorization", "Bearer " + token);
webRequest.Headers.Add("X-Api-Version", "");
using (var requeststreams = webRequest.GetRequestStream())
{
using (var sw = new StreamWriter(requeststreams))
{
sw.Write(JsonConvert.SerializeObject(dataModel));
}
}
try
{
var httpStatus = (((HttpWebResponse)webRequest.GetResponse()).StatusCode);
var httpMessage = (((HttpWebResponse)webRequest.GetResponse()).StatusDescription);
using (var s = webRequest.GetResponse().GetResponseStream())
{
if (s == null) return null;
using (var sr = new StreamReader(s))
{
var responseObj = sr.ReadToEnd();
if (!string.IsNullOrEmpty(responseObj))
{
apiCommonResponse = JsonConvert.DeserializeObject<ApiCommonResponse>(responseObj);
}
}
apiCommonResponse.httpStatus = (int)httpStatus;
apiCommonResponse.httpErrorMessage = httpMessage;
apiCommonResponse.Object = apiCommonResponse.Object;
}
}
catch (WebException we)
{
var stream = we.Response.GetResponseStream();
if (stream != null)
{
var resp = new StreamReader(stream).ReadToEnd();
dynamic obj = JsonConvert.DeserializeObject(resp);
throw new Exception(obj.ToString());
}
}
return apiCommonResponse;
}

Incorporating GZIP into Web API

I have a C# application that communicates to my Web API service fine when there is no GZIP being used. When I apply GZIP, my [FromBody] variable is null and I can't seem to pinpoint why this is happening.
If I put a breakpoint in the web api, I can see that data is null when GZIP is applied. When it's not, it is populated.
Any input would be greatly appreciated. If you need any other code examples please let me know
Update Routing Class
public class UpdateRouting
{
public UpdateRouting()
{
Routes = new List<Routing>();
}
public List<Routing> Routes { get; set; }
}
Collect data
private void btnTestRoutingSend_Click(object sender, EventArgs e)
{
try
{
var response = new ResponseObj();
using (var mainContext = new EntityModel())
{
var nodes = mainContext.Nodes.Where(x => x.SystemType == "Dev").ToList();
var updRouting = new UpdateRouting();
foreach (var n in nodes)
{
using (var nodeContext = new EntityModel(connString))
{
var accounts = nodeContext.Accounts;
foreach (var acct in accounts)
{
//code to map accounts to route
}
var customers = nodeContext.Customers;
foreach (var cust in customers)
{
//code to map customer to route
}
}
}
response = webcontext.SendPost<ResponseObj>(updRouting, "Routing", "SyncRouting");
}
}
catch (Exception ex)
{
string msg = ex.Message;
}
}
SendPost Method
public T SendPost<T>(object update, string controller, string action)
{
var url = _baseURL + String.Format("api/{0}/{1}", controller, action), update);
T retval = default(T);
var uri = new Uri(_baseUri, url);
try
{
var request = GetRequest(uri, method);
if (content != null)
{
var json = GetBodyString(content);
if (!_useCompression)
{
using (var streamWriter = new StreamWriter(request.GetRequestStream()))
{
var json = GetBodyString(content);
streamWriter.Write(json);
streamWriter.Flush();
streamWriter.Close();
}
} else{
using (var gzipStream = new GZipStream(request.GetRequestStream(), CompressionMode.Compress, false))
{
using (var streamWriter = new StreamWriter(gzipStream))
{
streamWriter.Write(json);
streamWriter.Flush();
streamWriter.Close();
}
}
httpWebRequest.Headers.Add("Content-Encoding", "gzip");
}
}
var response = (HttpWebResponse)request.GetResponse();
using (var streamReader = new StreamReader(response.GetResponseStream()))
{
var result = streamReader.ReadToEnd();
retval = GetObject<T>(result);
}
}
catch (Exception e)
{
_log.Error(string.Format("Error sending {0} request to {1}", method, uri.ToString()), e);
}
return retval;
}
Get Request
protected virtual HttpWebRequest GetRequest(Uri uri, string method)
{
var retval = (HttpWebRequest)WebRequest.Create(uri);
retval.Timeout = 1000 * 60 * 5;
retval.Accept = "application/json";
retval.KeepAlive = true;
retval.ContentType = "application/json";
retval.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
retval.Method = method;
return retval;
}
API Call
[System.Web.Http.HttpPost]
[Route("SyncRouting")]
public ResponseObj SyncRouting([FromBody] UpdateRouting data)
{
var response = new ResponseObj();
try
{
// do stuff
response.Successful = true;
return response;
}
catch (Exception e)
{
response.Successful = false;
response.Errors.Add(new ErrorMsg
{
Message = "Error - " + e.ToString(),
ExceptionMessage = e.Message,
StackTrace = e.StackTrace,
InnerException = e.InnerException.ToString()
});
return response;
}
}

Toggl API v8 with .NET

I'm trying to use the new Toggl API (v8) with .NET C#. I've based my code on the example from litemedia (http://litemedia.info/connect-to-toggl-api-with-net), but it was originally created for version 1 of the API.
private const string TogglTasksUrl = "https://www.toggl.com/api/v8/tasks.json";
private const string TogglAuthUrl = "https://www.toggl.com/api/v8/me"; //sessions.json";
private const string AuthenticationType = "Basic";
private const string ApiToken = "user token goes here";
private const string Password = "api_token";
public static void Main(string[] args)
{
CookieContainer container = new CookieContainer();
var authRequest = (HttpWebRequest)HttpWebRequest.Create(TogglAuthUrl);
authRequest.Credentials = CredentialCache.DefaultCredentials;
authRequest.Method = "POST";
authRequest.ContentType = "application/x-www-form-urlencoded";
authRequest.CookieContainer = container;
string value = ApiToken; //= Convert.ToBase64String(Encoding.Unicode.GetBytes(ApiToken));
value = string.Format("{1}:{0}", Password, value);
//value = Convert.ToBase64String(Encoding.Unicode.GetBytes(value));
authRequest.ContentLength = value.Length;
using (StreamWriter writer = new StreamWriter(authRequest.GetRequestStream(), Encoding.ASCII))
{
writer.Write(value);
}
try
{
var authResponse = (HttpWebResponse)authRequest.GetResponse();
using (var reader = new StreamReader(authResponse.GetResponseStream(), Encoding.UTF8))
{
string content = reader.ReadToEnd();
}
HttpWebRequest tasksRequest = (HttpWebRequest)HttpWebRequest.Create(TogglTasksUrl);
tasksRequest.CookieContainer = container;
//var jsonResult = string.Empty;
var tasksResponse = (HttpWebResponse)tasksRequest.GetResponse();
MemoryStream ms = new MemoryStream();
tasksResponse.GetResponseStream().CopyTo(ms);
//using (var reader = new StreamReader(tasksResponse.GetResponseStream(), Encoding.UTF8))
//{
// jsonResult = reader.ReadToEnd();
//}
ms.Seek(0, SeekOrigin.Begin);
DataContractJsonSerializer ser = new DataContractJsonSerializer(typeof(Task));
var tasks = ser.ReadObject(ms) as List<Task>;
ms.Close();
//var tasks = DataContractJsonConvert.DeserializeObject<Task[]>(jsonResult);
foreach (var task in tasks)
{
Console.WriteLine(
"{0} - {1}: {2} starting {3:yyyy-MM-dd HH:mm}",
task.Project.Name,
task.Description,
TimeSpan.FromSeconds(task.Duration),
task.Start);
}
}
catch (System.Exception ex)
{
throw;
}
}
The following line is returning a 404 error.
var authResponse = (HttpWebResponse)authRequest.GetResponse();
Here is code that works. Since I was looking for this answer recently there might still be others as lost as me.
Notes: I used Encoding.Default.GetBytes() because Encoding.Unicode.GetBytes() did not give me a correct result on my .NET string. I hope it doesn't depend on the default setup of Visual Studio.
The content-type is "application/json".
Sorry, I haven't tried a POST version yet.
string ApiToken = "user token goes here";
string url = "https://www.toggl.com/api/v8/me";
string userpass = ApiToken + ":api_token";
string userpassB64 = Convert.ToBase64String(Encoding.Default.GetBytes(userpass.Trim()));
string authHeader = "Basic " + userpassB64;
HttpWebRequest authRequest = (HttpWebRequest)WebRequest.Create(url);
authRequest.Headers.Add("Authorization", authHeader);
authRequest.Method = "GET";
authRequest.ContentType = "application/json";
//authRequest.Credentials = CredentialCache.DefaultNetworkCredentials;
try
{
var response = (HttpWebResponse)authRequest.GetResponse();
string result = null;
using (Stream stream = response.GetResponseStream())
{
StreamReader sr = new StreamReader(stream);
result = sr.ReadToEnd();
sr.Close();
}
if (null != result)
{
System.Diagnostics.Debug.WriteLine(result.ToString());
}
// Get the headers
object headers = response.Headers;
}
catch (Exception e)
{
System.Diagnostics.Debug.WriteLine(e.Message + "\n" + e.ToString());
}

Categories

Resources