I tried to scrap the page using HtmlAgilityPack.dll but some url get into the function, I got the error and I can't catch it in try-catch block. So can anyone help me out?
Error:
An unhandled exception of type 'System.StackOverflowException' occurred in HtmlAgilityPack.dll
public void HtmlLoad(string url)
{
try
{
HttpWebRequest myHttpWebRequest = null; //Declare an HTTP-specific implementation of the WebRequest class.
HttpWebResponse myHttpWebResponse = null; //Declare an HTTP-specific implementation of the WebResponse class
//Create Request //
myHttpWebRequest = (HttpWebRequest)HttpWebRequest.Create(url);
myHttpWebRequest.Method = "GET";
myHttpWebRequest.ContentType = "text/html; encoding='utf-8'";
//Get Response
myHttpWebResponse = (HttpWebResponse)myHttpWebRequest.GetResponse();
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
Stream data = myHttpWebResponse.GetResponseStream();//client.OpenRead(url);
doc.Load(data);
data.Close();
}
catch (Exception ex) { throw ex; }
}
You can try this clean
public static async Task<int> HtmlLoadAsync(string url/*, bool addUserAgent = false*/)
{
try
{
var client = new HttpClient();
//if (addUserAgent) OPTIONAL
//{
// client.DefaultRequestHeaders.UserAgent.ParseAdd(UserAgent);
//}
//client.Timeout = TimeOut;
var response = client.GetStringAsync(url);
var urlContents = await response;
var document = new HtmlAgilityPack.HtmlDocument();
document.LoadHtml(urlContents);
// process document now
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
return 0;
}
Now call it
private async void Process()
{
await HtmlLoadAsync("http://....");
}
Related
I've a NET.Core API with simple test method:
public async Task<IActionResult> TestApi()
{
try
{
throw new UnauthorizedAccessException("My custom error");
return Ok();
}
catch (UnauthorizedAccessException ex)
{
return StatusCode(401,ex.Message);
}
catch (Exception ex)
{
throw;
}
}
I need to retrieve the message from a client like this:
var request = WebRequest.Create($"{baseUrl}{url}") as HttpWebRequest;
request.Method = "POST";
request.ContentType = "application/json";
request.Expect = "application/json";
request.ContentLength = 0;
if (parameters != null)
{
request.ContentLength = serializedObject.Length;
using (var writer = new StreamWriter(request.GetRequestStream()))
{
writer.Write(serializedObject);
}
}
var response = request.GetResponse() as HttpWebResponse;
var responseEncoding = Encoding.GetEncoding(response.CharacterSet);
using (var sr = new StreamReader(response.GetResponseStream(), responseEncoding))
{
var result = sr.ReadToEnd();
return JsonConvert.DeserializeObject<T>(result);
}
Now request.GetResponse() as HttpWebResponse returns me:
The remote server returned an error: (401) Unauthorized.
instead of My custom error. Can someone point me in the right direction?
Here's a pared-down example which reads your custom message. Your message is returned in the response stream.
try
{
var response = request.GetResponse() as HttpWebResponse;
}
catch (WebException ex) // this exception is thrown because of the 401.
{
var responseStream = ex.Response.GetResponseStream();
using (var reader = new StreamReader(responseStream))
{
var message = reader.ReadToEnd();
}
}
Return an ActionResult
Task<ActionResult>
You can then wrap up the unauthorized error in an UnauthorizedObjectResult
return Unauthorized(new UnauthorizedObjectResult(errorModel));
I am trying to get the access token for the feed.Below is a code, i used to get the access token.
public async Task<string> GetAccessToken()
{
string postString = String.Format("username={0}&password={1}&grant_type=password", "userName", "pwd");
string url = "http://example.net/Token";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url.ToString());
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
UTF8Encoding utfenc = new UTF8Encoding();
byte[] bytes = utfenc.GetBytes(postString);
try
{
HttpWebResponse webResponse = (HttpWebResponse)(await request.GetResponseAsync());
Stream responseStream = webResponse.GetResponseStream();
StreamReader responseStreamReader = new StreamReader(responseStream);
string result = responseStreamReader.ReadToEnd();//parse token from result
}
catch(Exception ex)
{
}
return "";
}
The error below
"An error occurred while sending the request. The text associated with this error code could not be found.
The server name or address could not be resolved"
is throwing while it executes the below code
HttpWebResponse webResponse = (HttpWebResponse)(await request.GetResponseAsync());
Please help me to solve the issue
Try this if you are using POST request
public async Task<string> GetAccessToken()
{
string postString = String.Format("username={0}&password={1}&grant_type=password", "userName", "pwd");
try
{
using (var httpClient = new HttpClient())
{
var request1 = new HttpRequestMessage(HttpMethod.Post, "FeedURL");
request1.Content = new StringContent(postString);
var response = await httpClient.SendAsync(request1);
var result1 = await response.Content.ReadAsStringAsync();
result1 = Regex.Replace(result1, "<[^>]+>", string.Empty);
var rootObject1 = JObject.Parse(result1);
string accessToken = rootObject1["access_token"].ToString();
}
}
catch (Exception ex)
{
}
}
I'm trying use HttpWebRequest, and my BeginGetRequestStream works but it never enters the BeginGetResponse function and i have no idea why.. i've searched for a couple of hours and have not found a solution that works
public void Initialize(IScheduler scheduler)
{
if(_isCloud)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(_cloudMappingServer + "/Mapping/GetAllCentralPoints");
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "POST";
request.BeginGetRequestStream(new AsyncCallback(ReleaseReadCallback), request);
// Instruct the thread to wait until we resume it
_waitHandle.WaitOne();
_waitHandle.Dispose();
}
}
private void ReleaseReadCallback(IAsyncResult asynchronousResult)
{
try
{
HttpWebRequest httpRequest = (HttpWebRequest)asynchronousResult.AsyncState;
using (Stream postStream = httpRequest.EndGetRequestStream(asynchronousResult))
{
using (MemoryStream memStream = new MemoryStream())
{
string queryString = string.Empty;
byte[] bytes = System.Text.Encoding.UTF8.GetBytes(queryString);
memStream.Write(bytes, 0, bytes.Length);
memStream.Position = 0;
byte[] tempBuffer = new byte[memStream.Length];
memStream.Read(tempBuffer, 0, tempBuffer.Length);
postStream.Write(tempBuffer, 0, tempBuffer.Length);
}
}
httpRequest.BeginGetResponse(new AsyncCallback(ReleaseResponseCallback), httpRequest);
}
catch (Exception ex)
{
var test = ex;
}
}
private void ReleaseResponseCallback(IAsyncResult asynchronousResult)
{
HttpWebRequest responseRequest = (HttpWebRequest)asynchronousResult.AsyncState;
string responseString = string.Empty;
try
{
using (HttpWebResponse resp = (HttpWebResponse)responseRequest.EndGetResponse(asynchronousResult))
{
using (StreamReader streamRead = new StreamReader(resp.GetResponseStream()))
{
responseString = streamRead.ReadToEnd();
try
{
JsonSerializerSettings settings = new JsonSerializerSettings();
List<CentralPointViewModel> _allCentralPointViewModel = JsonConvert.DeserializeObject<List<CentralPointViewModel>>(responseString, settings);
}
catch (JsonReaderException)
{
responseString = responseString.Replace('\"'.ToString(), string.Empty);
string[] responseArray = responseString.Split(';');
}
catch (JsonSerializationException)
{
responseString = responseString.Replace('\"'.ToString(), string.Empty);
}
}
}
}
catch (Exception ex)
{
}
}
It never enters the ReleaseResponseCallback function! I am able to make my server call but the response never reaches me or I am not properly receiving it.. Any help is appreciated
I am trying to call a web service
which returns a JSON response. So far my code is working fine for GET Request but now services on server are POST and I have no idea how to do that! Below is the code for GET Request:
private void callSigninWebservice()
{
setProgressIndicator(true);
SystemTray.ProgressIndicator.Text = "Signing in please wait";
try
{
WebClient webClient = new WebClient();
Uri uri = new Uri(GlobalVariables.URL_USER +
GlobalVariables.URL_STUDENT_SIGNIN_MODE_LOGIN +
GlobalVariables.URL_EMAIL + tbUsername.Text +
GlobalVariables.URL_PASSWORD + tbPassword.Password);
webClient.DownloadStringCompleted += new DownloadStringCompletedEventHandler(webClient_DownloadStringCompleted);
webClient.DownloadStringAsync(uri);
}
catch (Exception ex)
{
MessageBox.Show(ex.Message + "error came here 1");
}
}
For Response
void webClient_DownloadStringCompleted(object sender, DownloadStringCompletedEventArgs e)
{
try
{
JObject parentObj = JObject.Parse(e.Result);
String strResult = (String)parentObj[SigninData.JSON_result];
bool bolresult = strResult.Equals(SigninData.JSON_result_success, StringComparison.Ordinal);
if (bolresult)
{
JObject dataObj = (JObject)parentObj[SigninData.JSON_data];
setUserData(dataObj);
NavigationService.Navigate(new Uri("/BasePage.xaml", UriKind.RelativeOrAbsolute));
}
else
{
String error = (String)parentObj[SigninData.JSON_data];
MessageBox.Show("Error : " + error);
}
setProgressIndicator(false);
}
catch (Exception)
{
setProgressIndicator(false);
}
}
This is a simple way of making a post request and receiving a response in JSON for future parse:
internal static async Task<String> GetHttpPostResponse(HttpWebRequest request, string postData)
{
String received = null;
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
byte[] requestBody = Encoding.UTF8.GetBytes(postData);
// ASYNC: using awaitable wrapper to get request stream
using (var postStream = await request.GetRequestStreamAsync())
{
// Write to the request stream.
// ASYNC: writing to the POST stream can be slow
await postStream.WriteAsync(requestBody, 0, requestBody.Length);
}
try
{
// ASYNC: using awaitable wrapper to get response
var response = (HttpWebResponse)await request.GetResponseAsync();
if (response != null)
{
var reader = new StreamReader(response.GetResponseStream());
// ASYNC: using StreamReader's async method to read to end, in case
// the stream i slarge.
received = await reader.ReadToEndAsync();
}
}
catch (WebException we)
{
var reader = new StreamReader(we.Response.GetResponseStream());
string responseString = reader.ReadToEnd();
Debug.WriteLine(responseString);
return responseString;
}
return received;
}
I am writing a program that will submit a XML to a website. The code written works fine, but sometimes it just stops working for some reason, throwing a
System.Net.ProtocolViolationException. I can close the program and re-run - it starts working again just fine.
Here is the code that I am using:
private string Summit(string xml)
{
string result = string.Empty;
StringBuilder sb = new StringBuilder();
try {
WebRequest request = WebRequest.Create(this.targetUrl);
request.Timeout = 800 * 1000;
RequestState requestState = new RequestState(xml);
requestState.Request = request;
request.ContentType = "text/xml";
// Set the 'Method' property to 'POST' to post data to a Uri.
requestState.Request.Method = "POST";
requestState.Request.ContentType = "text/xml";
// Start the Asynchronous 'BeginGetRequestStream' method call.
IAsyncResult r = (IAsyncResult)request.BeginGetRequestStream(new AsyncCallback(ReadCallBack), requestState);
// Pause the current thread until the async operation completes.
// Console.WriteLine("main thread waiting...");
allDone.WaitOne();
// Assign the response object of 'WebRequest' to a 'WebResponse' variable.
WebResponse response = null;
try {
response =request.GetResponse();
} catch (System.Net.ProtocolViolationException ex) {
response = null;
request.Abort();
request = null;
requestState = null;
return "";
}
//Console.WriteLine("The string has been posted.");
//Console.WriteLine("Please wait for the response...");
Stream streamResponse = response.GetResponseStream();
StreamReader streamRead = new StreamReader(streamResponse);
Char[] readBuff = new Char[256];
int count = streamRead.Read(readBuff, 0, 256);
//StringBuilder sb = new StringBuilder();
while (count > 0) {
String outputData = new String(readBuff, 0, count);
sb.Append(outputData);
count = streamRead.Read(readBuff, 0, 256);
}
// Close the Stream Object.
streamResponse.Close();
streamRead.Close();
//allDone.WaitOne();
// Release the HttpWebResponse Resource.
response.Close();
//return sb.ToString();
} catch (WebException webex) {
Debug.WriteLine(webex.Message);
} catch (System.Web.Services.Protocols.SoapException soapex) {
Debug.WriteLine(soapex.Message);
} catch (System.Net.ProtocolViolationException ex) {
Debug.WriteLine(ex.Message);
} catch (Exception ex) {
Debug.WriteLine(ex.Message);
}
return sb.ToString();
}
private static void ReadCallBack(IAsyncResult asyncResult)
{
try {
RequestState myRequestState = (RequestState)asyncResult.AsyncState;
WebRequest myWebRequest2 = myRequestState.Request;
// End of the Asynchronus request.
Stream responseStream = myWebRequest2.EndGetRequestStream(asyncResult);
//Convert the string into a byte array.
ASCIIEncoding encoder = new ASCIIEncoding();
byte[] ByteArray = encoder.GetBytes(myRequestState.Xml);
// Write data to the stream.
responseStream.Write(ByteArray, 0, myRequestState.Xml.Length);
responseStream.Close();
} catch (WebException e) {
Console.WriteLine("\nReadCallBack Exception raised!");
Console.WriteLine("\nMessage:{0}", e.Message);
Console.WriteLine("\nStatus:{0}", e.Status);
}
allDone.Set();
}
response =request.GetResponse() is when it fails and gives an error
You must provide a request body if you set ContentLength>0 or
SendChunked==true. Do this by calling [Begin]GetRequestStream before
[Begin]GetResponse.
Any help would be greatly appreciated.
This gets tricky since we're doing async calls.
Do this in the following order:
request.BeginGetRequestStream(new AsyncCallback(GetRequestStreamCallback), request)
Then in 'GetRequestStreamCallback(IAsyncResult asynchronousResult)' call:
request.BeginGetResponse(new AsyncCallback(GetResponseCallback), request)
Lastly, in the GetResponse, be sure to close the stream:
response.Close();
allDone.Set();
MSDN Does a really good job explaining it: http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.begingetrequeststream.aspx