I'm trying to get the HTML of a website that is javascript heavy, then use HtmlAgilityPack to analyze the HTML received.
I am getting the desired response, however, when trying to load the data using this line
var docB = fromDoc.LoadHtml(docBContent);
Visual studio is having none of it, saying the following:
cannot assign void to an implicitly-typed variable
Full code:
var fromDoc = new HtmlDocument();
var docBContent = await renderHtmlAsync(url2);
var docB = fromDoc.LoadHtml(docBContent); // error
public static async Task<string> renderHtmlAsync(string url2)
{
await new BrowserFetcher().DownloadAsync(BrowserFetcher.DefaultChromiumRevision);
Browser browser = await Puppeteer.LaunchAsync(new LaunchOptions
{
Headless = true
});
var page = await browser.NewPageAsync();
page.DefaultTimeout = 0;
var navigation = new NavigationOptions
{
Timeout = 0,
WaitUntil = new[] {
WaitUntilNavigation.DOMContentLoaded }
};
await page.GoToAsync(url2, navigation);
var content = page.GetContentAsync();
return await content;
}
you dosnt need asign fromDoc.LoadHtml(docBContent); to variable.
var fromDoc = new HtmlDocument();
var docBContent = await renderHtmlAsync(url2);
fromDoc.LoadHtml(docBContent);
and now you can use formDoc.
like:
var data = fromDoc.DocumentNode.SelectSingleNode("//div");
Related
I use a strange API that sends JSON objects in kind of chunks (line by line).
I need to be able to read each line of the response to my request asynchronously. I'm already trying to display in the console each of these requests.
The code below is the main one (the main{} function only calls this function). I want this code to be executed asynchronously too, that's why the function is declared as async task.
When I launch the application, the query runs fine, but the program doesn't display the lines one by one: it waits for the end of the query to display everything at once.
I'm very new to async/streams, sorry if the code is so bad.
public async Task test()
{
HttpClient client = new HttpClient();
var values = new Dictionary<string, string> // POST data
{
{ "api", "v2" },
{ "field1", "data1" },
{ "field2", "data2" }
};
var content = new FormUrlEncodedContent(values);
var response = await client.PostAsync("http://api.domain/post", content);
using (var theStream = await response.Content.ReadAsStreamAsync())
{
StreamReader theStreamReader = new StreamReader(theStream);
string theLine = null;
while ((theLine = await theStreamReader.ReadLineAsync()) != null)
{
Console.WriteLine("----- New Line:"); //Just for the eyes
Console.WriteLine(theLine);
}
};
}
You are missing HttpCompletionOption.ResponseHeadersRead, which will only block while reading the headers, and the content comes through asynchronously. You must use SendAsync and a HttpRequestMessage for this.
Also you are missing some using blocks, and HttpClient should be cached in a static field.
static HttpClient _client = new HttpClient();
public async Task test()
{
var values = new Dictionary<string, string> // POST data
{
{ "api", "v2" },
{ "field1", "data1" },
{ "field2", "data2" }
};
using (var content = new FormUrlEncodedContent(values))
using (var request = new HttpRequestMessage(HttpMethod.Post, "http://api.domain/post"){ Content = content })
using (var response = await _client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead))
using (var theStream = await response.Content.ReadAsStreamAsync())
using (var theStreamReader = new StreamReader(theStream))
{
string theLine = null;
while ((theLine = await theStreamReader.ReadLineAsync()) != null)
{
Console.WriteLine("----- New Line:"); //Just for the eyes
Console.WriteLine(theLine);
}
};
}
I am trying to scrape Taobao website with Puppeteer Sharp.
Here is the code:
private static async Task SurfWithPuppeteer()
{
var options = new LaunchOptions{ Devtools = true };
Console.WriteLine("Downloading chromium");
await new BrowserFetcher().DownloadAsync(BrowserFetcher.DefaultRevision);
Console.WriteLine("Navigating to Hacker News");
using (var browser = await Puppeteer.LaunchAsync(options))
using (var page = await browser.NewPageAsync())
{
page.DefaultNavigationTimeout = 50000;
await page.GoToAsync("https://login.tmall.com/?spm=875.7931836/B.a2226mz.1.66144265pHmhvt&redirectURL=https%3A%2F%2Fwww.tmall.com%2F");
var frameElement= await page.QuerySelectorAsync("#J_loginIframe");
//var frameElement = await page.QuerySelectorAsync("div#mallPage iframe");
//var frameElement = await page.Frames.Select(f=>f.QuerySelectorAsync("#J_loginIframe")).FirstOrDefault();
var frame = await frameElement.ContentFrameAsync();
var frameContent = await frame.GetContentAsync();
await frame.TypeAsync("#TPL_username_1", "compuwizpiyu");
await frame.TypeAsync("#TPL_password_1", "Priyanka24$");
var btn = await frame.QuerySelectorAsync("#J_SubmitStatic");
await btn.ClickAsync();
var res= await frame.WaitForNavigationAsync();
var t= await frame.GetContentAsync();
//var url = page.Url;
}
}
But I am unable to navigate to the frame that has the login form (frame has no name, only src and id).
I have tried to check the frames with page.Frames, but since the iframes have no name, difficult to find the correct frame I am looking for.
I have tried couple of other options too like :
var frameElement = await page.QuerySelectorAsync("div#mallPage iframe");
var frameElement = await page.Frames.Select(f=>f.QuerySelectorAsync("#J_loginIframe")).FirstOrDefault()
But still unable to get the intended frame. Please help me with what is wrong here.
This may be due to CORS.
Try the following code.
var options = new LaunchOptions
{
Devtools = true,
Args = new[]
{
"--disable-web-security",
"--disable-features=IsolateOrigins,site-per-process",
},
};
i have the following API json request
"myjsonrequest": {
"ServiceKey": "Hello",
"Identityvals": {
"IDName": "regnum",
"IDValue": "112233"
}
}
any simple way to get the response , im using ASP.net c#
i tried this code
HttpClient client = new HttpClient();
string x = "{'IDName','regnum'},{'IDValue','112233'}";
var Keys = new Dictionary<string, string>
{
{ "ServiceKey", "hello" },
{ "PractitionerIdentity",x}
};
var content = new FormUrlEncodedContent(Keys);
var response = await client.PostAsync("https://apiurl", content);
var responseval = await response.Content.ReadAsStringAsync();
Try this :
var json = new {
ServiceKey = "",
PractitionerIdentity = new {
IDName = "" ,
IDValue = ""
}
};
HttpClient client = new HttpClient();
var content = new StringContent(json, Encoding.UTF8, "application/json")
var response = await client.PostAsync("https://apiurl", content);
Your Json data should be saved to a model in the following way:
public class YourJsonData{
public string ServiceKey {get; set;}
//add other names
}
The best thing about this is that if you call your object, you get a variable back for easy usage.
then you can add it in a task:
public async Task<List<YourJsonData>> GetJsonAsync(CancellationToken cancellationToken = default)
{
using (var client = new HttpClient())
{
//Make the request, and ensure we can reach it
var response = await client.GetAsync(yourJosnUrl, cancellationToken);
if (response.IsSuccessStatusCode)
{
//Read the actual stream (download the content)
var content = await response.Content.ReadAsStringAsync();
//Make sure we do have some valid content before we try to deserialize it
if (string.IsNullOrEmpty(content))
{
return new List<YourJsonData>();
}
//Deserialize into a list of yourjsondata
return JsonConvert.DeserializeObject<List<YourJsonData>>(content);
}
}
return new List<YourJsonData>();
}
also if you are lazy, you can replace YourJsonData with dynamic. the downpart here is that you won't be able to see what you are revering to.
I am attempting to send POST data to my server and get back a response. For some reason, no POST data is actually getting sent. A request is being sent to my server but the POST array is empty.
Here is my code for sending the request:
public class GlobalMethods
{
public async Task<string> callAjax(string mthd,NameValueCollection parameters)
{
var client = new HttpClient();
var content = JsonConvert.SerializeObject(parameters);
var result = await client.PostAsync("http://dev.adex-intl.com/adex/mvc/receiving/"+mthd, new StringContent(content)).ConfigureAwait(false);
var tokenJson = "";
if (result.IsSuccessStatusCode)
{
tokenJson = await result.Content.ReadAsStringAsync();
}
return tokenJson;
}
}
And here is my code that calls the above method:
public void loginPressed(object sender, EventArgs e)
{
if(String.IsNullOrEmpty(badge.Text)) {
DisplayAlert("Error", "Enter your badge number", "Ok");
} else {
IsBusy = true;
NameValueCollection parameters = new NameValueCollection();
parameters["badgetNumber"] = badge.Text;
GlobalMethods globalMethods = new GlobalMethods();
var results = globalMethods.callAjax("login", parameters);
}
}
I'm not sure what I'm doing wrong. Also, I'm a newbie to Xamarin and C# so I'm not even sure if the way I am attempting to do things is the best way.
You haven't specify the type of content that you want to send, in your case it's 'application/json', you can set it like that:
"var client = new HttpClient();
var content = new StringContent(JsonConvert.SerializeObject(parameters));
content.Headers.ContentType = new MediaTypeHeaderValue("application/json");". Also, I would suggest you to write code like that:
var uri = new Uri(url);
using (var body = new StringContent(JsonConvert.SerializeObject(data)))
{
body.Headers.ContentType = new MediaTypeHeaderValue("application/json");
var request = new HttpRequestMessage
{
Version = new Version(1, 0),
Content = body,
Method = HttpMethod.Post,
RequestUri = uri
};
try
{
using (var response = await _client.SendAsync(request,cancellationToken))
{
if (response.IsSuccessStatusCode)
{
//Deal with success response
}
else
{
//Deal with non-success response
}
}
}
catch(Exception ex)
{
//Deal with exception.
}
}
You can use PostAsync for async sending data to server. your code should be something like this:
HttpClient client = new HttpClient();
var values = new Dictionary<string, string>
{
{ "p1", "data1" },
{ "p2", "data2" }
};
var content = new FormUrlEncodedContent(values);
var response = await client.PostAsync("http://www.example.com/index.php", content);
var responseString = await response.Content.ReadAsStringAsync();
In a Xamarin Form app, from the Master Page menu, in the Portable project, I call Content page InfoScreen into the Detail. InfoScreen code looks something like this:
public InfoScreen()
{
InitializeComponent();
ListAccounts();
}
public void ListAccounts()
{
SearchAccounts SA = new SearchAccounts();
SearchAccountRequest searchAccountRequest = new SearchAccountRequest("000123456789", "","","","");
var searchAccountResult = SA.SearchAccountAsync(searchAccountRequest);
}
Notice that on load it calls ListAccounts, located in a class in the Portable project, which looks as follows:
public async Task<SearchAccountResult> SearchAccountAsync(SearchAccountRequest searchAccountRequest)
{
HttpClient client = new HttpClient();
client.BaseAddress = new Uri("http://localhost:00/api/");
string jsonData = JsonConvert.SerializeObject(searchAccountRequest);
var content = new StringContent(jsonData, Encoding.UTF8, "application/json");
var response = await client.PostAsync("SearchForAccount", content);
var result = response.Content.ReadAsStringAsync().Result;
if (result != "")
{
var sessionResponseJson = JsonConvert.DeserializeObject<SearchAccountResult>(result);
}
However, when var response = await client.PostAsync("SearchForAccount", content); gets hit, it just goes back to InfoScreen and continues to load. The API is never hit, which I'm running locally on debug mode. I tested with Postman and it's working correctly.
I also tried:
var client = new HttpClient();
var data = JsonConvert.SerializeObject(searchAccountRequest);
var content = new StringContent(data, Encoding.UTF8, "application/json");
var response = await client.PostAsync("http://localhost:00/api/SearchForAccount", content);
if (response.IsSuccessStatusCode)
{
var result = JsonConvert.DeserializeObject<SearchAccountResult>(response.Content.ReadAsStringAsync().Result);
}
else
{
//
}
Same results. What am I missing?
Thanks
===============
Tried calling the DEV server, utilizing IP, works fine from Postman. Only thing I'm getting in the app is
Id = 1, Status = WaitingForActivation, Method = {null}