I am trying to scrape Taobao website with Puppeteer Sharp.
Here is the code:
private static async Task SurfWithPuppeteer()
{
var options = new LaunchOptions{ Devtools = true };
Console.WriteLine("Downloading chromium");
await new BrowserFetcher().DownloadAsync(BrowserFetcher.DefaultRevision);
Console.WriteLine("Navigating to Hacker News");
using (var browser = await Puppeteer.LaunchAsync(options))
using (var page = await browser.NewPageAsync())
{
page.DefaultNavigationTimeout = 50000;
await page.GoToAsync("https://login.tmall.com/?spm=875.7931836/B.a2226mz.1.66144265pHmhvt&redirectURL=https%3A%2F%2Fwww.tmall.com%2F");
var frameElement= await page.QuerySelectorAsync("#J_loginIframe");
//var frameElement = await page.QuerySelectorAsync("div#mallPage iframe");
//var frameElement = await page.Frames.Select(f=>f.QuerySelectorAsync("#J_loginIframe")).FirstOrDefault();
var frame = await frameElement.ContentFrameAsync();
var frameContent = await frame.GetContentAsync();
await frame.TypeAsync("#TPL_username_1", "compuwizpiyu");
await frame.TypeAsync("#TPL_password_1", "Priyanka24$");
var btn = await frame.QuerySelectorAsync("#J_SubmitStatic");
await btn.ClickAsync();
var res= await frame.WaitForNavigationAsync();
var t= await frame.GetContentAsync();
//var url = page.Url;
}
}
But I am unable to navigate to the frame that has the login form (frame has no name, only src and id).
I have tried to check the frames with page.Frames, but since the iframes have no name, difficult to find the correct frame I am looking for.
I have tried couple of other options too like :
var frameElement = await page.QuerySelectorAsync("div#mallPage iframe");
var frameElement = await page.Frames.Select(f=>f.QuerySelectorAsync("#J_loginIframe")).FirstOrDefault()
But still unable to get the intended frame. Please help me with what is wrong here.
This may be due to CORS.
Try the following code.
var options = new LaunchOptions
{
Devtools = true,
Args = new[]
{
"--disable-web-security",
"--disable-features=IsolateOrigins,site-per-process",
},
};
Related
I'm trying to get the HTML of a website that is javascript heavy, then use HtmlAgilityPack to analyze the HTML received.
I am getting the desired response, however, when trying to load the data using this line
var docB = fromDoc.LoadHtml(docBContent);
Visual studio is having none of it, saying the following:
cannot assign void to an implicitly-typed variable
Full code:
var fromDoc = new HtmlDocument();
var docBContent = await renderHtmlAsync(url2);
var docB = fromDoc.LoadHtml(docBContent); // error
public static async Task<string> renderHtmlAsync(string url2)
{
await new BrowserFetcher().DownloadAsync(BrowserFetcher.DefaultChromiumRevision);
Browser browser = await Puppeteer.LaunchAsync(new LaunchOptions
{
Headless = true
});
var page = await browser.NewPageAsync();
page.DefaultTimeout = 0;
var navigation = new NavigationOptions
{
Timeout = 0,
WaitUntil = new[] {
WaitUntilNavigation.DOMContentLoaded }
};
await page.GoToAsync(url2, navigation);
var content = page.GetContentAsync();
return await content;
}
you dosnt need asign fromDoc.LoadHtml(docBContent); to variable.
var fromDoc = new HtmlDocument();
var docBContent = await renderHtmlAsync(url2);
fromDoc.LoadHtml(docBContent);
and now you can use formDoc.
like:
var data = fromDoc.DocumentNode.SelectSingleNode("//div");
I don't understand why the response comes only if I use CompleteAsync().
using var call = _apiClient.GetToken(headers: _threadContext.Metadata, deadline: DateTime.UtcNow.AddSeconds(5));
var keyReq = new GetTokenRequest()
{
Key = publicKey
};
var readTask = Task.Run(async () =>
{
await foreach(var message in call.ResponseStream.ReadAllAsync())
{
if (message.Challenge != null)
{
var challenge = message.Challenge.ToByteArray();
var signature = await VerifySignature(challenge);
var signReq = new GetTokenRequest
{
Signature = ByteString.CopyFrom(signature)
};
await call.RequestStream.WriteAsync(signReq);
await call.RequestStream.CompleteAsync();
}
else if (message.Token != null)
{
token = message.Token;
}
}
});
await call.RequestStream.WriteAsync(keyReq);
await readTask;
If I change the end with this, I receive messages but in the response the next WriteAsync fails because the stream is closed.
await call.RequestStream.WriteAsync(keyReq);
await call.RequestStream.CompleteAsync();
await readTask;
And if I doesn't complete the request, response message never comes.
Any idea ?
Note: the server is in go.
This code doesn't work with Grpc.Net.Client.Web only. With classic SocketHttpHandler it's ok. Problem is solved. thanks.
I have bulk of records to send push notifications by mobile device to other mobile devices.
For that I am using for loop inside task.Start(). I wonder whether task.Start() running in background or not? So that while sending remote notifications to mobile device at the same time I could do some other stuff too & it wont block mobile UI.
The code below I am using
var pushTask = new Task(() =>
{
if (myPushDataFilterd.Any())
{
var title = txtHomeworkTitle.Value.Trim();
for (int index = 0; index < myPushDataFilterd.Count; index++)
{
var row = myPushDataFilterd[index];
jData.Add("moduleName", "Homework");
jData.Add("organizationId", ddlOrganization.SelectedValue);
jData.Add("studentId", studentId);
jGcmData.Add("to", to);
jGcmData.Add("data", jData);
api = row.ServerKeyPush;
var url = new Uri("https://fcm.googleapis.com/fcm/send");
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.TryAddWithoutValidation("Authorization", "key=" + api);
var r = client.PostAsync(url, new StringContent(jGcmData.ToString(), Encoding.Default, "application/json")).Result;
}
}
}
});
pushTask.Start();
Actually this the web application part I am now using in mobile application. In mobile application do I have other better options too, to send notification?
There is no point in creating task at all. You are doing IO operations, so you can make use of already provided async api.
private async Task PostData()
{
if (myPushDataFilterd.Any())
{
var title = txtHomeworkTitle.Value.Trim();
using (var client = new HttpClient())
{
for (int index = 0; index < myPushDataFilterd.Count; index++)
{
var row = myPushDataFilterd[index];
jData.Add("moduleName", "Homework");
jData.Add("organizationId", ddlOrganization.SelectedValue);
jData.Add("studentId", studentId);
jGcmData.Add("to", to);
jGcmData.Add("data", jData);
api = row.ServerKeyPush;
var url = new Uri("https://fcm.googleapis.com/fcm/send");
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.TryAddWithoutValidation("Authorization", "key=" + api);
var r = await client.PostAsync(url, new StringContent(jGcmData.ToString(), Encoding.Default, "application/json"));
}
}
}
}
Task in general represents an asynchronous operation, so code inside a task block won't block unless you wanted to read its value by explicitly waiting, check the following for example:
Task myTask = new Task( () => Console.WriteLine("It is me myTask ^_^ "));
myTask.Start();
Console.WriteLine("Currently not waiting the output from myTask");
myTask.Wait();//Now I am waiting
//Output:
//Currently not waiting the output from myTask
//It is me myTask ^_^
Also you can create and start a task in one statement using Task.Run & TaskFactory.StartNew.
For more information about the differences in usage between them check the link.
I have an MVC5 application in which I use Owin to authentiate users,
now I'm trying to create a Windows 8 app for this application. But I can't find the best why to implement authentication. I found two possible solutions "Web Authentication Broker" and "Credential Locker" but I'm not sure yet
using (var client = new HttpClient())
{
HttpResponseMessage response = null;
var requestContent = string.Format("UserLogin={0}&UserPassword={1}", userName, password);
HttpContent content = new StringContent(requestContent);
content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
Task task = Task.Run(async () =>
{
response = await client.PostAsync(new Uri((string)ApplicationData.Current.LocalSettings.Values["LoginApiUrl"]), content);
});
task.Wait(); // Wait
var responseData = await response.Content.ReadAsStringAsync();
var deserializedResponse = JsonConvert.DeserializeObject<Dictionary<string, string>>(responseData);
if (deserializedResponse.ContainsKey("success") && Convert.ToBoolean(deserializedResponse["success"]))
{
LoadData();
}
}
I have the following code that i can call from my universal app, and it works fine
// create our request with our data
var data = new JsonObject();
data.Add("param", JsonValue.CreateStringValue("abc"));
// convert to bytes
var dataAsBytes = System.Text.Encoding.UTF8.GetBytes(data.Stringify());
var httpRequest = HttpWebRequest.Create(URL + "Add");
httpRequest.Method = "POST";
httpRequest.ContentType = "application/json;charset=utf-8";
using (var requestStream = await httpRequest.GetRequestStreamAsync())
{
requestStream.Write(dataAsBytes, 0, dataAsBytes.Length);
}
var httpResponse = await httpRequest.GetResponseAsync();
using (var responseStream = httpResponse.GetResponseStream()) {
....
}
But this code seems to stop on var httpResponse = await httpRequest.GetResponseAsync() when i execute this from a backgroundTask?
To setup my background task, i have the following code.
var taskRegistered = false;
var taskName = "Upload Background Task";
var background = Windows.ApplicationModel.Background;
var iter = background.BackgroundTaskRegistration.allTasks.first();
while (iter.hasCurrent) {
var task = iter.current.value;
if (task.name === taskName) {
taskRegistered = true;
break;
}
iter.moveNext();
}
if (taskRegistered) {
successCallback(); // my javascript success callback
} else {
Windows.ApplicationModel.Background.BackgroundExecutionManager.requestAccessAsync().then(function () {
var builder = new Windows.ApplicationModel.Background.BackgroundTaskBuilder();
builder.name = taskName;
builder.taskEntryPoint = "CordovaApp.Library.UploadTask";
builder.setTrigger(new Windows.ApplicationModel.Background.TimeTrigger(15, false));
builder.addCondition(new Windows.ApplicationModel.Background.SystemCondition(Windows.ApplicationModel.Background.SystemConditionType.internetAvailable));
return builder.register();
}).done(function () {
successCallback();
}, function (err) {
errorCallback(err);
});
}
Why is this not working? If i ran the task manually using visual studio it fails over on this line and it fails over on this line if i let the backgroundTask run itself after 15 minutes?
Problem solved.
I added the async keyword to my backgroundTask Run method so i could call my async method.
Instead of this, i removed the async keywork from my background Run method, and called the async code using
var result = Task.Run(async() => {
return myAsyncMethod();
}).Result;
Now all working