ScrapySharp does not POST form - c#

I'm using ScrapySharp to post a form to search cases on the Maryland Case Search web app.
Using Fiddler, I found the form name and form fields; however, every time I post, it always come back the initial search page, not the results.
Not sure what I'm missing, see code below.
Any assistance is truly appreciated.
string url = #"http://casesearch.courts.state.md.us/casesearch/processDisclaimer.jis?disclaimer=Y";
ScrapingBrowser Browser = new ScrapingBrowser();
Browser.AllowAutoRedirect = true;
Browser.AllowMetaRedirect = true;
WebPage PageResult = Browser.NavigateToPage(new Uri(url));
PageWebForm form = PageResult.FindForm("inquiryForm");
form["firstName"] = "";
form["middleName"] = "";
form["partyType"] = "";
form["filingStart"] = "";
form["filingEnd"] = "";
form["action"] = "Search";
form["company"] = "N";
form["countyName"] = "MONTGOMERY COUNTY";
form["courtSystem"] = "B";
form["filingDate"] = "4/4/2016";
form["lastName"] = "A";
form["site"] = "CIVIL";
form.Method = HttpVerb.Post;
WebPage results = form.Submit();
Console.WriteLine(results.ToString());

You need to make async calls.
Ex:
WebPage mainPage = await browser.NavigateToPageAsync(new Uri(url), HttpVerb.Get,"", "text/html; charset=UTF-8");
PageWebForm form = mainPage.FindFormById("some-form_id");
...
WebPage web = await Task.Run(() => form.Submit()); // submit is not async, so let's force it

Try this:
form.FormFields.Where(f => f.Name == "countyName").FirstOrDefault().Value = "MONTGOMERY COUNTY";

Related

.NET Core 2.0 Is the below code thread safe

Here's my code and I have doubt on thread safe implementation. My questions are below
The return value from GetHtmlPageAsync is object. Is it thread safe? I will use this object and add into the collection and finally upload into database.
The main method logic is below (implementation in-progress). I have set of domains, I have list of 10000 domains in the collection, the idea is, I will put it in the queue and call the GetHtmlPageAsync to get the HTML of the page. Based on the HTML, I will get the necessary hyperlinks. Once I get the hyper links, I will check certain word is available in the link. If the word is available in the link, I will call the same method GetHTMLPageAsync to get the HTML of that page. So the same thread may call the GetHtmlPageAsync to process another link. I am trying to reuse the same method for multiple calls in thread safe way. Please help.
#edit1 . I have added the main method. Instead of Queue. I have used ForEach
public static async Task<int> ProcessDomainAsync(List<string> domains)
{
Parallel.ForEach(domains, async (currentDomain) =>
{
var domainBody = await GetHtmlPageAsync(currentDomain);
var language = string.Empty;
var country = string.Empty;
var createdOn = DateTime.SpecifyKind(DateTime.Now, DateTimeKind.Local);
var updatedOn = DateTime.SpecifyKind(DateTime.Now, DateTimeKind.Local);
var machine = Environment.MachineName;
var message = "[" + domainBody.ErrorCode + "] - " + domainBody.ErrorMessage;
var active = false;
var stage = "End";
var url = currentDomain;
if (domainBody.ErrorCode == 0)
{
var html = domainBody.Body;
language = Common.GetLanguageIdentification(html);
country = Common.GetCountryIdentification(currentDomain);
message = string.Empty;
active = true;
stage = "Stage1";
var hyperLinks = Common.GetAllAHrefTags(html);
//Process Hyper Links
}
_domainList.Add(new Domain
{
Url = url,
Language = language,
Country = country,
MachineName = machine,
Message = message,
Active = active,
Stage = stage,
CreatedOn = createdOn,
UpdatedOn = updatedOn
});
domainCount++;
});
return domainCount;
}
public class DomainBody
{
public string Body;
public string ErrorMessage;
public int ErrorCode;
}
public static class DomainProcessing {
static async Task<DomainBody> GetHtmlPageAsync(string url)
{
#region Initialize Proxy
var sessionId = new Random().Next().ToString();
var proxy = new WebProxy(Constant.ProxyUrl, Constant.ProxyPort);
var login = Constant.ProxyUserName + "-session-" + sessionId;
proxy.Credentials = new NetworkCredential(login,Constant.ProxyPassword);
#endregion
#region Initialize Variables
var user_agent = Common.GenerateRandomUserAgent();
var body = string.Empty;
var errorCode = 0;
var errorMessage = string.Empty;
#endregion
try
{
#region Format URL with Http Protocol
var domainSB = new StringBuilder();
domainSB.Append("http://");
domainSB.Append(url);
#endregion
#region Process Domain
var request = (HttpWebRequest) WebRequest.Create(new Uri(url));
request.Proxy = proxy;
request.UserAgent = user_agent;
request.Timeout = Constant.TimeOut;
using (var response = await request.GetResponseAsync().ConfigureAwait(true))
using (var content = new MemoryStream())
using (var responseStream = response.GetResponseStream())
{
await responseStream.CopyToAsync(content);
var bodyArray = content.ToArray();
body = Encoding.UTF8.GetString(bodyArray, 0, bodyArray.Length);
}
errorCode = 0;
errorMessage = string.Empty;
#endregion
}
catch (HttpRequestException ex)
{
body = string.Empty;
errorCode = ex.InnerException.HResult;
errorMessage = ex.InnerException.Message;
}
catch (Exception ex)
{
body = string.Empty;
errorCode = ex.HResult;
errorMessage = ex.Message;
}
var domainBody = new DomainBody
{
Body = body,
ErrorCode = errorCode,
ErrorMessage = errorMessage
};
return domainBody;
}
}enter code here
Generally speaking, local variables should be thread safe (simply because they have no idea there even is another thread and other threads have no way to access them).
Anything that can be accessed by multiple threads should be looked at. _domainList for example. Make sure the Add method is thread-safe because you are calling it potentially in parallel.

DevExpress MVC Editors get value on Controller

I have a popupcontrol declared in my razor cshtml file as follow:
#Html.DevExpress().PopupControl(settings =>
{
settings.Name = "popSendBackReview";
settings.HeaderText = "Send Review Back to Scheduler";
settings.AllowResize = false;
settings.ShowHeader = true;
settings.ShowOnPageLoad = false;
settings.AllowDragging = true;
settings.CloseAction = CloseAction.CloseButton;
settings.CloseOnEscape = false;
settings.Modal = true;
settings.PopupElementID = "popSendBackReview";
settings.AutoUpdatePosition = true;
settings.PopupHorizontalAlign = PopupHorizontalAlign.WindowCenter;
settings.PopupVerticalAlign = PopupVerticalAlign.WindowCenter;
settings.Height = 280;
settings.Width = 450;
settings.SetContent(() =>
{
Html.RenderPartial("_SendBackReviewPanel");
});
}).GetHtml()
The partial view contains a memo box and button that calls an action:
#Html.DevExpress().Memo(settings =>
{
settings.Width = 300;
settings.Height = 150;
settings.Style.Add("margin-bottom", "10px");
settings.Name = "txtReviewComment";
settings.Properties.ValidationSettings.RequiredField.IsRequired = true;
settings.Properties.ValidationSettings.RequiredField.ErrorText = "A Review Comment is Required.";
settings.Properties.ValidationSettings.ErrorDisplayMode = ErrorDisplayMode.Text;
settings.Properties.ValidationSettings.ErrorTextPosition = ErrorTextPosition.Bottom;
settings.Properties.ValidationSettings.Display = Display.Dynamic;
settings.Properties.ValidationSettings.ValidationGroup = "Review";
}).GetHtml()
#Html.DevExpress().Button(settings =>
{
settings.Name = "btnSaveReview";
settings.Text = "Send Back for Scheduler Review";
settings.UseSubmitBehavior = false;
settings.ValidationGroup = "Review";
settings.RouteValues = new { Controller = "Matter", Action = "ResolveReview", Pass = false, Comment = Model.CommentText };
}).GetHtml()
#Html.DevExpress().Button(settings =>
{
settings.Name = "btnCancelReview";
settings.Text = "Cancel";
settings.UseSubmitBehavior = false;
settings.ClientSideEvents.Click = "function(s,e) { popSendBackReview.Hide(); }";
}).GetHtml()
I am trying to get the text that is typed into this box on the server side (in the action on my controller). In other places in my application I have been able to use the following code to get values of controls:
public ActionResult ResolveReview(bool Pass)
{ ...
EditorExtension.GetValue<string>("txtReviewComment")
...}
However this returns null in this scenario. What is the correct way to get this value from a control in a partial view rendered in a popupcontrol?
In general, it is necessary to wrap editors within a form container in order to pass the entire form's content on submit. Then, it is possible to retrive the required editor's value using the standard Model Binding mechanism. When using DevExpress MVC Editors, make sure that the DevExpressEditorsBinder is registered:
#using(Html.BeginForm("ResolveReview")) {
#Html.DevExpress().Memo(settings => {
settings.Name = "txtReviewComment";
}).GetHtml()
#Html.DevExpress().Button(settings => {
settings.Name = "btnSaveReview";
settings.UseSubmitBehavior = true;
}).GetHtml()
}
public ActionResult ResolveReview(bool Pass) {
EditorExtension.GetValue<string>("txtReviewComment")
}
or
public ActionResult ResolveReview(string txtReviewComment) { ... }
Check the MVC Data Editors - Model Binding and Editing learning resource.
I found the answer on my own, my button was causing a GET method to fire instead of POST. By setting "UseSubmitBehavior" to true on my save button, it started firing the POST function and allowing the
EditorExtension.GetValue<string>("txtReviewComment")
line to get a value.

Visual Studio WebTestRequest

I need some help with WebTestRequest on visual studio
I have 2 requests that the 2nd request need some data from the 1st request response
how do i use the specific data that i need from the 1st response to the 2nd request ? (in my case its Sessionid)
Test requests:
{
using System;
using System.Collections.Generic;
using System.Text;
using Microsoft.VisualStudio.TestTools.WebTesting;
public class WebTest1Coded : WebTest
{
public WebTest1Coded()
{
this.PreAuthenticate = true;
this.Proxy = "default";
}
public override IEnumerator<WebTestRequest> GetRequestEnumerator()
{
WebTestRequest request1 = new WebTestRequest("https://xxx.xxx.com/api/xxx");
request1.Method = "POST";
request1.QueryStringParameters.Add("format", "json", false, false);
StringHttpBody request1Body = new StringHttpBody();
request1Body.ContentType = "application/json";
request1Body.InsertByteOrderMark = false;
request1Body.BodyString = "{\"UserName\":\"xxx\",\"Password\":\"xxx\"}";
request1.Body = request1Body;
yield return request1;
request1 = null;
WebTestRequest request2 = new WebTestRequest("https://xxx.xxx.com/api/xxx");
request2.Method = "POST";
request2.QueryStringParameters.Add("format", "json", false, false);
StringHttpBody request2Body = new StringHttpBody();
request2Body.ContentType = "application/json";
request2Body.InsertByteOrderMark = false;
request2Body.BodyString = #"
RequestHeader"":
""SessionId"""": ""xxx"",
""ApiKey"""": ""xxx""
request2.Body = request2Body;
yield return request2;
request2 = null;
}
}
It is easier to do in the original Web Test file than in a Coded web Test. Just add an extraction rule to the first request. It saves the value into a context parameter which can then be used in the second request.
If you really want to use a Coded Web Test then you might create a sandbox web test with a suitable extraction rule and context parameter usage. Then convert that to Coded and copy the interesting bits into your test.
An ExtractText rule to extract text between abc and def and store the results in the acbdef context parameter generated the code below.
WebTestRequest request3 = new WebTestRequest("http://localhost/");
request3.ThinkTime = 1;
request3.ParseDependentRequests = false;
ExtractText extractionRule1 = new ExtractText();
extractionRule1.StartsWith = "abc";
extractionRule1.EndsWith = "def";
extractionRule1.IgnoreCase = false;
extractionRule1.UseRegularExpression = false;
extractionRule1.Required = true;
extractionRule1.ExtractRandomMatch = false;
extractionRule1.Index = 0;
extractionRule1.HtmlDecode = true;
extractionRule1.SearchInHeaders = false;
extractionRule1.ContextParameterName = "abcdef";
request3.ExtractValues += new EventHandler<ExtractionEventArgs>(extractionRule1.Extract);
yield return request3;
request3 = null;
Other extraction rules have different properties that will need to be given values. I recommend putting the main details of what you want into the web test and then doing a generate code to find out exactly what C# code is necessary.
The extracted value is used in three places in the next request:
WebTestRequest request4 = new WebTestRequest(
("http://localhost/" + (this.Context["abcdef"].ToString() + "/more")));
request4.ThinkTime = 1;
request4.ParseDependentRequests = false;
request4.Headers.Add(new WebTestRequestHeader("Use-abcdef-again",
("pqr" + (this.Context["abcdef"].ToString() + "stu"))));
request4.Headers.Add(new WebTestRequestHeader("Use-abcdef",
this.Context["abcdef"].ToString()));
yield return request4;
request4 = null;

Unable Post tweet From C# Aaplication

i'm using oauth and twitterizer in my C# apps, and after trying for several times. and it won't get any exception, error. but it's doesn't want to post it to twitter.
here is my code
protected void Page_Load(object sender, EventArgs e)
{
BLTgUser objBLTgUser = new BLTgUser();
if (!objBLTgUser.IsLogin) objBLTgUser.GoToLoginPage(HttpContext.Current.Request.Url.AbsoluteUri);
m_strIdGods = objBLTgUser.IdGods;
BLGodsProfile objGods = new BLGodsProfile();
tbmgodsprofile objGodsProfile = objGods.GetGodsByIdGods(m_strIdGods);
string reqToken = Request.QueryString["oauth_token"].ToString();
string pin = Request.QueryString["oauth_verifier"].ToString();
var oauth_consumerKey = System.Configuration.ConfigurationManager.AppSettings["TwitterConsumerKey"];
var oauth_consumerSecret = System.Configuration.ConfigurationManager.AppSettings["TwitterConsumerSecret"];
var tokens = OAuthUtility.GetAccessToken(
oauth_consumerKey,
oauth_consumerSecret,
reqToken,
pin);
string accessToken = tokens.Token;
string accessTokenSecret = tokens.TokenSecret;
objGodsProfile.twittertoken = accessToken;
objGodsProfile.twitterpin = accessTokenSecret;
objGodsProfile.twitterstatus = "1";
objGods.UpdateGodsProfile(objGodsProfile);
}
i'm trying store the token and tokensecret to Database, it's for the next uses.
string v_str = "";
BLEnumHelper m_BLEnumHelper = new BLEnumHelper();
BLGodsProfile userprofile = new BLGodsProfile();
tbmgodsprofile godsAccessToken = userprofile.GetGodsByIdGods(m_strIdGods);
string reqToken = godsAccessToken.twittertoken;
string reqTokenAccess = godsAccessToken.twitterpin;
var oauth_consumerKey = System.Configuration.ConfigurationManager.AppSettings["TwitterConsumerKey"];
var oauth_consumerSecret = System.Configuration.ConfigurationManager.AppSettings["TwitterConsumerSecret"];
OAuthTokens accesstoken = new OAuthTokens()
{
AccessToken = reqToken,
AccessTokenSecret = reqTokenAccess ,
ConsumerKey = oauth_consumerKey,
ConsumerSecret = oauth_consumerSecret
};
TwitterResponse<TwitterStatus> response = TwitterStatus.Update(accesstoken,p_strMessage);
if (response.Result == RequestResult.Success)
{
Response.Write("we did it!");
}
else
{
Response.Write("it's all bad.");
}
and after the 2nd code be called, it wont' post to twitter.
Twitterizer is no longer maintained.
https://github.com/Twitterizer/Twitterizer/blob/develop/README.mediawiki
Twitterizer not supporting Twitter's new API version 1.1, you can't use it for development.
As an alternative, I recommend CoreTweet for you.

Using a WebAPI method to return an etagged javascript snippet

For the purposes of user tracking, I have an etagged javascript snippet which i return from a .aspx page.
The relevant code is below:
string eTag = this.Request.Headers["If-None-Match"];
if (eTag != null)
{
Response.StatusCode = 304;
Response.SuppressContent = true;
}
else
{
string data = "asdfasdfs";
string script = "(function(){DoSomething('" + data + "');})();";
Response.ContentType = "application/javascript";
Response.Cache.SetLastModified(DateTime.Now);
Response.Cache.SetCacheability(HttpCacheability.Public);
Response.Cache.SetETag(data);
Response.Write(script);
}
I have converted this project to an MVC WebAPI2 (VS 2013) project, but I'm not sure exactly how to achieve the same functionality.
If this was a full MVC project I could create a custom ActionResult like in this example: http://blogs.microsoft.co.il/alon_nativ/2011/07/10/aspnet-mvc-imagepixel-actionresult/ . But, WebAPI methods do not normally return an ActionResult.
I believe I could use HttpResponseMessage - http://msdn.microsoft.com/en-us/library/system.net.http.httpresponsemessage(v=vs.110).aspx, but this class appears to be missing several things from the code above.
What is the cleanest/best practice way to do this in WebAPI?
I was able to individually track down each item, appears to be working.
Hope this helps someone in the future.
HttpResponseMessage response = null;
var eTag = Request.Headers.IfNoneMatch;
if (eTag != null && eTag.Count > 0)
{
response = Request.CreateResponse(HttpStatusCode.NotModified);
}
else
{
string data = "asdfasdfs";
string script = "(function(){DoSomething('" + data + "');})();";
response = Request.CreateResponse(HttpStatusCode.OK);
response.Content = new StringContent(script, Encoding.UTF8, "application/javascript");
response.Content.Headers.LastModified = DateTime.Now;
response.Headers.CacheControl = new CacheControlHeaderValue() { Public = true };
response.Headers.ETag = new EntityTagHeaderValue("\"" + data + "\"");
}

Categories

Resources