The other day I had my scraper working very well, when I realized that I wasn't getting enough records, I decided to search by postal code and had to change some of my code. The first POST and RESPONSE goes off without a hitch, but when I post to get the next page of results it fails on the Response.
the starting code is...
public void StartScrape()
{
List<string> a = lstPostalCodes();
for (int i = 0; i <= lstPostalCodes().Count; i++)
{
b = a[i];
FirstRequestResponse(b);
if (GoBackToStartSearch == "CONTINUEON")
StartNextRequest(GetViewState(ResponseData));
else
{
WriteDataToFile(ResponseData);
FinalClean();
}
}
}
The method that I am calling is StartNextRequest. In the StartNextRequest method is
private void GetResults(HttpWebRequest wr)
{
using (StreamReader responseReader = new StreamReader(wr.GetResponse().GetResponseStream()))
{
// Add response/results to string
ResponseData = responseReader.ReadToEnd();
}
string strFind = "<li id='nextdisabled'>";
if (ResponseData.Contains(strFind)) GoBackToStartSearch = "BACKTOSTART";
else
GoBackToStartSearch = "CONTINUEON";
}
The error that its throwing is saying that the
Input string was not in correct format
and pointing at the using (StreamReader...)
I have gotten other errors that I seem to fix, but then it creates another error which has to do with waiting for bytes to be written. I'm using Fiddler2 to give me more info on the errors, the headers are pretty much the same, I'm not seeing much of a difference, other than 1 being a request and the other a response.
I have no idea on why its saying that, I have stepped through the working code that I wrote to scrape all the data results, and compared the two. Everything looks the same. The values are all the same.
Any ideas on where I should look to fix this?
Related
I was hoping to get some insight on the error that are produced by the system. I am using a already built message system that I got some time ago and it works but sometimes on the forms I will get errors that I do not understand. For instance on a Create I have a try / catch block that produces a message if it has successfully Executed. I have tried to search for these errors in my project and it does not come up with anything. Even if it was in meta data a search should find it.
I use System.Text.StringBuilder sb = new System.Text.StringBuilder(); for the message and the code looks like this:
public ActionResult Create(Vendors model)
{
System.Text.StringBuilder sb = new System.Text.StringBuilder();
try
{
if (ModelState.IsValid)
{
var userId = User.Identity.GetUserId();
//var getdata = ExtendedViewModels.VendorToEntity(model);
model.VendorId = Guid.NewGuid();
model.CreatedDate = System.DateTime.Now;
model.CreatedBy = User.Identity.Name;
model.Status = true;
db.Vendors.Add(model);
db.SaveChanges();
sb.Append("Submitted");
return Content(sb.ToString());
}
else
{
foreach (var key in this.ViewData.ModelState.Keys)
{
foreach (var err in this.ViewData.ModelState[key].Errors)
{
sb.Append(err.ErrorMessage + "<br/>");
}
}
}
}
catch (Exception ex)
{
sb.Append("Error :" + ex.Message);
}
return Content(sb.ToString());
}
When this returns or closes the Modal it produces a message or if there is an error it will produce that so you can fix it like a Required field. If everything is okay it will produce from this:
#Html.StarkAjaxFormSubmiter("frmVendors", "tbVendors", true, "Action Successfully Executed")
This is a green box that shows up as "Action Successfully Executed". If something is wrong a red box shows up and you get a message. In my case I am getting a red box that says Submitted Read Warnings Alerts This is how it is spelled. I doubt this is a error that comes from ASP.Net it looks more like a custom message, I dont know what it means and I cannot find it anywhere. Regardless, it does create the record in the db. The other error I have gotten shows Something is went wrong [object, object] Not only do I want to find out what these mean, I also want to clean them up and give a proper message that makes sense. Does anyone have any ideas as to how to correct this? Could they be encypted in the custom package that was written for this? That is why I cannot find them. I have also viewed the package and did not find anything for this.
This is from Meta data:
//
// Parameters:
// stark:
//
// FormId:
// Enter Here Form ID LIKE So you have to pass = frmCreate
//
// DataTableId:
// Which DataTable You have update after submit provide that ID
//
// IsCloseAfterSubmit:
// Do you want to opened popup close after submit , So pass=true or false any
//
// SuccessMessage:
// Give any Success message
public static MvcHtmlString StarkAjaxFormSubmiter(this HtmlHelper stark, string FormId, string DataTableId, bool IsCloseAfterSubmit, string SuccessMessage);
//
// Parameters:
// stark:
//
// FormId:
// Enter Here Form ID LIKE So you have to pass = frmCreate
//
// DataTableId:
// Which DataTable You have update after submit provide that ID
//
// IsCloseAfterSubmit:
// Do you want to opened popup close after submit , So pass=true or false any
//
// SuccessMessage:
// Give any Success message
//
// AfterSuccessCode:
// Add other JQuery code if you want
public static MvcHtmlString StarkAjaxFormSubmiter(this HtmlHelper stark, string FormId, string DataTableId, bool IsCloseAfterSubmit, string SuccessMessage, string AfterSuccessCode);
Thanks for our help
UPDATE:
I did some searching on the web and found a program called JetBrains dotPeek. I decompiled the dll and sure enough the messages are in there. So I should be able to change them and recompile it and add if I want, to it.
I was not able to edit the decompiled dll. So I decided to just create a class in the main project and copy the the code to that class. Changing what I needed. Where my trouble was, was with misspellings. The dll used Sumitted as the sb.Append("Sumitted") I changed that in the controller to be Submitted. So the dll did not find "Sumitted" in the action, and in the dll class there is an If statement that faults to error if not found - which was listed as Read Warnings Error. I changed that and fixed all the misspellings. I also got rid of the Something is went wrong and changed it to something more meaningful. I will continue to add to this to give more meaningful messages. It helps to know what the error is, instead of [object], [object]. I dont know if this will help others, maybe if they have downloaded the same code I have and have issues.
Before anyone points it out, yes, I know there have been similar questions asked before. I've already tried solutions from them and they do not work, which is why I'm asking this question.
For reference, I'm using Android API 30.
So I'm performing a query on all audio files on the device using MediaStore. As of now I'm able to properly access artist/album/song IDs, names, etc. and use them elsewhere in my app. The one thing I'm unable to get though is the album art.
This is what my code looks like:
string[] columns = {
MediaStore.Audio.Media.InterfaceConsts.IsMusic,
MediaStore.Audio.Media.InterfaceConsts.Data,
MediaStore.Audio.Media.InterfaceConsts.Title,
MediaStore.Audio.Media.InterfaceConsts.CdTrackNumber,
MediaStore.Audio.Media.InterfaceConsts.Duration,
MediaStore.Audio.Media.InterfaceConsts.Id,
MediaStore.Audio.Media.InterfaceConsts.Album,
MediaStore.Audio.Media.InterfaceConsts.AlbumId,
MediaStore.Audio.Media.InterfaceConsts.Artist,
MediaStore.Audio.Media.InterfaceConsts.ArtistId,
};
ICursor? cursor = context.ContentResolver?.Query(MediaStore.Audio.Media.ExternalContentUri!, columns, null, null, null);
if (cursor is null)
{
return;
}
while (cursor.MoveToNext())
{
if (cursor.GetString(0) is not "1")
{
continue;
}
string trackPath = cursor.GetString(1)!;
string trackTitle = cursor.GetString(2)!;
int trackIndex = cursor.GetInt(3);
uint trackDuration = (uint)(cursor.GetFloat(4) / 1000);
long trackId = cursor.GetLong(5);
string albumTitle = cursor.GetString(6)!;
long albumId = cursor.GetLong(7);
string artistName = cursor.GetString(8)!;
long artistId = cursor.GetLong(9);
I initially tried the obvious method, which was adding MediaStore.Audio.Media.InterfaceConsts.AlbumArt to columns and getting the data from that column. But just adding that causes the application to freeze. Logging shows that adding it causes the method to freeze midway and not do anything. So this solution is out.
I then tried MediaMetadataRetriever, like this:
MediaMetadataRetriever retriever = new();
retriever.SetDataSource(trackPath);
byte[] artwork = retriever.GetEmbeddedPicture()!;
Bitmap albumArt = BitmapFactory.DecodeByteArray(artwork, 0, artwork.Length)!;
However, this also fails. I get an error message in the logs saying that MediaMetadataRetriever was unable to set the data source to that source.
So I figured maybe my data source was wrong. After doing some digging around I tried using a different path:
Uri contentUri = Uri.Parse("content://media/external/audio/albumart")!;
Uri albumArtUri = ContentUris.WithAppendedId(contentUri, albumId);
MediaMetadataRetriever retriever = new();
retriever.SetDataSource(albumArtUri.Path);
...
This also does not work. Neither does using MediaStore.Audio.Albums.ExternalContentUri.
I even tried opening a ParcelFileDescriptor from those URIs. Also doesn't work.
Does anyone know of a way that would definitely work? Most of the answers on StackOverflow seem to be quite dated, so they possibly don't apply to API 30 anymore. But I don't know what I'm doing wrong since the documentation isn't very detailed.
I'm trying to get multiple data from different urls using HtmlAgilityPack.
It will get product prices.
But when product stock is 0. They are closing the page.
My program adding prices to listbox. When page giving 404 It should add empty listbox item.
Is there any way to make program simpler? I can't use same Variables at the same button. I'm adding same code changing the numbers (6).
WebRequest SiteyeBaglantiTalebi06 = HttpWebRequest.Create("https://www.themia.com.tr/The-Mia-Dekor-Mermer-22-Cm-Gri,PR-2432.html");
WebResponse GelenCevap06 = SiteyeBaglantiTalebi06.GetResponse();
StreamReader CevapOku06 = new StreamReader(GelenCevap06.GetResponseStream());
string KaynakKodlar06 = CevapOku06.ReadToEnd();
int IcerikBaslangicIndex06 = KaynakKodlar06.IndexOf("<div class=\"productPrice\">") + 122;
int IcerikBitisIndex06 = KaynakKodlar06.Substring(IcerikBaslangicIndex06).IndexOf("</div>");
listBox3.Items.Add((KaynakKodlar06.Substring(IcerikBaslangicIndex06, IcerikBitisIndex06)));
If you cast the WebResponse you got to an HttpWeResponse you can access the StatusCode property - https://learn.microsoft.com/en-us/dotnet/api/system.net.httpwebresponse.statuscode?view=netframework-4.7.2#System_Net_HttpWebResponse_StatusCode;
Just one thing to notice - you can't make HttpWebRequest NOT throw an exception when it recieves a status code that does not indicate success (all the more reason not to use this method). This means you have to be ready to catch the exception that will be thrown.
So in the case of your example, it would be -
WebRequest SiteyeBaglantiTalebi06 = HttpWebRequest.Create("https://www.somesite.com/NotARealPath");
try
{
WebResponse GelenCevap06 = SiteyeBaglantiTalebi06.GetResponse();
// do things with the result
}
catch (WebException ex)
{
using (WebResponse response = ex.Response)
{
HttpWebResponse asHttp = (HttpWebResponse)response;
if (asHttp.StatusCode == System.Net.HttpStatusCode.NotFound)
{
// your 404 logic here
}
else
{
// your "something went wrong but it's not a 404" logic
}
}
}
As for making the code simpler - it's hard to understand exactly what you mean by that without understanding more about your program and what you're trying to do. In general, here are a few ideas -
Stop using HttpWebRequest - here's a nice articale detailing why, and what are your other (and better) options - https://www.diogonunes.com/blog/webclient-vs-httpclient-vs-httpwebrequest/
Put code that is used again and again in a method instead of just copy/pasting it around - https://learn.microsoft.com/en-us/dotnet/csharp/programming-guide/classes-and-structs/methods
Use the Command pattern to bind your button clicks in a more clever way do you can pass parameters (or at least use the 'simple' method in this SO answer) - Add parameter to Button click event
So I'm building an app with twilio voice, and I've got all the phonecall stuff working. But I'm having a little trouble understanding which parameters my callback should have.
I've registered the URL as described in the docs:
options.From = formatPhoneNumber(callout.callback_number);
options.To = formatPhoneNumber(offer.employee_phone_number);
options.Url = TwilioCallBotController.TwilioCalloutScriptURL;
options.StatusCallback = TwilioCallBotController.StatusCallbackURL;
options.StatusCallbackEvents = new []{"initiated", "ringing", "answered", "completed" };
options.StatusCallbackMethod = "POST";
I've also made a callback method here, but I'm not having much luck finding out how the parameters are supposed to work with their API. I'm kindof at a loss as to what could be the reason behind this one not working:
[HttpPost]
public ActionResult TwilioStatusCallback()
{
var twiml = new Twilio.TwiML.TwilioResponse();
twiml.Say("This is a test");
string CallSid = Request.Form["CallSid"];
string CallStatus = Request.Form["CallStatus"];
Debug.WriteLine("Status Callback Delivered");
Shift_Offer shoffer = db.Shift_Offers.Where(s => s.twillio_sid == CallSid).ToList()[0];
shoffer.status = CallStatus.ToString();// + DateTime.Now.ToString();
return TwiML(twiml);
}
Edit:
So it turns out that the API is very sensitive about the method signature (the call was previously throwing a method not found exception in a number of microsoft DLLs, including System.Web and System.Web.Mvc.
So I've actually gotten the software to call the method by using an empty method signature (no parameters).
However I'm still having trouble getting the parameters from the HTTPPOST
Edit: So upon further investigation I've managed to inspect the Request. The values I'm after exist in Request.Form["foo"], but they don't seem to be getting put into the two strings I have declared. I've removed the ["HttpPost"] attribute to try to troubleshoot the issue, but I'm really at a loss as to why I can see the values in the debugger, but they're not translating into memory.
public ActionResult TwilioStatusCallback()
{
var twiml = new Twilio.TwiML.TwilioResponse();
string sid = Request.Form["CallSid"];
string status = Request.Form["CallStatus"];
Shift_Offer shoffer = db.Shift_Offers.Where(s => s.twillio_sid == sid).ToList()[0];
shoffer.status = status;// + DateTime.Now.ToString();
return TwiML(twiml);
}
Last issue was that the database wasn't being saved.
Just added a db.SaveChanges() and we're good.
I am trying to get all my status messages using the following code:
public dynamic downloadStatuses(FacebookOAuthResult facebookOAuthResult, string userInput)
{
dynamic result = null;
if (facebookOAuthResult != null)
{
if (facebookOAuthResult.IsSuccess)
{
this.accessToken = facebookOAuthResult.AccessToken;
var fb = new FacebookClient(facebookOAuthResult.AccessToken);
result = fb.Get(userInput + "/statuses?format=json&limit=1500");
return result;
}
else
{
MessageBox.Show(facebookOAuthResult.ErrorDescription);
return result;
}
}
return result;
}
userInput would be my Facebook ID. Some statuses are returned but definitely does not total up to the 1500 that I have indicated. Was wondering if anyone knew how to access your first ever facebook status message or retrieve all the status message by slightly modifying the request url in the code. Is it a must to use fql?
When you query /statuses you don't get the Photo/Album/video related statuses/entries from your stream (including their comments etc). This also includes event creation and questions.
You will have to query the API with separate calls (i.e. /me/photos) to get everything. If you want to have almost the full stream of some person or page or group. Consider using FQL, querying the stream table.
EDIT : Just saw this question is probably outdated, my bad