Can anyone suggest a way to search a string on the web page, using ASP .net C#
Scenario: A textbox have some value(string), and on the click of button it searches that value(string) on some xyz page.
Example: I have "youtube"(string) in the textbox, and when I press submit button. That submit button gets the entire information of say Google.com(rendered page), and searches for "youtube"(string) on that Google.com
Possible solution:
I think that can be achieved by some-how rendering the page temporarily some-where, or some-how storing the response in any string or array, and then look that string.
Can anyone suggest a way to solve the above scenario. If possible with an example.
use a web request to get the page:
WebRequest request = WebRequest.Create(http://www.google.com);
request.Method = "GET";
WebResponse response = request.GetResponse();
Stream stream = response.GetResponseStream();
StreamReader reader = new StreamReader(stream);
string content = reader.ReadToEnd();
reader.Close();
response.Close();
then just search on your content string:
int i = content.IndexOf("youtube");
or however you want to search for it.
You could do this with async/await if you are using .NET 4.5
static void Main(string[] args)
{
var content = GetUrlContents("http://www.google.com");
if (content.Result.Contains("Google"))
Console.WriteLine("Google found!");
Console.Read();
}
static async Task<string> GetUrlContents(string url)
{
HttpClient client = new HttpClient();
var content = await client.GetStringAsync(url);
return content;
}
Or if you want synchronous, you could do this kind of method
public static string GetUrlContents(string url)
{
return new WebClient().DownloadString(address);
}
Related
Sorry if my question was not in proper manner do edit if required.
USE CASE
I want a function which will find the given string or text from a web page as soon as it updated in wepage in c#.
Take example as https://www.worldometers.info/world-population/ i am trying to get "40" data.So,when the current page will load "40" data or content my function should stop.I think its not loading the AJAX calls.
public static void WaitForWebPageContent(string url,string text)
{
while (true)
{
string pageContent = null;
HttpWebRequest myReq = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse myres = (HttpWebResponse)myReq.GetResponse();
using (StreamReader sr = new StreamReader(myres.GetResponseStream()))
{
pageContent = sr.ReadToEnd();
}
if(pageContent.Contains(text))
{
Debug.WriteLine("Found it");
break;
}
}
}
My Question
I am searching for a method where i can get the page content by calling it from my function.Currently i am using httpclient to get the response from a URL but the data its not stopping even the content is loaded in browser.I am continously sending the request in each and every second by using the above code.Please guide me to proper solution
Thanks
In my Windows phone app, I want to take a simple string url, and when that url is entered in a browser, it only shows a JSON string on the webpage, as a response.
So I want to enter that url in my app and just get that JSON string in return, How can I do it? I've tried following but getResponse function isn't present in Silverlight.
string strUrl = "http://.....";
WebRequest request = HttpWebRequest.Create(strUrl);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream s = (Stream)response.GetResponseStream();
StreamReader readStream = new StreamReader(s);
string dataString = readStream.ReadToEnd();
response.Close();
s.Close();
readStream.Close();
I would use the HttpClient instead it is much easier to use. You need to add the HttpClient Nuget package to use it in the WP Silverlight project.
private async void MainPage_Loaded(object sender, RoutedEventArgs e)
{
using (HttpClient client = new HttpClient())
{
string data = await client.GetStringAsync("http://msdn.microsoft.com");
}
}
I have two servers. One is a private server and I don't want users to have direct access to it, and the other one is the server that public does have access to.
I can access my private server by URL like: http://xxx.xx.xxx.xxx/
What i want to do is create some kind of "proxy", only to work with my private server. My idea is to go to: http://www.domain.com/server/path/here/something
This page should show me the content of http://xxx.xx.xxx.xxx/path/here/something
I have this working, but the only way I could make it work was to return the content as a string, and then the browser would interpret the HTML.
This works fine for pages that return HTML content, but it doesn't work (of course) if I want to access a .gif or any kind of file directly.
Here's the code I currently have:
public string Index(string url)
{
string uri = "http://xxx.xx.xxx.xxx/" + url;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.Method = "GET";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader responseStream = new StreamReader(response.GetResponseStream());
string resultado = responseStream.ReadToEnd();
return resultado;
}
How can I change my code so that it works for any file ?
You can check the response content type and do what you need based on that.
You'll need to change your action to return ActionResult instead of string.
if(response.ContentType.Equals("text/html"))
{
//show html stuff
return Content(resultado);
}
else if(response.ContentType.Contains("image/"))
{
var ms = new MemoryStream();
responseStream.BaseStream.CopyTo(ms);
var imageBytes = ms.ToArray();
return File(imageBytes, response.ContentType);
}
you have to write a system which reads your html or images from resultado and do something according to that PLUS you need to control your Url as well.
I am working on a C# project where I need to get data from a secured web site that does not have an API or web services. My plan is to login, get to the page I need, and parse out the HTML to get to the data bits I need to log to a database. Right now I'm testing with a console app, but eventually this will be converted to an Azure Service bus application.
In order to get to anything, you have to login at their login.cfm page, which means I need to load the username and password input controls on the page and click the submit button. Then navigate to the page I need to parse.
Since I don't have a 'browser' to parse for controls, I am trying to use various C# .NET classes to get to the page, set the username and password, and click submit, but nothing seems to work.
Any examples I can look at, or .NET classes I should be reviewing that were designed for this sort of project?
Thanks!
Use the WebClient class in System.Net
For persistence of session cookie you'll have to make a custom WebClient class.
#region webclient with cookies
public class WebClientX : WebClient
{
public CookieContainer cookies = new CookieContainer();
protected override WebRequest GetWebRequest(Uri location)
{
WebRequest req = base.GetWebRequest(location);
if (req is HttpWebRequest)
(req as HttpWebRequest).CookieContainer = cookies;
return req;
}
protected override WebResponse GetWebResponse(WebRequest request)
{
WebResponse res = base.GetWebResponse(request);
if (res is HttpWebResponse)
cookies.Add((res as HttpWebResponse).Cookies);
return res;
}
}
#endregion
Use a browser add-on like FireBug or the development tools built into Chrome to get the HTTP POST data being sent when you submit a form. Send those POSTs using the WebClientX class and parse the response HTML.
The fastest way to parse HTML when you already know the format is using a simple Regex.Match. So you'd go through the actions in your browser using the development tools to record your POSTs, URLs and HTML content then you'll perform the same tasks using the WebClientX.
Ok, so here is the complete Code to login to one page, then read from a 2nd page after the login.
class Program
{
static void Main(string[] args)
{
string uriString = "http://www.remotesite.com/login.cfm";
// Create a new WebClient instance.
WebClientX myWebClient = new WebClientX();
// Create a new NameValueCollection instance to hold some custom parameters to be posted to the URL.
NameValueCollection myNameValueCollection = new NameValueCollection();
// Add necessary parameter/value pairs to the name/value container.
myNameValueCollection.Add("userid", "myname");
myNameValueCollection.Add("mypassword", "mypassword");
Console.WriteLine("\nUploading to {0} ...", uriString);
// 'The Upload(String,NameValueCollection)' implicitly method sets HTTP POST as the request method.
byte[] responseArray = myWebClient.UploadValues(uriString, myNameValueCollection);
// Decode and display the response.
Console.WriteLine("\nResponse received was :\n{0}", Encoding.ASCII.GetString(responseArray));
Console.WriteLine("\n\n\n pausing...");
Console.ReadKey();
// Go to 2nd page on the site to get additional data
Stream myStream = myWebClient.OpenRead("https://www.remotesite.com/status_results.cfm?t=8&prog=d");
Console.WriteLine("\nDisplaying Data :\n");
StreamReader sr = new StreamReader(myStream);
StringBuilder sb = new StringBuilder();
using (StreamReader reader = new StreamReader(myStream, System.Text.Encoding.UTF8))
{
string line;
while ((line = reader.ReadLine()) != null)
{
sb.Append(line + "\r\n");
}
}
using (StreamWriter outfile = new StreamWriter(#"Logfile1.txt"))
{
outfile.Write(sb.ToString());
}
Console.WriteLine(sb.ToString());
Console.WriteLine("\n\n\n pausing...");
Console.ReadKey();
}
}
public class WebClientX : WebClient
{
public CookieContainer cookies = new CookieContainer();
protected override WebRequest GetWebRequest(Uri location)
// public override WebRequest GetWebRequest(Uri location)
{
WebRequest req = base.GetWebRequest(location);
if (req is HttpWebRequest)
(req as HttpWebRequest).CookieContainer = cookies;
return req;
}
protected override WebResponse GetWebResponse(WebRequest request)
{
WebResponse res = base.GetWebResponse(request);
if (res is HttpWebResponse)
cookies.Add((res as HttpWebResponse).Cookies);
return res;
}
}
I almost dare to ask, but how can i get the response data of a URL?
I just can't remember anymore.
My scenario: I'm using the twitter API to get the profile picture of an user. That API URL returns the JPEG location.
So if I actually write this HTML in my views:
<img src="https://api.twitter.com/1/users/profile_image?screen_name=twitterapi&size=bigger"/>
The browser auto uses the response JPEG for the SRC property. Like this:
Now is my question very simple: how can I get that .jpg location in C# to put in my database?
I'm not exactly sure what you are asking.
I think you can use WebClient.DownloadData in c# to call that url. Once you download the file, you can then place it in the database.
byte[] response = new System.Net.WebClient().DownloadData(url);
Download a file over HTTP into a byte array in C#?
EDIT: THIS IS WORKING FOR ME
WebRequest request = WebRequest.Create("https://api.twitter.com/1/users/profile_image?screen_name=twitterapi&size=bigger");
WebResponse response = request.GetResponse();
Console.WriteLine(response.ResponseUri);
Console.Read( );
from A way to figure out redirection URL
EDIT: THIS IS ANOTHER METHOD I THINK...using show.json from Read the absolute redirected SRC attribute URL for an image
http://api.twitter.com/1/users/show.json?screen_name=twitterapi
You can also do it using HttpClient:
public class UriFetcher
{
public Uri Get(string apiUri)
{
using (var httpClient = new HttpClient())
{
var httpResponseMessage = httpClient.GetAsync(apiUri).Result;
return httpResponseMessage.RequestMessage.RequestUri;
}
}
}
[TestFixture]
public class UriFetcherTester
{
[Test]
public void Get()
{
var uriFetcher = new UriFetcher();
var fetchedUri = uriFetcher.Get("https://api.twitter.com/1/users/profile_image?screen_name=twitterapi&size=bigger");
Console.WriteLine(fetchedUri);
}
}
You can use the HttpWebRequest and HttpWebResponse classes (via using System.Net)to achieve this;
HttpWebRequest webRequest =
WebRequest.Create("https://api.twitter.com/1/users/profile_image?screen_name=twitterapi&size=bigger") as HttpWebRequest;
webRequest.Credentials = CredentialCache.DefaultCredentials;
HttpWebResponse response = webRequest.GetResponse() as HttpWebResponse;
string url = response.ResponseUri.OriginalString;
url now contains the string "https://si0.twimg.com/profile_images/1438634086/avatar_bigger.png"