I am trying to get HTML content from the amazon website. Here is my code to create request, response, and get string:
public static HttpWebResponse GetHttpWebResponse(string url)
{
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url);
webRequest.ContentType = "text/xml";
try
{
return (HttpWebResponse)webRequest.GetResponse();
}
catch (WebException e)
{
if (e.Response == null)
throw new Exception("Cannot get response");
return (HttpWebResponse)e.Response;
}
}
public static string GetString(HttpWebResponse response)
{
Encoding encoding = Encoding.UTF8;
using (var reader = new StreamReader(response.GetResponseStream(), encoding))
{
string responseText = reader.ReadToEnd();
return responseText;
}
}
It is working fine with other web sites. However, when I try to get content from amazon, for example:
https://www.amazon.com/gp/product/B00AEISSHA/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1
I am seeing encoded content:
I tried to change Encoding and used HttpUtility.HtmlDecode(html); but it couldn't help. Is there any simple way to get content from Amazon?
You're not catering for compression. If you update your webrequest like this, it should do the trick.
public static HttpWebResponse GetHttpWebResponse(string url)
{
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url);
webRequest.ContentType = "text/xml";
webRequest.AutomaticDecompression = DecompressionMethods.GZip;
try
{
return (HttpWebResponse)webRequest.GetResponse();
}
catch (WebException e)
{
if (e.Response == null)
throw new Exception("Cannot get response");
return (HttpWebResponse)e.Response;
}
}
http://ssw.com/profile/?apikey = skdwkdkfkkdj
I tried to use
public async Task<string> GetFromUriAsync(string requestUri, string token)
{
var client = new HttpClient();
client.BaseAddress = new Uri(BaseUri);
client.DefaultRequestHeaders.Authorization =
new AuthenticationHeaderValue("apikey", "=" + token);
HttpResponseMessage response = await client.GetAsync(requestUri);
response.EnsureSuccessStatusCode();
string responseBody = await response.Content.ReadAsStringAsync();
return responseBody;
}
Then it returns null
Am I missing something or is it just totally wrong?
Thanks
You're trying to pass the API key in the header information of your HTTP request. What you need to do is just pass that whole URL without any additional header information.
IE: use "http://ssw.com/profile?apikey=abcdef" as the requestUri and send token as null. Also, remove the setting of the client.DefaultRequestHeaders.Authorization property. Authorization was meant to be a user/pass system and not a token-based system.
To test this, download Fiddler 4 (https://www.telerik.com/download/fiddler). Once you have fiddler installed, on the "Composer" tab, you can test different queries you need by putting the URL directly into the URL box and clicking "Execute". You'll then be able to use the inspectors to see the responses and figure out where you need to go from there.
Here are the classes I use for HTTP GET and POST operations:
public static string HTTPGET(string url)
{
try
{
HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url);
request.Timeout = 100000;
HttpWebResponse response = (HttpWebResponse) request.GetResponse();
Stream responseStream = response.GetResponseStream();
if (responseStream != null)
using (StreamReader resStream = new StreamReader(responseStream))
return resStream.ReadToEnd();
return null;
}
catch (Exception e)
{
Console.WriteLine(url);
Console.WriteLine(e);
return null;
}
}
public static string HTTPPOST(string url, string postData)
{
try
{
HttpWebRequest webRequest = (HttpWebRequest) WebRequest.Create(url);
webRequest.Method = "POST";
webRequest.ContentType = "x-www-form-urlencoded";
byte[] byteArray = Encoding.UTF8.GetBytes(postData);
using (Stream requestStream = webRequest.GetRequestStream())
requestStream.Write(byteArray, 0, byteArray.Length);
using (Stream responseStream = webRequest.GetResponse().GetResponseStream())
if (responseStream != null)
using (StreamReader responseReader = new StreamReader(responseStream))
return responseReader.ReadToEnd();
return null;
}
catch (Exception e)
{
Console.WriteLine(url);
Console.WriteLine(postData);
Console.WriteLine(e);
return null;
}
}
A friend show me this sample code to implement HTTP POST in C# and did work out in a WINFORM App:
http://www.terminally-incoherent.com/blog/2008/05/05/send-a-https-post-request-with-c/
And implemented in a METRO APP:
// this is what we are sending
string post_data = "user=user#example.com&pass=example123";
// this is where we will send it
string uri = "http://app.proceso.com.mx/win8/login";
// create a request
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.Method = "POST";
// turn our request string into a byte stream
byte[] postBytes = Encoding.UTF8.GetBytes(post_data);
// this is important - make sure you specify type this way
request.ContentType = "application/x-www-form-urlencoded";
Stream requestStream = await request.GetRequestStreamAsync();
// now send it
requestStream.Write(postBytes, 0, postBytes.Length);
// grab te response and print it out to the console along with the status code
WebResponse response = await request.GetResponseAsync();
//var a = new StreamReader(response.GetResponseStream()).ReadToEnd();
StreamReader requestReader = new StreamReader(response.GetResponseStream());
String webResponse = requestReader.ReadToEnd();
I realized, HttpWebRequest does not contain ProtocolVersion and is throwing me this error in this line:
WebResponse response = await request.GetResponseAsync();
// ERROR: The remote server returned an error: (417) Expectation Failed.
I guess the last property is the solution. How can I solve this problem?
Thanks in advance
I recently wrote a small function to handle the posting of trivial data to a server.
private struct HttpPostParam
{
private string _key;
private string _value;
public string Key { get { return HttpUtility.UrlEncode(this._key); } set { this._key = value; } }
public string Value { get { return HttpUtility.UrlEncode(this._value); } set { this._value = value; } }
public HttpPostParam(string key, string value)
{
this._key = key;
this._value = value;
}
};
private static string PostTrivialData(Uri page, HttpPostParam[] parameters)
{
string pageResponse = string.Empty;
try
{
var request = (HttpWebRequest)WebRequest.Create(page); //create the initial request.
request.Method = WebRequestMethods.Http.Post; //set the method
request.AllowAutoRedirect = true; //couple of settings I personally prefer.
request.KeepAlive = true;
request.ContentType = "application/x-www-form-urlencoded";
//create the post data.
byte[] bData = Encoding.UTF8.GetBytes(string.Join("&", Array.ConvertAll(parameters, kvp => string.Format("{0}={1}", kvp.Key, kvp.Value))));
using (var reqStream = request.GetRequestStream())
reqStream.Write(bData, 0, bData.Length); //write the data to the request.
using (var response = (HttpWebResponse)request.GetResponse()) //attempt to get the response.
if (response.StatusCode == HttpStatusCode.OK || response.StatusCode == HttpStatusCode.NotModified) //check for a valid status (should only return 200 if successful)
using (var reader = new System.IO.StreamReader(response.GetResponseStream()))
pageResponse = reader.ReadToEnd();
}
catch (Exception e)
{
/* todo: any error handling, for my use case failing gracefully was all that was needed. */
}
return pageResponse;
}
Essentially it posts the value pairs defined in the "parameters" argument. Will require a reference and import of the System.Web namespace to compile.
I just tested it with your website and got a response back:
HttpPostParam[] httpparams = {
new HttpPostParam("user", "censored#email.com"),
new HttpPostParam("pass", "example123")
};
string response = PostTrivialData(new Uri("http://app.proceso.com.mx/win8/login"), httpparams);
Let me know if there's any issues.
Suppose I have the following web method using C# and .NET:
[WebInvoke(UriTemplage="/users", Method="POST")]
[OperationContract]
public User AddNewUser(User u);
It is expected that when you implement POST web method you will accept a request body as part of the incoming HTTP request message. The parameter u is expected to be deserialized from the incoming HTTP message body.
My question is: how do we set this request body on the client side ? It's got to be set somewhere. It really confuses me.
Besides if I added "ResponseFormat = WebMessageFormat.Json" to WebInvoke, how can I deserialize from the returned json string into the User object ?
Thanks.
Your question doesn't reveal what you have tried. If you are using .net on the client then you can use the DataContractSerializer to get the serialized data.
You can then use an HttpWebRequest with the method set to POST.
Add the serialized data to the web request and use the GetResponse() method.
Alternatively you could use Fiddlr to test your web service using requests that you create. It gives you a concise view of exactly what is going up to the server.
To perform a POST request to your WCF Rest method:
private string UseHttpWebApproach<T>(string serviceUrl, string resourceUrl, string method, T requestBody)
{
string responseMessage = null;
var request = WebRequest.Create(string.Concat(serviceUrl, resourceUrl)) as HttpWebRequest;
if (request != null)
{
request.ContentType = "application/json";
request.Method = method;
}
if(method == "POST" && requestBody != null)
{
byte[] requestBodyBytes = ToByteArrayUsingJsonContractSer(requestBody);
request.ContentLength = requestBodyBytes.Length;
using (Stream postStream = request.GetRequestStream())
postStream.Write(requestBodyBytes, 0, requestBodyBytes.Length);
}
if (request != null)
{
var response = request.GetResponse() as HttpWebResponse;
if(response.StatusCode == HttpStatusCode.OK)
{
Stream responseStream = response.GetResponseStream();
if (responseStream != null)
{
var reader = new StreamReader(responseStream);
responseMessage = reader.ReadToEnd();
}
}
else
{
responseMessage = response.StatusDescription;
}
}
return responseMessage;
}
private static byte[] ToByteArrayUsingJsonContractSer<T> (T requestBody)
{
byte[] bytes = null;
var serializer1 = new DataContractJsonSerializer(typeof(T));
var ms1 = new MemoryStream();
serializer1.WriteObject(ms1, requestBody);
ms1.Position = 0;
var reader = new StreamReader(ms1);
bytes = ms1.ToArray();
return bytes;
}
Now Assuming your User object as shown below:
Public Class User
{
Public int UserId {get;set;}
Public string UserName {get;set;}
Public string Password {get;set;}
}
The to call the above method i do:
User objUser = new objUser();
objUser.Username = "Test";
objUser.Password = "Test";
UseHttpWebApproach<User>(serviceBaseUrl, "users", "POST", objUser);
I need to post data to a URL (https://somesite.com) to download file in responseStrem based on the parameters I posted.
How can I do that using a C# console application?
Parameters:
filename,
userid,
password,
type
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
using System.IO;
namespace WebserverInteractionClassLibrary
{
public class RequestManager
{
public string LastResponse { protected set; get; }
CookieContainer cookies = new CookieContainer();
internal string GetCookieValue(Uri SiteUri,string name)
{
Cookie cookie = cookies.GetCookies(SiteUri)[name];
return (cookie == null) ? null : cookie.Value;
}
public string GetResponseContent(HttpWebResponse response)
{
if (response == null)
{
throw new ArgumentNullException("response");
}
Stream dataStream = null;
StreamReader reader = null;
string responseFromServer = null;
try
{
// Get the stream containing content returned by the server.
dataStream = response.GetResponseStream();
// Open the stream using a StreamReader for easy access.
reader = new StreamReader(dataStream);
// Read the content.
responseFromServer = reader.ReadToEnd();
// Cleanup the streams and the response.
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
finally
{
if (reader != null)
{
reader.Close();
}
if (dataStream != null)
{
dataStream.Close();
}
response.Close();
}
LastResponse = responseFromServer;
return responseFromServer;
}
public HttpWebResponse SendPOSTRequest(string uri, string content, string login, string password, bool allowAutoRedirect)
{
HttpWebRequest request = GeneratePOSTRequest(uri, content, login, password, allowAutoRedirect);
return GetResponse(request);
}
public HttpWebResponse SendGETRequest(string uri, string login, string password, bool allowAutoRedirect)
{
HttpWebRequest request = GenerateGETRequest(uri, login, password, allowAutoRedirect);
return GetResponse(request);
}
public HttpWebResponse SendRequest(string uri, string content, string method, string login, string password, bool allowAutoRedirect)
{
HttpWebRequest request = GenerateRequest(uri, content, method, login, password, allowAutoRedirect);
return GetResponse(request);
}
public HttpWebRequest GenerateGETRequest(string uri, string login, string password, bool allowAutoRedirect)
{
return GenerateRequest(uri, null, "GET", null, null, allowAutoRedirect);
}
public HttpWebRequest GeneratePOSTRequest(string uri, string content, string login, string password, bool allowAutoRedirect)
{
return GenerateRequest(uri, content, "POST", null, null, allowAutoRedirect);
}
internal HttpWebRequest GenerateRequest(string uri, string content, string method, string login, string password, bool allowAutoRedirect)
{
if (uri == null)
{
throw new ArgumentNullException("uri");
}
// Create a request using a URL that can receive a post.
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(uri);
// Set the Method property of the request to POST.
request.Method = method;
// Set cookie container to maintain cookies
request.CookieContainer = cookies;
request.AllowAutoRedirect = allowAutoRedirect;
// If login is empty use defaul credentials
if (string.IsNullOrEmpty(login))
{
request.Credentials = CredentialCache.DefaultNetworkCredentials;
}
else
{
request.Credentials = new NetworkCredential(login, password);
}
if (method == "POST")
{
// Convert POST data to a byte array.
byte[] byteArray = Encoding.UTF8.GetBytes(content);
// Set the ContentType property of the WebRequest.
request.ContentType = "application/x-www-form-urlencoded";
// Set the ContentLength property of the WebRequest.
request.ContentLength = byteArray.Length;
// Get the request stream.
Stream dataStream = request.GetRequestStream();
// Write the data to the request stream.
dataStream.Write(byteArray, 0, byteArray.Length);
// Close the Stream object.
dataStream.Close();
}
return request;
}
internal HttpWebResponse GetResponse(HttpWebRequest request)
{
if (request == null)
{
throw new ArgumentNullException("request");
}
HttpWebResponse response = null;
try
{
response = (HttpWebResponse)request.GetResponse();
cookies.Add(response.Cookies);
// Print the properties of each cookie.
Console.WriteLine("\nCookies: ");
foreach (Cookie cook in cookies.GetCookies(request.RequestUri))
{
Console.WriteLine("Domain: {0}, String: {1}", cook.Domain, cook.ToString());
}
}
catch (WebException ex)
{
Console.WriteLine("Web exception occurred. Status code: {0}", ex.Status);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
return response;
}
}
}
Take a look at the System.Net.WebClient class, it can be used to issue requests and handle their responses, as well as to download files:
http://www.hanselman.com/blog/HTTPPOSTsAndHTTPGETsWithWebClientAndCAndFakingAPostBack.aspx
http://msdn.microsoft.com/en-us/library/system.net.webclient(VS.90).aspx
For this you can simply use the "HttpWebRequest" and "HttpWebResponse" classes in .net.
Below is a sample console app I wrote to demonstrate how easy this is.
using System;
using System.Collections.Generic;
using System.Text;
using System.Net;
using System.IO;
namespace Test
{
class Program
{
static void Main(string[] args)
{
string url = "www.somewhere.com";
string fileName = #"C:\output.file";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Timeout = 5000;
try
{
using (WebResponse response = (HttpWebResponse)request.GetResponse())
{
using (FileStream stream = new FileStream(fileName, FileMode.Create, FileAccess.Write))
{
byte[] bytes = ReadFully(response.GetResponseStream());
stream.Write(bytes, 0, bytes.Length);
}
}
}
catch (WebException)
{
Console.WriteLine("Error Occured");
}
}
public static byte[] ReadFully(Stream input)
{
byte[] buffer = new byte[16 * 1024];
using (MemoryStream ms = new MemoryStream())
{
int read;
while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
{
ms.Write(buffer, 0, read);
}
return ms.ToArray();
}
}
}
}
Enjoy!
HttpWebRequest request =(HttpWebRequest)WebRequest.Create("some url");
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.UserAgent = "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 7.1; Trident/5.0)";
request.Accept = "/";
request.UseDefaultCredentials = true;
request.Proxy.Credentials = System.Net.CredentialCache.DefaultCredentials;
doc.Save(request.GetRequestStream());
HttpWebResponse resp = request.GetResponse() as HttpWebResponse;
Hope it helps
Insted of using System.Net.WebClient I would recommend to have a look on System.Net.Http.HttpClient which was introduced with net 4.5 and makes your life much easier.
Also microsoft recommends to use the HttpClient on this article
http://msdn.microsoft.com/en-us/library/system.net.webclient(VS.90).aspx
An example could look like this:
var client = new HttpClient();
var content = new MultipartFormDataContent
{
{ new StringContent("myUserId"), "userid"},
{ new StringContent("myFileName"), "filename"},
{ new StringContent("myPassword"), "password"},
{ new StringContent("myType"), "type"}
};
var responseMessage = await client.PostAsync("some url", content);
var stream = await responseMessage.Content.ReadAsStreamAsync();