I'm using this code to make a request to an HTML page:
public static String MakeHttpRequest(String url)
{
try
{
HttpWebRequest httpWReq = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)httpWReq.GetResponse();
string responseString = new StreamReader(response.GetResponseStream()).ReadToEnd();
return responseString;
}
catch (Exception e)
{
return null;
}
}
And I noticed that sometimes I receive from it not the full HTML code and only half of it(I recognize it by that sometimes the responseString is not full and does not include all the HTML)
http://ssw.com/profile/?apikey = skdwkdkfkkdj
I tried to use
public async Task<string> GetFromUriAsync(string requestUri, string token)
{
var client = new HttpClient();
client.BaseAddress = new Uri(BaseUri);
client.DefaultRequestHeaders.Authorization =
new AuthenticationHeaderValue("apikey", "=" + token);
HttpResponseMessage response = await client.GetAsync(requestUri);
response.EnsureSuccessStatusCode();
string responseBody = await response.Content.ReadAsStringAsync();
return responseBody;
}
Then it returns null
Am I missing something or is it just totally wrong?
Thanks
You're trying to pass the API key in the header information of your HTTP request. What you need to do is just pass that whole URL without any additional header information.
IE: use "http://ssw.com/profile?apikey=abcdef" as the requestUri and send token as null. Also, remove the setting of the client.DefaultRequestHeaders.Authorization property. Authorization was meant to be a user/pass system and not a token-based system.
To test this, download Fiddler 4 (https://www.telerik.com/download/fiddler). Once you have fiddler installed, on the "Composer" tab, you can test different queries you need by putting the URL directly into the URL box and clicking "Execute". You'll then be able to use the inspectors to see the responses and figure out where you need to go from there.
Here are the classes I use for HTTP GET and POST operations:
public static string HTTPGET(string url)
{
try
{
HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url);
request.Timeout = 100000;
HttpWebResponse response = (HttpWebResponse) request.GetResponse();
Stream responseStream = response.GetResponseStream();
if (responseStream != null)
using (StreamReader resStream = new StreamReader(responseStream))
return resStream.ReadToEnd();
return null;
}
catch (Exception e)
{
Console.WriteLine(url);
Console.WriteLine(e);
return null;
}
}
public static string HTTPPOST(string url, string postData)
{
try
{
HttpWebRequest webRequest = (HttpWebRequest) WebRequest.Create(url);
webRequest.Method = "POST";
webRequest.ContentType = "x-www-form-urlencoded";
byte[] byteArray = Encoding.UTF8.GetBytes(postData);
using (Stream requestStream = webRequest.GetRequestStream())
requestStream.Write(byteArray, 0, byteArray.Length);
using (Stream responseStream = webRequest.GetResponse().GetResponseStream())
if (responseStream != null)
using (StreamReader responseReader = new StreamReader(responseStream))
return responseReader.ReadToEnd();
return null;
}
catch (Exception e)
{
Console.WriteLine(url);
Console.WriteLine(postData);
Console.WriteLine(e);
return null;
}
}
I am attempting to replicate the following C# code in Java. This code is a helper class that sends a request containing xml, and reads a response.
internal static String Send(String url, String body)
{
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
try
{
// create the new httpwebrequest with the uri
request.ContentLength = 0;
// set the method to POST
request.Method = "POST";
if (!String.IsNullOrEmpty(body))
{
request.ContentType = "application/xml; charset=utf-8";
byte[] postData = Encoding.Default.GetBytes(body);
request.ContentLength = postData.Length;
using (Stream s = request.GetRequestStream())
{
s.Write(postData, 0, postData.Length);
}
}
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
String responseString = new StreamReader(response.GetResponseStream()).ReadToEnd();
if (response.StatusCode != HttpStatusCode.OK)
{
throw new ResponseException(((int)response.StatusCode),
response.StatusCode.ToString(), request.RequestUri.ToString(),
responseString);
}
return responseString;
}
}
catch (WebException e)
{
using (WebResponse response = e.Response)
{
HttpWebResponse httpResponse = response as HttpWebResponse;
if (httpResponse != null)
{
using (Stream data = response.GetResponseStream())
{
data.Position = 0;
throw new ResponseException(((int)httpResponse.StatusCode),
httpResponse.StatusCode.ToString(), request.RequestUri.ToString(),
new StreamReader(data).ReadToEnd()
);
}
}
else
{
throw;
}
After reading other threads I determined that the Apache HttpComponents library would be my best bet to get the same functionality. After reading the documentation and following the example here:
http://hc.apache.org/httpcomponents-client-ga/quickstart.html
I am unable to figure out how to send the body string as xml. When I attempt to set the entity for the request it requires that I declare a BasicNameValuePair, and I do not understand what this is, or how I would format the body string to meet this specification.
Below is what I have currently done.
protected static String Send(String url, String body)
{
HttpPost request = new HttpPost(url);
try
{
request.setHeader("ContentType", "application/xml; charset=utf=8");
// Encode the body if needed
request.setEntity(new UrlEncodedFormEntity());
//get the response
// if the response code is not valid throw a ResponseException
// else return the response string.
} finally {
request.releaseConnection();
}
return null;
}
EDIT : or should I use a StringEntity and do the following
protected static String SendToJetstream(String url, String body)
{
HttpPost request = new HttpPost(url);
try
{
StringEntity myEntity = new StringEntity(body,
ContentType.create("application/xml", "UTF-8"));
// Encode the body if needed
request.setEntity(myEntity);
//get the response
// if the response code is not valid throw a ResponseException
// else return the response string.
} finally {
request.releaseConnection();
}
return null;
}
Use a FileEntity
File file = new File("somefile.xml");
FileEntity entity = new FileEntity(file, ContentType.create("application/xml", "UTF-8"));
Lots of good examples here: http://hc.apache.org/httpcomponents-client-ga/tutorial/html/fundamentals.html#d5e165
I need to post data to a URL (https://somesite.com) to download file in responseStrem based on the parameters I posted.
How can I do that using a C# console application?
Parameters:
filename,
userid,
password,
type
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
using System.IO;
namespace WebserverInteractionClassLibrary
{
public class RequestManager
{
public string LastResponse { protected set; get; }
CookieContainer cookies = new CookieContainer();
internal string GetCookieValue(Uri SiteUri,string name)
{
Cookie cookie = cookies.GetCookies(SiteUri)[name];
return (cookie == null) ? null : cookie.Value;
}
public string GetResponseContent(HttpWebResponse response)
{
if (response == null)
{
throw new ArgumentNullException("response");
}
Stream dataStream = null;
StreamReader reader = null;
string responseFromServer = null;
try
{
// Get the stream containing content returned by the server.
dataStream = response.GetResponseStream();
// Open the stream using a StreamReader for easy access.
reader = new StreamReader(dataStream);
// Read the content.
responseFromServer = reader.ReadToEnd();
// Cleanup the streams and the response.
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
finally
{
if (reader != null)
{
reader.Close();
}
if (dataStream != null)
{
dataStream.Close();
}
response.Close();
}
LastResponse = responseFromServer;
return responseFromServer;
}
public HttpWebResponse SendPOSTRequest(string uri, string content, string login, string password, bool allowAutoRedirect)
{
HttpWebRequest request = GeneratePOSTRequest(uri, content, login, password, allowAutoRedirect);
return GetResponse(request);
}
public HttpWebResponse SendGETRequest(string uri, string login, string password, bool allowAutoRedirect)
{
HttpWebRequest request = GenerateGETRequest(uri, login, password, allowAutoRedirect);
return GetResponse(request);
}
public HttpWebResponse SendRequest(string uri, string content, string method, string login, string password, bool allowAutoRedirect)
{
HttpWebRequest request = GenerateRequest(uri, content, method, login, password, allowAutoRedirect);
return GetResponse(request);
}
public HttpWebRequest GenerateGETRequest(string uri, string login, string password, bool allowAutoRedirect)
{
return GenerateRequest(uri, null, "GET", null, null, allowAutoRedirect);
}
public HttpWebRequest GeneratePOSTRequest(string uri, string content, string login, string password, bool allowAutoRedirect)
{
return GenerateRequest(uri, content, "POST", null, null, allowAutoRedirect);
}
internal HttpWebRequest GenerateRequest(string uri, string content, string method, string login, string password, bool allowAutoRedirect)
{
if (uri == null)
{
throw new ArgumentNullException("uri");
}
// Create a request using a URL that can receive a post.
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(uri);
// Set the Method property of the request to POST.
request.Method = method;
// Set cookie container to maintain cookies
request.CookieContainer = cookies;
request.AllowAutoRedirect = allowAutoRedirect;
// If login is empty use defaul credentials
if (string.IsNullOrEmpty(login))
{
request.Credentials = CredentialCache.DefaultNetworkCredentials;
}
else
{
request.Credentials = new NetworkCredential(login, password);
}
if (method == "POST")
{
// Convert POST data to a byte array.
byte[] byteArray = Encoding.UTF8.GetBytes(content);
// Set the ContentType property of the WebRequest.
request.ContentType = "application/x-www-form-urlencoded";
// Set the ContentLength property of the WebRequest.
request.ContentLength = byteArray.Length;
// Get the request stream.
Stream dataStream = request.GetRequestStream();
// Write the data to the request stream.
dataStream.Write(byteArray, 0, byteArray.Length);
// Close the Stream object.
dataStream.Close();
}
return request;
}
internal HttpWebResponse GetResponse(HttpWebRequest request)
{
if (request == null)
{
throw new ArgumentNullException("request");
}
HttpWebResponse response = null;
try
{
response = (HttpWebResponse)request.GetResponse();
cookies.Add(response.Cookies);
// Print the properties of each cookie.
Console.WriteLine("\nCookies: ");
foreach (Cookie cook in cookies.GetCookies(request.RequestUri))
{
Console.WriteLine("Domain: {0}, String: {1}", cook.Domain, cook.ToString());
}
}
catch (WebException ex)
{
Console.WriteLine("Web exception occurred. Status code: {0}", ex.Status);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
return response;
}
}
}
Take a look at the System.Net.WebClient class, it can be used to issue requests and handle their responses, as well as to download files:
http://www.hanselman.com/blog/HTTPPOSTsAndHTTPGETsWithWebClientAndCAndFakingAPostBack.aspx
http://msdn.microsoft.com/en-us/library/system.net.webclient(VS.90).aspx
For this you can simply use the "HttpWebRequest" and "HttpWebResponse" classes in .net.
Below is a sample console app I wrote to demonstrate how easy this is.
using System;
using System.Collections.Generic;
using System.Text;
using System.Net;
using System.IO;
namespace Test
{
class Program
{
static void Main(string[] args)
{
string url = "www.somewhere.com";
string fileName = #"C:\output.file";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Timeout = 5000;
try
{
using (WebResponse response = (HttpWebResponse)request.GetResponse())
{
using (FileStream stream = new FileStream(fileName, FileMode.Create, FileAccess.Write))
{
byte[] bytes = ReadFully(response.GetResponseStream());
stream.Write(bytes, 0, bytes.Length);
}
}
}
catch (WebException)
{
Console.WriteLine("Error Occured");
}
}
public static byte[] ReadFully(Stream input)
{
byte[] buffer = new byte[16 * 1024];
using (MemoryStream ms = new MemoryStream())
{
int read;
while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
{
ms.Write(buffer, 0, read);
}
return ms.ToArray();
}
}
}
}
Enjoy!
HttpWebRequest request =(HttpWebRequest)WebRequest.Create("some url");
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.UserAgent = "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 7.1; Trident/5.0)";
request.Accept = "/";
request.UseDefaultCredentials = true;
request.Proxy.Credentials = System.Net.CredentialCache.DefaultCredentials;
doc.Save(request.GetRequestStream());
HttpWebResponse resp = request.GetResponse() as HttpWebResponse;
Hope it helps
Insted of using System.Net.WebClient I would recommend to have a look on System.Net.Http.HttpClient which was introduced with net 4.5 and makes your life much easier.
Also microsoft recommends to use the HttpClient on this article
http://msdn.microsoft.com/en-us/library/system.net.webclient(VS.90).aspx
An example could look like this:
var client = new HttpClient();
var content = new MultipartFormDataContent
{
{ new StringContent("myUserId"), "userid"},
{ new StringContent("myFileName"), "filename"},
{ new StringContent("myPassword"), "password"},
{ new StringContent("myType"), "type"}
};
var responseMessage = await client.PostAsync("some url", content);
var stream = await responseMessage.Content.ReadAsStreamAsync();
I'm trying to create a method in C# to return a string of a web pages html content from the url. I have tried several different ways, but I am getting the error System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive.
The following works fine locally, but gets the above error when running on a remote server:
public static string WebPageRead(string url)
{
string result = String.Empty;
WebResponse response = null;
StreamReader reader = null;
try
{
if (!String.IsNullOrEmpty(url))
{
HttpWebRequest request = HttpWebRequest.Create(url) as HttpWebRequest;
request.Method = "GET";
request.KeepAlive = false;
request.ProtocolVersion = HttpVersion.Version10;
response = request.GetResponse();
reader = new StreamReader(response.GetResponseStream(), Encoding.UTF8);
result = reader.ReadToEnd();
}
}
catch (Exception exc)
{
throw exc;
}
finally
{
if (reader != null)
{
reader.Close();
}
if (response != null)
{
response.Close();
}
}
return result;
}
This is probably not the problem, but try the following:
public static string WebPageRead(string url)
{
if (String.IsNullOrEmpty(url))
{
return null;
}
HttpWebRequest request = HttpWebRequest.Create(url) as HttpWebRequest;
if (request == null)
{
return null;
}
request.Method = "GET";
request.KeepAlive = false;
request.ProtocolVersion = HttpVersion.Version10;
using (WebResponse response = request.GetResponse())
{
using (Stream stream = response.GetResponseStream())
{
using (StreamReader reader =
new StreamReader(stream, Encoding.UTF8))
{
return reader.ReadToEnd();
}
}
}
}
I echo the earlier answer that suggests you try this with a known good URL. I'll add that you should try this with a known good HTTP 1.1 URL, commenting out the line that sets the version to 1.0. If that works, then it narrows things down considerably.
Thanks for the responses, the problem was due to a DNS issue on the remote server! Just to confirm, I went with the following code in the end:
public static string WebPageRead(string url)
{
string content = String.Empty;
if (!String.IsNullOrEmpty(url))
{
HttpWebRequest request = HttpWebRequest.Create(url) as HttpWebRequest;
if (request != null)
{
request.Method = "GET";
request.KeepAlive = false;
request.ProtocolVersion = HttpVersion.Version10;
try
{
using (WebResponse response = request.GetResponse())
{
using (Stream stream = response.GetResponseStream())
{
using (StreamReader reader = new StreamReader(stream, Encoding.UTF8))
{
content = reader.ReadToEnd();
}
}
}
}
catch (Exception exc)
{
throw exc;
}
}
}
return content;
}
Had a problem like this before that was solved by opening the url in IE on the machine with the problem. IE then asks you whether you want to add the url to the list of secure sites. Add it and it works for that url.
This is just one of the possible causes. Seriously a lot of other problems could cause this. Besides the problem described above, the best way I've found to solve this is the just catch the exception and retry the request.