Develop Reference

Getting strange chars from Content-Encoding = Gzip / response in C# [duplicate]

I am trying to connect to an api, that returns GZip encoded JSON, from a WCF service (WCF service to WCF service). I am using the HTTPClient to connect to the API and have been able to return the JSON object as a string. However I need to be able to store this returned data in a database and as such I figured the best way would be to return and store the JSON object in an array or byte or something along those lines.
What I am having trouble with specifically is the decompressing of the GZip encoding and have been trying lots of different example but still cant get it.
The below code is how I am establishing my connection and getting a response, this is the code that returns a string from the API.
public string getData(string foo)
{
string url = "";
HttpClient client = new HttpClient();
HttpResponseMessage response;
string responseJsonContent;
try
{
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
response = client.GetAsync(url + foo).Result;
responseJsonContent = response.Content.ReadAsStringAsync().Result;
return responseJsonContent;
}
catch (Exception ex)
{
System.Windows.Forms.MessageBox.Show(ex.Message);
return "";
}
}
I have been following a few different examples like these StackExchange API, MSDN, and a couple on stackoverflow, but I haven't been able to get any of these to work for me.
What is the best way to accomplish this, am I even on the right track?
Thanks guys.

Just instantiate HttpClient like this:
HttpClientHandler handler = new HttpClientHandler()
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
using (var client = new HttpClient(handler)) //see update below
{
// your code
}
Update June 19, 2020:
It's not recommended to use httpclient in a 'using' block as it might cause port exhaustion.
private static HttpClient client = null;
ContructorMethod()
{
if(client == null)
{
HttpClientHandler handler = new HttpClientHandler()
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
client = new HttpClient(handler);
}
// your code
}
If using .Net Core 2.1+, consider using IHttpClientFactory and injecting like this in your startup code.
var timeout = Policy.TimeoutAsync<HttpResponseMessage>(
TimeSpan.FromSeconds(60));
services.AddHttpClient<XApiClient>().ConfigurePrimaryHttpMessageHandler(() => new HttpClientHandler
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
}).AddPolicyHandler(request => timeout);

I used code from below link to decompress GZip stream.Then used the decompressed byte array to get the required JSON object. Hope it may help some one.
var readTask = result.Content.ReadAsByteArrayAsync().Result;
var decompressedData = Decompress(readTask);
string jsonString = System.Text.Encoding.UTF8.GetString(decompressedData, 0, decompressedData.Length);
ResponseObjectClass responseObject = Newtonsoft.Json.JsonConvert.DeserializeObject<ResponseObjectClass>(jsonString);
https://www.dotnetperls.com/decompress
static byte[] Decompress(byte[] gzip)
{
using (GZipStream stream = new GZipStream(new MemoryStream(gzip), CompressionMode.Decompress))
{
const int size = 4096;
byte[] buffer = new byte[size];
using (MemoryStream memory = new MemoryStream())
{
int count = 0;
do
{
count = stream.Read(buffer, 0, size);
if (count > 0)
{
memory.Write(buffer, 0, count);
}
}
while (count > 0);
return memory.ToArray();
}
}
}

Ok so I eventually solved my problem. If there are better ways please let me know :-)
public DataSet getData(string strFoo)
{
string url = "foo";
HttpClient client = new HttpClient();
HttpResponseMessage response;
DataSet dsTable = new DataSet();
try
{
//Gets the headers that should be sent with each request
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
//Returned JSON
response = client.GetAsync(url).Result;
//converts JSON to string
string responseJSONContent = response.Content.ReadAsStringAsync().Result;
//deserializes string to list
var jsonList = DeSerializeJsonString(responseJSONContent);
//converts list to dataset. Bad name I know.
dsTable = Foo_ConnectAPI.ExtentsionHelpers.ToDataSet<RootObject>(jsonList);
//Returns the dataset
return dsTable;
}
catch (Exception ex)
{
System.Windows.Forms.MessageBox.Show(ex.Message);
return null;
}
}
//deserializes the string to a list. Utilizes JSON.net. RootObject is a class that contains the get and set for the JSON elements
public List<RootObject> DeSerializeJsonString(string jsonString)
{
//Initialized the List
List<RootObject> list = new List<RootObject>();
//json.net deserializes string
list = (List<RootObject>)JsonConvert.DeserializeObject<List<RootObject>>(jsonString);
return list;
}
The RootObject contains the get set that will get the values of the JSON.
public class RootObject
{
//These string will be set to the elements within the JSON. Each one is directly mapped to the JSON elements.
//This only takes into account a JSON that doesn't contain nested arrays
public string EntityID { get; set; }
public string Address1 { get; set; }
public string Address2 { get; set; }
public string Address3 { get; set; }
}
The easiest way to create the above class(es) is to use json2charp which will format it accordingly and also provide the correct datatypes.
The following is from another answer on Stackoverflow
again it does not take into account nested JSON.
internal static class ExtentsionHelpers
{
public static DataSet ToDataSet<T>(this List<RootObject> list)
{
try
{
Type elementType = typeof(RootObject);
DataSet ds = new DataSet();
DataTable t = new DataTable();
ds.Tables.Add(t);
try
{
//add a column to table for each public property on T
foreach (var propInfo in elementType.GetProperties())
{
try
{
Type ColType = Nullable.GetUnderlyingType(propInfo.PropertyType) ?? propInfo.PropertyType;
t.Columns.Add(propInfo.Name, ColType);
}
catch (Exception ex)
{
System.Windows.Forms.MessageBox.Show(ex.Message);
}
}
}
catch (Exception ex)
{
System.Windows.Forms.MessageBox.Show(ex.Message);
}
try
{
//go through each property on T and add each value to the table
foreach (RootObject item in list)
{
DataRow row = t.NewRow();
foreach (var propInfo in elementType.GetProperties())
{
row[propInfo.Name] = propInfo.GetValue(item, null) ?? DBNull.Value;
}
t.Rows.Add(row);
}
}
catch (Exception ex)
{
System.Windows.Forms.MessageBox.Show(ex.Message);
}
insert.insertCategories(t);
return ds.
}
catch (Exception ex)
{
System.Windows.Forms.MessageBox.Show(ex.Message);
return null;
}
}
};
Then finally to insert the above dataset into a table with columns that were mapped to the JSON I utilized SQL bulk copy and the following class
public class insert
{
public static string insertCategories(DataTable table)
{
SqlConnection objConnection = new SqlConnection();
//As specified in the App.config/web.config file
objConnection.ConnectionString = System.Configuration.ConfigurationManager.ConnectionStrings["foo"].ToString();
try
{
objConnection.Open();
var bulkCopy = new SqlBulkCopy(objConnection.ConnectionString);
bulkCopy.DestinationTableName = "dbo.foo";
bulkCopy.BulkCopyTimeout = 600;
bulkCopy.WriteToServer(table);
return "";
}
catch (Exception ex)
{
System.Windows.Forms.MessageBox.Show(ex.Message);
return "";
}
finally
{
objConnection.Close();
}
}
};
So the above works to insert JSON from a webAPI into a database. This is something that I get to work. But by no means do I expect it to be perfect. If you have any improvements then please update it accordingly.

Sooner or later your code might brake if your server use another compression scheme like 'Brotli' with content-type: br
I have made a video to handle decompression in httpClient with the clientHandler:
compression algos handled by httpClientHandler

Forward HTTP request to another server

I have the following code that receives webhook messages:
// Read posted data
string requestBody;
using (var reader = new StreamReader(HttpContext.Current.Request.InputStream))
{
requestBody = reader.ReadToEnd();
}
requestBody.Log();
// Attempt to forward request
context.CopyTo(Settings.Payments.Paypal.ScirraPaypalIPNEndpoint);
requestBody contains data which is logged. I then attempt to forward the request to another URL:
public static void CopyTo(this HttpContext source, string url)
{
var destination = (HttpWebRequest) WebRequest.Create(url);
var request = source.Request;
destination.Method = request.HttpMethod;
// Copy unrestricted headers
foreach (var headerKey in request.Headers.AllKeys)
{
if (WebHeaderCollection.IsRestricted(headerKey)) continue;
destination.Headers[headerKey] = request.Headers[headerKey];
}
// Copy restricted headers
if (request.AcceptTypes != null && request.AcceptTypes.Any())
{
destination.Accept = string.Join(",", request.AcceptTypes);
}
destination.ContentType = request.ContentType;
destination.Referer = request.UrlReferrer?.AbsoluteUri ?? string.Empty;
destination.UserAgent = request.UserAgent;
// Copy content (if content body is allowed)
if (request.HttpMethod != "GET"
&& request.HttpMethod != "HEAD"
&& request.ContentLength > 0)
{
using (var destinationStream = destination.GetRequestStream())
{
request.InputStream.Position = 0;
request.InputStream.CopyTo(destinationStream);
destinationStream.Close();
}
}
if (!Settings.Deployment.IsLive)
{
ServicePointManager.ServerCertificateValidationCallback =
(sender, certificate, chain, sslPolicyErrors) => true;
}
using (var response = destination.GetResponse() as HttpWebResponse)
{
if (response == null) throw new Exception("Failed to post to " + url);
}
}
The handler that receives this forwarded request has the code:
public void ProcessRequest(HttpContext context)
{
string requestBody;
using (var reader = new StreamReader(HttpContext.Current.Request.InputStream))
{
requestBody = reader.ReadToEnd();
}
requestBody.Log();
}
However on the handler forwarded to, requestBody is always empty! What am I doing wrong here?

Both servers are hosted in Clouflare, when posting from one to the other I get a CF 1000 prohibited IP error.
Solution is to add target servers IP address into requesting servers hosts file.

Read website text content using Web Browser from System.Window.Form in plain C# class

I am intended to read text content from given url. I have managed to put together logic using HtmlAgilityPack however it is giving me javaScript and other syntax tags, unless I am missing something, however I have found I can use System.Window.Form--> WebBrowser to achieve this. Now I need to achieve this in C# class with no GUI or Window control as I am intended to filter and put in database followed by display on ASP.NET-MVC app.
Now how can I use this library to upload web url and read all the text content, followed by store in string?????
however for reference here is following code using HtmlAgilityPack
public string BuildWebReponseString(string _url)
{
if (obj1.initializeWebURL(_url))
{
HttpWebRequest request = (HttpWebRequest)
WebRequest.Create(_url);
HttpWebResponse response = (HttpWebResponse) request.GetResponse();
Stream resStream = response.GetResponseStream();
StreamReader reader = new StreamReader(resStream);
WebURLContent = reader.ReadToEnd();
return WebURLContent;
}
else
{
return null;
}
}
...
public List<string> StripWebContent(string _url)
{
var _responseContent = _reponseObj.BuildWebReponseString(_url);
if (_responseContent != null)
{
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(_responseContent);
var textNodes = doc.DocumentNode.SelectNodes("//text()");
var FilteredWebContent = string.Join(" ", textNodes.Select(n => n.InnerText.Trim()));
//
string[] tempString = FilteredWebContent.Split(' ');
for (int i = 0; i < tempString.Length; i++)
{
if (tempString[i] != "" && tempString[i] != null)
{
_WebContentsList.Add(tempString[i]);
}
}
return _WebContentsList;
}
else
{
return null;
}
}

How to post data to specific URL using WebClient in C#

I need to use "HTTP Post" with WebClient to post some data to a specific URL I have.
Now, I know this can be accomplished with WebRequest but for some reasons I want to use WebClient instead. Is that possible? If so, can someone show me some example or point me to the right direction?

I just found the solution and yea it was easier than I thought :)
so here is the solution:
string URI = "http://www.myurl.com/post.php";
string myParameters = "param1=value1&param2=value2&param3=value3";
using (WebClient wc = new WebClient())
{
wc.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
string HtmlResult = wc.UploadString(URI, myParameters);
}
it works like charm :)

There is a built in method called UploadValues that can send HTTP POST (or any kind of HTTP methods) AND handles the construction of request body (concatenating parameters with "&" and escaping characters by url encoding) in proper form data format:
using(WebClient client = new WebClient())
{
var reqparm = new System.Collections.Specialized.NameValueCollection();
reqparm.Add("param1", "<any> kinds & of = ? strings");
reqparm.Add("param2", "escaping is already handled");
byte[] responsebytes = client.UploadValues("http://localhost", "POST", reqparm);
string responsebody = Encoding.UTF8.GetString(responsebytes);
}

Using WebClient.UploadString or WebClient.UploadData you can POST data to the server easily. I’ll show an example using UploadData, since UploadString is used in the same manner as DownloadString.
byte[] bret = client.UploadData("http://www.website.com/post.php", "POST",
System.Text.Encoding.ASCII.GetBytes("field1=value1&field2=value2") );
string sret = System.Text.Encoding.ASCII.GetString(bret);
More: http://www.daveamenta.com/2008-05/c-webclient-usage/

string URI = "site.com/mail.php";
using (WebClient client = new WebClient())
{
System.Collections.Specialized.NameValueCollection postData =
new System.Collections.Specialized.NameValueCollection()
{
{ "to", emailTo },
{ "subject", currentSubject },
{ "body", currentBody }
};
string pagesource = Encoding.UTF8.GetString(client.UploadValues(URI, postData));
}

//Making a POST request using WebClient.
Function()
{
WebClient wc = new WebClient();
var URI = new Uri("http://your_uri_goes_here");
//If any encoding is needed.
wc.Headers["Content-Type"] = "application/x-www-form-urlencoded";
//Or any other encoding type.
//If any key needed
wc.Headers["KEY"] = "Your_Key_Goes_Here";
wc.UploadStringCompleted +=
new UploadStringCompletedEventHandler(wc_UploadStringCompleted);
wc.UploadStringAsync(URI,"POST","Data_To_Be_sent");
}
void wc__UploadStringCompleted(object sender, UploadStringCompletedEventArgs e)
{
try
{
MessageBox.Show(e.Result);
//e.result fetches you the response against your POST request.
}
catch(Exception exc)
{
MessageBox.Show(exc.ToString());
}
}

Using simple client.UploadString(adress, content); normally works fine but I think it should be remembered that a WebException will be thrown if not a HTTP successful status code is returned. I usually handle it like this to print any exception message the remote server is returning:
try
{
postResult = client.UploadString(address, content);
}
catch (WebException ex)
{
String responseFromServer = ex.Message.ToString() + " ";
if (ex.Response != null)
{
using (WebResponse response = ex.Response)
{
Stream dataRs = response.GetResponseStream();
using (StreamReader reader = new StreamReader(dataRs))
{
responseFromServer += reader.ReadToEnd();
_log.Error("Server Response: " + responseFromServer);
}
}
}
throw;
}

Using webapiclient with model send serialize json parameter request.
PostModel.cs
public string Id { get; set; }
public string Name { get; set; }
public string Surname { get; set; }
public int Age { get; set; }
WebApiClient.cs
internal class WebApiClient : IDisposable
{
private bool _isDispose;
public void Dispose()
{
Dispose(true);
GC.SuppressFinalize(this);
}
public void Dispose(bool disposing)
{
if (!_isDispose)
{
if (disposing)
{
}
}
_isDispose = true;
}
private void SetHeaderParameters(WebClient client)
{
client.Headers.Clear();
client.Headers.Add("Content-Type", "application/json");
client.Encoding = Encoding.UTF8;
}
public async Task<T> PostJsonWithModelAsync<T>(string address, string data,)
{
using (var client = new WebClient())
{
SetHeaderParameters(client);
string result = await client.UploadStringTaskAsync(address, data); // method:
//The HTTP method used to send the file to the resource. If null, the default is POST
return JsonConvert.DeserializeObject<T>(result);
}
}
}
Business caller method
public async Task<ResultDTO> GetResultAsync(PostModel model)
{
try
{
using (var client = new WebApiClient())
{
var serializeModel= JsonConvert.SerializeObject(model);// using Newtonsoft.Json;
var response = await client.PostJsonWithModelAsync<ResultDTO>("http://www.website.com/api/create", serializeModel);
return response;
}
}
catch (Exception ex)
{
throw new Exception(ex.Message);
}
}

Most of the answers are old. Just wanted to share what worked for me. In the interest of doing things asynchronously i.e. to post data to specific URL using WebClient asynchronously in .NET 6.0 Preview 7, .NET Core and other versions can be done using WebClient.UploadStringTaskAsync Method.
Use namespace System.Net; and for a class ResponseType to capture the response from the server, we can use this method to POST data to a specific URL. Make sure to use the await keyword while calling this method
public async Task<ResponseType> MyAsyncServiceCall()
{
try
{
var uri = new Uri("http://your_uri");
var body= "param1=value1&param2=value2&param3=value3";
using (var wc = new WebClient())
{
wc.Headers[HttpRequestHeader.Authorization] = "yourKey"; // Can be Bearer token, API Key etc.....
wc.Headers[HttpRequestHeader.ContentType] = "application/json"; // Is about the payload/content of the current request or response. Do not use it if the request doesn't have a payload/ body.
wc.Headers[HttpRequestHeader.Accept] = "application/json"; // Tells the server the kind of response the client will accept.
wc.Headers[HttpRequestHeader.UserAgent] = "PostmanRuntime/7.28.3";
string result = await wc.UploadStringTaskAsync(uri, body);
return JsonConvert.DeserializeObject<ResponseType>(result);
}
}
catch (Exception e)
{
throw new Exception(e.Message);
}
}

Here is the crisp answer:
public String sendSMS(String phone, String token) {
WebClient webClient = WebClient.create(smsServiceUrl);
SMSRequest smsRequest = new SMSRequest();
smsRequest.setMessage(token);
smsRequest.setPhoneNo(phone);
smsRequest.setTokenId(smsServiceTokenId);
Mono<String> response = webClient.post()
.uri(smsServiceEndpoint)
.header(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE)
.body(Mono.just(smsRequest), SMSRequest.class)
.retrieve().bodyToMono(String.class);
String deliveryResponse = response.block();
if (deliveryResponse.equalsIgnoreCase("success")) {
return deliveryResponse;
}
return null;
}

Send XML String as Response

I am getting my Request from a third party application(different domain) to my ASP application. I am handling the request and doing the business part in my application and as a acknowledgement I need to send XML string as Response to the same Page which POSTED the request to my Application. I was successful in retrieving the input from Request using the following code
NameValueCollection postPageCollection = Request.Form;
foreach (string name in postPageCollection.AllKeys)
{
... = postPageCollection[name]);
}
But i am not sure how to send back the response along with XML String to the site(different domain)?
EDIT: How to get the URL from where the POST happened.

You can get the url that come from Request.ServerVariables["HTTP_REFERER"]
For the XML, here are 2 functions that I use
public static string ObjectToXML(Type type, object obby)
{
XmlSerializer ser = new XmlSerializer(type);
using (System.IO.MemoryStream stm = new System.IO.MemoryStream())
{
//serialize to a memory stream
ser.Serialize(stm, obby);
//reset to beginning so we can read it.
stm.Position = 0;
//Convert a string.
using (System.IO.StreamReader stmReader = new System.IO.StreamReader(stm))
{
string xmlData = stmReader.ReadToEnd();
return xmlData;
}
}
}
public static object XmlToObject(Type type, string xml)
{
object oOut = null;
//hydrate based on private string var
if (xml != null && xml.Length > 0)
{
System.Xml.Serialization.XmlSerializer serializer = new System.Xml.Serialization.XmlSerializer(type);
using (System.IO.StringReader sReader = new System.IO.StringReader(xml))
{
oOut = serializer.Deserialize(sReader);
sReader.Close();
}
}
return oOut;
}
And here is an example how I use it
[Serializable]
public class MyClassThatKeepTheData
{
public int EnaTest;
}
MyClassThatKeepTheData cTheObject = new MyClassThatKeepTheData();
ObjectToXML(typeof(MyClassThatKeepTheData), cTheObject)

Cant you just use the following code:
Request.UrlReferrer.ToString();