Develop Reference

streamReader.ReadToEndAsync() changing message content

I have a .Net Core 3.1 Web API that needs to interact with an external service that is providing a multipart/related response from a WCF Service with attachments in the context as the below block.
HTTP/1.1 200 OK
Cache-Control: private
Content-Length: 1790633
Content-Type: multipart/related; type="text/xml"; start="<8fb5b3efa6474add9032cb80870dc065>"; boundary="------=_Part_20200731120159.494997"
Server: Microsoft-IIS/10.0
Set-Cookie: ASP.NET_SessionId=jga2opbhoirgnhjhmmsuxi1k; path=/; HttpOnly; SameSite=Lax
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Set-Cookie: ARRAffinity=f7b4f7f5d8cbd8c8bc5d9a4bf49e8b5938412c4294f6e2c2b5c2d5cfc4b4d437;Path=/;HttpOnly;Domain=lparouterpoc.azurewebsites.net
Date: Fri, 31 Jul 2020 13:33:04 GMT
--------=_Part_20200731120159.494997
Content-Type: text/xml; charset=UTF-8
Content-Transfer-Encoding: binary
Content-Id: <8fb5b3efa6474add9032cb80870dc065>
<?xml version="1.0" encoding="iso-8859-1"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
...
</SOAP-ENV:Envelope>
--------=_Part_20200731120159.494997
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
Content-Id: 7b1fff220df5#xxxxx
%PDF-1.5
%����
...
Additional characters added in here
...
%%EOF
--------=_Part_20200731120159.494997
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
Content-Id: 844eb478dc05#xxxxx
%PDF-1.5
...
Additional characters added here
...
%%EOF
--------=_Part_20200731120159.494997--
Using SOAP UI, I see that there are attachments, but each attachment is a blank PDF.
The code in the .Net Core API is
using (var response = await Client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead))
{
var stream = await response.Content.ReadAsStreamAsync();
using var streamReader = new StreamReader(stream, Encoding.UTF8);
var message = await streamReader.ReadToEndAsync(); // This is where output is changed to add additional characters
var headers = response.Headers;
return new customResponse
{
ContentResult = new ContentResult
{
Content = message,
ContentType = response.Content.Headers.ContentType?.ToString(),
StatusCode = (int)response.StatusCode
},
ResponseHeaders = headers
};
}
The image below shows an example of where the additional characters are coming in causing the PDF's to be invalid \ corrupt \ blank.
An additional characteristic of this is the content length going from 1021690 to 1790633.
I have been able to debug it this far and test that the files before they get into the message (string) are correct as I am able to read the multipart content and write the files out to disk when checking that the files are indeed correct using
var content = await response.Content.ReadAsMultipartAsync();
for (var i = 0; i < content.Contents.Count; i++)
{
var item = content.Contents[i];
if (item.Headers.ContentType.ToString() == "application/octet-stream")
{
var bytes = await item.ReadAsByteArrayAsync();
File.WriteAllBytes(#$"c:\temp\{i}.pdf", bytes);
}
}
and then iterating through the content.Contents.
I would be grateful if anyone could tell me how \ why this is occurring and how I can stop this from occuring
Further investigation and hypothesis is could this be down to encoding? The WCF service for the attachments is TransferEncoding = "binary"
Left and middle of the image are the same, the middle written to disk direct from my API (not what I need) shows me that the file can be read and written.
The right is the direct output from the service, which comes back into SOAP UI and downloads as expected.
When looking at the response in the watch window the transfer encoding is empty so am a little confused on how to resolve this.

try this,
public async Task<HttpResponseMessage> Post(string requestBody, string action)
{
using (var request = new HttpRequestMessage(HttpMethod.Post, ""))
{
request.Headers.Add("SOAPAction", action);
request.Content = new StringContent(requestBody, Encoding.UTF8, "text/xml");
var response = await Client.SendAsync(request);
return response;
}
}
and then in your controller return it like this
var result = await _service.Post(requestBody, action);
var stream = await result.Content.ReadAsStreamAsync();
await stream.CopyToAsync(HttpContext.Response.Body);

Make POST request

I'm trying to post to a web form defined as:
<form name="frmdata" method='post' enctype ='multipart/form-data' action ="http://www.rzp.cz/cgi-bin/aps_cacheWEB.sh">
<input type ="hidden" name ="VSS_SERV" value="ZVWSBJXML">
<input type="file" name="filename">
<input type ='submit' name ='x' value ='ODESLI'>
</form>
There is some additional documentation on the form here:
http://www.rzp.cz/docs/RZP02_XML_28.pdf
My latest try:
using (WebClient client = new WebClient())
{
NameValueCollection vals = new NameValueCollection();
vals.Add("VSS_SERV", "ZVWSBJXML");
string filecontent = #"<?xml version=""1.0"" encoding=""ISO-8859-2""?>";
filecontent = filecontent + #"
<VerejnyWebDotaz
elementFormDefault=""qualified""
targetNamespace=""urn:cz:isvs:rzp:schemas:VerejnaCast:v1""
xmlns=""urn:cz:isvs:rzp:schemas:VerejnaCast:v1"" version=""2.8"">";
filecontent = filecontent + #"
<Kriteria>
<IdentifikacniCislo>03358437</IdentifikacniCislo>
<PlatnostZaznamu>0</PlatnostZaznamu></Kriteria>";
filecontent = filecontent + #"</VerejnyWebDotaz>";
vals.Add("filename", filecontent);
client.Headers["ContentType"] = "multipart/form-data";
byte[] responseArray = client.UploadValues(#"http://www.rzp.cz/cgi-bin/aps_cacheWEB.sh", "POST", vals);
string str = Encoding.ASCII.GetString(responseArray);
}
But I can't get past this error:
<KodChyby>-1</KodChyby> (the xml filename does not contain xml defined by namespace)
How can I send this xml data to the form or rather there is a working form - http://stuff.petrovsky.cz/subdom/stuff/RZP/rzp-test-form.php - how to call and catch xml data? I would like to do the same request and get xml.

Using System.Net.Http I was able to construct the form request as a proof of concept using MultipartFormDataContent
Now initially when I tested it, I received 403 Forbidden response but I guessed that was to be expected given my location and that the endpoint might be region locked.
Raw Fiddler response
HTTP/1.1 403 Forbidden
Date: Sat, 27 Oct 2018 01:37:09 GMT
Server: IIS
Content-Length: 225
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /cgi-bin/aps_cacheWEB.sh
on this server.</p>
</body></html>
I was wrong and the forbidden appeared to be the default response for bad requests as you commented that you received the same forbidden error from within the region. So back to the drawing board I went.
I then copied the example HTML form locally and then proceeded to compare the requests from the form (which did actually work) and my code. Gradually making changes to match I was finally able to get a 200 OK response, but the body of the response was empty.
Apparently there was an issue with the server interpreting the boundary in the content type header if it is wrapped in quotes boundary="...".
After more adjustments it then started returning a message based on the content dispositions generated.
HTTP/1.1 200 OK
Date: Sat, 27 Oct 2018 19:55:11 GMT
Server: IIS
Serial: 10.145
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/plain; charset=ISO-8859-2
Content-Length: 169
Multiple definitions of VSS_SERV encountered in input.
If you're trying to do this intentionally (such as with select),
the variable must have a "List" suffix.
So it turns out that the XML API is expecting the request to be in a very specific format. Deviate from that and the request fails.
The MultipartFormDataContent was not generating the request correctly and this caused the server to not behave as expected. Other headers were being placed before the Content-Disposition headers of the parts and the Content-Disposition parameters were also not being enclosed in quotes. So by not including the content-type it in the parts and making sure the content-disposition headers were generated correctly eventually fixed the problem.
It is important to note the order of how the headers are added to the content so that the Content-Disposition header is set first for each part.
Working Code that generates the request in the desired format and gets the XML data.
[Test]
public async Task Post_Form() {
//Arrange
var stream = getXml();
var fileContent = new StreamContent(stream);
fileContent.Headers.ContentDisposition = new ContentDispositionHeaderValue("form-data") {
Name = #"""filename""",
FileName = #"""req-details.xml""",
};
fileContent.Headers.ContentType = new MediaTypeHeaderValue("text/xml");
var stringContent = new ByteArrayContent(Encoding.UTF8.GetBytes("ZVWSBJXML"));
stringContent.Headers.ContentDisposition = new ContentDispositionHeaderValue("form-data") {
Name = #"""VSS_SERV""",
};
//could have let system generate it but wanteed to rule it out as a problem
var boundary = "---------------------------" + DateTime.Now.Ticks.ToString("x", NumberFormatInfo.InvariantInfo);
var form = new MultipartFormDataContent(boundary);
//FIX: boundary quote issue
var contentType = form.Headers.ContentType.Parameters.First(o => o.Name == "boundary");
contentType.Value = contentType.Value.Replace("\"", String.Empty);
form.Add(stringContent);
form.Add(fileContent);
//var data = await form.ReadAsStringAsync(); //FOR TESTING PORPOSES ONLY!!
var client = createHttpClient("http://www.rzp.cz/");
//Act
var response = await client.PostAsync("cgi-bin/aps_cacheWEB.sh", form);
var body = await response.Content.ReadAsStringAsync();
//Assert
response.IsSuccessStatusCode.Should().BeTrue();
body.Should().NotBeEmpty();
var document = XDocument.Parse(body); //should be valid XML
document.Should().NotBeNull();
}
The code above generated the following request, which I extracted using fiddler (Pay close attention to the working format)
POST http://www.rzp.cz/cgi-bin/aps_cacheWEB.sh HTTP/1.1
User-Agent: System.Net.Http.HttpClient
Accept-Language: en-US, en; q=0.9
Accept: text/xml, application/xml
Cache-Control: max-age=0
Content-Type: multipart/form-data; boundary=---------------------------8d63c301f3e044f
Host: www.rzp.cz
Content-Length: 574
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
-----------------------------8d63c301f3e044f
Content-Disposition: form-data; name="VSS_SERV"
ZVWSBJXML
-----------------------------8d63c301f3e044f
Content-Disposition: form-data; name="filename"; filename="req-details.xml"
Content-Type: text/xml
<?xml version="1.0" encoding="iso-8859-2"?>
<VerejnyWebDotaz xmlns="urn:cz:isvs:rzp:schemas:VerejnaCast:v1" version="2.8">
<Kriteria>
<IdentifikacniCislo>75848899</IdentifikacniCislo>
<PlatnostZaznamu>0</PlatnostZaznamu>
</Kriteria>
</VerejnyWebDotaz>
-----------------------------8d63c301f3e044f--
Which was able to get the following response.
HTTP/1.1 200 OK
Date: Sat, 27 Oct 2018 21:17:50 GMT
Server: IIS
Serial: 10.145
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/xml;charset=ISO-8859-2
Content-Length: 931
<?xml version='1.0' encoding='iso-8859-2'?>
<VerejnyWebOdpoved xmlns="urn:cz:isvs:rzp:schemas:VerejnaCast:v1" version="2.8">
<Datum>27.10.2018</Datum>
<Kriteria>
<IdentifikacniCislo>75848899</IdentifikacniCislo>
<PlatnostZaznamu>0</PlatnostZaznamu>
</Kriteria>
<PodnikatelSeznam>
<PodnikatelID>212fbf8314e01506b0d7</PodnikatelID>
<ObchodniJmenoSeznam Popis="Jméno a příjmení:">Filip Zrůst</ObchodniJmenoSeznam>
<IdentifikacniCisloSeznam Popis="Identifikační číslo osoby:">75848899</IdentifikacniCisloSeznam>
<TypPodnikatele Popis="Typ podnikatele:">Fyzická osoba</TypPodnikatele>
<AdresaPodnikaniSeznam Popis="Adresa sídla:">Vlašská 358/7, 118 00, Praha 1 - Malá Strana</AdresaPodnikaniSeznam>
<RoleSubjektu Popis="Role subjektu:">podnikatel</RoleSubjektu>
<EvidujiciUrad Popis="Úřad příslušný podle §71 odst.2 živnostenského zákona:">Úřad městské části Praha 1</EvidujiciUrad>
</PodnikatelSeznam>
</VerejnyWebOdpoved>
From there it should be small work to parse the resulting XML as needed.
Supporting code
Generate or load the stream of the XML for the form
private static Stream getXml() {
var xml = #"<?xml version=""1.0"" encoding=""ISO-8859-2""?>
<VerejnyWebDotaz
xmlns=""urn:cz:isvs:rzp:schemas:VerejnaCast:v1""
version=""2.8"">
<Kriteria>
<IdentifikacniCislo>75848899</IdentifikacniCislo>
<PlatnostZaznamu>0</PlatnostZaznamu>
</Kriteria>
</VerejnyWebDotaz>";
var doc = XDocument.Parse(xml);//basically to validate XML
var stream = new MemoryStream();
doc.Save(stream);
stream.Position = 0;
return stream;
}
Was gradually able to whittle down the headers needed for a successful request after find the match that worked. Try removing others gradually to test if more can be removed safely to reduce the amount of unnecessary code needed.
private static HttpClient createHttpClient(string baseAddress) {
var handler = createHandler();
var client = new HttpClient(handler);
client.BaseAddress = new Uri(baseAddress);
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.TryAddWithoutValidation("User-Agent", "System.Net.Http.HttpClient");
client.DefaultRequestHeaders.TryAddWithoutValidation("Accept-Language", "en-US,en;q=0.9");
client.DefaultRequestHeaders.TryAddWithoutValidation("Accept", "text/xml,application/xml");
client.DefaultRequestHeaders.ExpectContinue = false;
client.DefaultRequestHeaders.ConnectionClose = false;
client.DefaultRequestHeaders.CacheControl = new CacheControlHeaderValue() {
MaxAge = TimeSpan.FromSeconds(0)
};
return client;
}
private static HttpClientHandler createHandler() {
var handler = new HttpClientHandler();
// if the framework supports automatic decompression set automatic decompression
if (handler.SupportsAutomaticDecompression) {
handler.AutomaticDecompression = System.Net.DecompressionMethods.GZip |
System.Net.DecompressionMethods.Deflate;
}
return handler;
}
While I chose to use the asynchronous API of System.Net.Http, I found a similar question
Reference UploadFile with POST values by WebClient
With an answer that was done using WebClient that could be adapted to your question so that a request can be constructed similar to what was produced above.
I tried testing that one as well but got into the same forbidden error. Now that the correct format is know you should also be able to correctly craft a working request using WebClient/WebRequest

Web API: multipart/form-data post not working

I've read every internet post on this and I still don't know what I'm doing wrong. From Fiddler, I'm trying to post multiple items and it's not working. I'm not able to post a .jpeg, but a .txt is no working. First requests to the method are successful, but all subsequent requests aren't and will generate "Unexpected end of MIME multipart stream. MIME multipart message is not complete" or "An asynchronous module or handler completed while an asynchronous operation was still pending". If I just send text entries in the parts/boundaries, it will only see the first one - the others are ignored
Web API
[HttpPost]
[MimeMultipart]
public async Task<IHttpActionResult> PostAsync()
{
var uploadPath = HttpContext.Current.Server.MapPath("~/Uploads");
Directory.CreateDirectory(uploadPath);
var provider = new MultipartFormDataStreamProvider(uploadPath);
// THIS IS WHERE THE ERRORS HAPPEN
await Request.Content.ReadAsMultipartAsync(provider);
// NEVER HAS A NAME(S),, EVEN WHEN NO ERROR
var localFileName = provider.FileData.Select(multiPartData => multiPartData.LocalFileName).FirstOrDefault();
foreach (var stream in provider.Contents)
{
var fileBytes = stream.ReadAsByteArrayAsync();
var response = Request.CreateResponse(HttpStatusCode.OK);
response.Content = new StringContent("Successful upload", Encoding.UTF8, "text/plain");
response.Content.Headers.ContentType = new MediaTypeWithQualityHeaderValue(#"text/html");
}
// omitted code
Fidder Headers
Content-Type: multipart/form-data; boundary=-------------------------acebdf13572468
User-Agent: Fiddler
Host: localhost:1703
Content-Length: 3199
Fiddler Body
---------------------------acebdf13572468
Content-Disposition: form-data; name="description"
Content-Type: text
the_text_is_here
---------------------------acebdf13572468
Content-Disposition: form-data; name="fieldNameHere"; filename="today.txt"
Content-Type: text/plain
<#INCLUDE *C:\Users\loser\Desktop\today.txt*#>
---------------------------acebdf13572468-
Should I use a different content-type? I'm dying over here.

Batch request - Dynamics CRM

All,
I am trying to implement a batch request to Dynamics CRM with the following source code:
public async Task<HttpResponseMessage> HttpPatchCrmApi(string resource, string data)
{
string uniq = Guid.NewGuid().ToString();
MultipartContent content = new MultipartContent("mixed", "batch_" + uniq);
HttpRequestMessage batchRequest = new HttpRequestMessage(HttpMethod.Post, CrmBaseUrl + "/api/data/v8.0/$batch");
HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Post, CrmBaseUrl + resource);
request.Content = new StringContent(data, Encoding.UTF8, "application/json");
HttpMessageContent query = new HttpMessageContent(request);
content.Add(query);
batchRequest.Content = content;
HttpResponseMessage response = await RbWebApi.SendAsync(batchRequest);
return response;
}
The problem is that I am getting "400 Bad request"
EDIT:
As suggested in the comments here is the stack trace of the request from fiddler:
POST https://Hidden.api.crm4.dynamics.com/api/data/v8.0/$batch HTTP/1.1
Authorization: Bearer eyJ0eXAiOiJKV.... very long string
Content-Type: multipart/mixed; boundary="batch_7b6e3c60-1284-4958-a39a-4653af21833c"
Host: Hidden.api.crm4.dynamics.com
Content-Length: 313
Expect: 100-continue
--batch_7b6e3c60-1284-4958-a39a-4653af21833c
Content-Type: application/http; msgtype=request
POST /api/data/v8.0/my_recurringgifts HTTP/1.1
Host: Hidden.api.crm4.dynamics.com
Content-Type: application/json; charset=utf-8
{"my_name":"slavi"}
--batch_7b6e3c60-1284-4958-a39a-4653af21833c--
While writing the code I was inspiring myself from here and here

I think your request is wrong.
You must build the request Body EXACTLY like defined by Microsoft
This means the Blank lines must be there at the right place all the attributes must exist in the body (like "--changeset_XXX" for example) and as I see you dont meet this requirements.
I just build a Request in Postman against my CRM and it worked:
URL
https://yourTenant.api.crm.dynamics.com/api/data/v8.0/$batch
Headers
OData-MaxVersion:4.0
OData-Version:4.0
Accept:application/json
Authorization:Bearer aVeryLongStringTokenHere
Content-Type: multipart/mixed;boundary=batch_1234567
Body
--batch_1234567
Content-Type:multipart/mixed;boundary=changeset_555666
--changeset_555666
Content-Type:application/http
Content-Transfer-Encoding:binary
Content-ID:1
POST https://yourTenant.api.crm.dynamics.com/api/data/v8.0/accounts HTTP/1.1
Content-Type:application/json;type=entry
{name: 'BatchJobTest788'}
--changeset_555666
Content-Type:application/http
Content-Transfer-Encoding:binary
Content-ID:2
POST https://yourTenant.api.crm.dynamics.com/api/data/v8.0/accounts HTTP/1.1
Content-Type:application/json;type=entry
{new_name: 'BatchJobTest348'}
--changeset_555666--
--batch_1234567--
Additional Remarks:
The Content-Type of your Header holds your BatchId
The Content-Type of your Batch holds your ChangesetId (if it is a change to data)
Before starting to programm REST calls try to define them in a REST tool like POSTMAN and make them work. Then build the working request in your code.
Here a good explanation-source for the batching in CRM

Sending JSON in POST from WP8 to AppEngine - Error 500

Hi I'm trying to send the following JSON string to AppEngine server. The string looks as following:
{"param2":50.0,"param1":50.0,"additionalParams":{"param3":"123","userID":"1234561"}}
And the code that I use for sending it is below:
public async Task<string> SendJSONData(string urlToCall, string JSONData)
{
// server to POST to
string url = urlToCall;
// HTTP web request
var httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
httpWebRequest.ContentType = "action";
httpWebRequest.Method = "POST";
// Write the request Asynchronously
using (var stream = await Task.Factory.FromAsync<Stream>(httpWebRequest.BeginGetRequestStream,
httpWebRequest.EndGetRequestStream, null))
{
//create some json string
string json = JSONData;
// convert json to byte array
byte[] jsonAsBytes = Encoding.UTF8.GetBytes(json);
// Write the bytes to the stream
await stream.WriteAsync(jsonAsBytes, 0, jsonAsBytes.Length);
}
WebResponse response = await httpWebRequest.GetResponseAsync();
StreamReader requestReader = new StreamReader(response.GetResponseStream());
String webResponse = requestReader.ReadToEnd();
return webResponse;
}
I've sniffed what is being sent to the server, using Fiddler:
POST http://server.appspot.com/method HTTP/1.1
Accept: */*
Content-Length: 85
Accept-Encoding: identity
Content-Type: action
User-Agent: NativeHost
Host: server.appspot.com
Connection: Keep-Alive
Cache-Control: no-cache
Pragma: no-cache
{"param2":50.0,"param1":50.0,"additionalParams":{"param3":"123","userID":"1234561"}}
Please mind that I've expeimented with "Content-Type" parameter, setting it to both "text/plain" and "application/json".
Still the answer from the server looks like this:
HTTP/1.1 500 Internal Server Error
Date: Wed, 20 Feb 2013 18:54:34 GMT
Content-Type: text/html; charset=UTF-8
Server: Google Frontend
Content-Length: 466
<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>500 Server Error</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Server Error</h1>
<h2>The server encountered an error and could not complete your request.<p>If the problem persists, please report your problem and mention this error message and the query that caused it.</h2>
What should I do, to receive the desired "OK" response?

Ok so the problem was the lack of "action" parameter in my POST.
Workaround looks like this:
// Write the request Asynchronously
using (var stream = await Task.Factory.FromAsync<Stream>
(httpWebRequest.BeginGetRequestStream,httpWebRequest.EndGetRequestStream, null))
{
//create some json string
string json = "action="+JSONData;
// convert json to byte array
byte[] jsonAsBytes = Encoding.UTF8.GetBytes(json);
// Write the bytes to the stream
await stream.WriteAsync(jsonAsBytes, 0, jsonAsBytes.Length);
}