I have an application that uploads videos to YouTube on behalf of our customers. We manage thousands of channels and upload thousands of videos a day. This has been working for years. The code is written in C#, .Net 4.7 (it was actually using a slightly older version, but I had to update it in order to recompile and test). It uses the REST API (not the SDK).
For some reason, the code broke the evening of Friday, Oct 12th, 2018. All of our requests to upload return with a 400 status and "Malformed multipart body." as the body.
We have not changed the code in at least 6 months (though likely over a year). I am able to reproduce the error on my dev machine. I looked at the raw http request in Fiddler and I don't see anything wrong with it, though I'm having difficulty finding the plain REST documentation to confirm the content-disposition headers. I know the JSON and video are fine (I validated the JSON and verified I'm able to upload the video directly to YouTube). Also, all of the other API calls we make work just fine. It's just uploading that is a problem.
Here's a copy of the HTTP request from Fiddler...
POST https://www.googleapis.com/upload/youtube/v3/videos?part=snippet,status HTTP/1.1
Authorization: Bearer <token>
Content-Type: multipart/form-data; boundary="590ce98e-6411-4e49-8dde-d7aa06cb067d"
Host: www.googleapis.com
Content-Length: 8305362
Expect: 100-continue
--590ce98e-6411-4e49-8dde-d7aa06cb067d
Content-Type: application/json; charset=utf-8
Content-Disposition: form-data; name=json; filename=file.json; filename*=utf-8''file.json
{"snippet":{"title":"77 Wakefield Street, Bald Hills, QLD, 4036","description":"77 Wakefield Street\r\n\r\nContact Brian Brewder for more information.\nTourFactory Corporate Headquarters\n123-456-7890\n\r\n","categoryId":19,"tags":["Tag1","Tag2"]},"status":{"privacyStatus":"public","embeddable":true}}
--590ce98e-6411-4e49-8dde-d7aa06cb067d
Content-Disposition: form-data; name="files"; filename="video"
Content-Type: video/x-msvideo
<video>
It seems like something must have changed on the YouTube side of the API, but I checked the blog and don't see anything listed. Since Google uses StackOverflow for support, I'm hoping somebody can help me identify the problem.
So, apparently what caused the issue (And still does) is that now the
Content-Type: multipart/mixed;
is no longer accepted. I managed to get an error message by changing some stuff around, and said to only send video/* or application/octet-stream. Unfortunately, while i still got OK from the server by doing so, resulted in a broken youtube video
At this point i tried playing with headers but with no success. The file would be accepted by YouTube but wouldn't be displayed correctly. I suspect they changed - or dropped - the non-resumable uploads API. If they did, i can't find an announcement on that.
So, my "solution" - well, a workaround - was to reimplement my upload function using the Resumable Upload protocol.
https://developers.google.com/youtube/v3/guides/using_resumable_upload_protocol
This works and didn't take too long. Just make sure to set both X-Upload-Content-Length and x-upload-content-type for BOTH requests (metadata first and payload after getting the upload put URL).
So you first POST the same metadata (snippet and status) and get the PUT url
QByteArray responseLocation = reply->rawHeader( "Location" );
and then create the PUT request using the same X-Upload-Content-Length and x-upload-content-type
newrequest.setRawHeader( "X-Upload-Content-Length", QByteArray::number( video->size() ) );
newrequest.setRawHeader( "x-upload-content-type", "video/*" );
That worked for me - I hope this helps!
Related
We're building an ASP.NET web application which integrates with a DAM (digital asset management) system. The DAM stores files in Azure blob storage and makes them available publicly using Azure CDN.
We will be making these files (most are PDFs) available to download from our web app. When a user requests one of these files we will provide a custom URL which will run some code on the server (logging the download etc) before returning the relevant file for download.
The client requires that the file is always returned as a browser attachment (i.e. content disposition attachment header). I am curious about what options I have here.
My ideal would be that the CDN URL is abstracted and my custom URL is the public URL for the file. That would allow me to set relevant response headers etc. However, I assume the only solution here would be to download the file from CDN and cache it on my web server which would obfuscate the CDN's purpose. So presumably I have to redirect the client to the CDN public URL once I've done my server processing. But then is there a way I can ensure the file is returned by Azure with the correct response headers to ensure the browser's default download behaviour is delegated?
* Update *
In seeing the answers to this question I realised I was perhaps asking the wrong question. Thank you to those of you who answered here. Follow-up question is here.
TL;DR
You need to configure the default version on the blob storage in order for it to show the required header to non-authenticated clients. the question in this question has the code to make it work.
Once this is set, and working for anonymous clients the CDN will copy all of the headers across and it should work as expected.
Setting ContentDisposition
The functionality is present, you can set ContentDisposition on a blob property However, while this will set the property on the blob, it does not pass through to the header.
I tested this with Powershell using the following (just because its quicker than c#)
$context = New-AzureStorageContext -StorageAccountName $StorageAccountName -StorageAccountKey $StorageAccountKey
$container = Get-AzureStorageContainer -Name $ContainerName -Context $context
$blobref = ($script:container.CloudBlobContainer.GetBlobReferenceFromServer("images/pier.jpg"))
$blobref.Properties
$blobref.Properties.ContentDisposition = 'attachment; filename="fname.ext"'
$blobref.SetProperties()
$blobref = ($script:container.CloudBlobContainer.GetBlobReferenceFromServer("images/pier.jpg"))
$blobref.Properties
Which produces (amongst others)
ContentDisposition : attachment; filename="fname.ext"
However nothing is set when the headers are queried
([system.Net.HttpWebRequest]::Create($blobref.Uri.AbsoluteUri)).getresponse()
(to answer comment,. these are the headers returned - while experimenting I also tried with and without a contenttype - hence it being blank here)
IsMutuallyAuthenticated : False
Cookies : {}
Headers : {x-ms-request-id, x-ms-version, x-ms-lease-status, x-ms-blob-type...}
SupportsHeaders : True
ContentLength : 142224
ContentEncoding :
ContentType :
CharacterSet :
Server : Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
LastModified : 01/03/2016 11:29:04
StatusCode : OK
StatusDescription : OK
ProtocolVersion : 1.1
ResponseUri : https://xxxx.blob.core.windows.net/cdn/images/pier.jpg
Method : GET
IsFromCache : False
And since CDN will only copy the information from the HTTP headers themselves, this data isn't making it into CDN.
edited (after extended comment chatter!)
For reasons best known to itself Powershell wasn't sending the x-ms-version, so I fell back to telnet which did indeed produce the header -
HEAD /cdn/images/pier.jpg HTTP/1.1
HOST: xxxx.blob.core.windows.net
x-ms-version: 2015-04-05
HTTP/1.1 200 OK
Content-Length: 142224
Last-Modified: Tue, 01 Mar 2016 11:29:04 GMT
Accept-Ranges: bytes
ETag: "0x8D341C4B1C4F34F"
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: b4f41b01-0001-00d7-7cc9-7384c9000000
x-ms-version: 2015-04-05
x-ms-lease-status: unlocked
x-ms-lease-state: available
x-ms-blob-type: BlockBlob
Content-Disposition: attachment; filename="fname.ext"
Date: Tue, 01 Mar 2016 14:49:17 GMT
I am trying to store a JSON response from the following url in a string. Most times I run the code, the last several characters of the JSON response are cut off. When you visit the url in a browser, the full JSON response is shown. Sometimes my code works, but most times it returns partial content.
If I reduce the length of the querysting in the url I am requesting, I get more content back and closer to the full response, which is very strange. If I remove the querystring entirely, usually the full JSON response is returned. The problem is I DO want to keep the querystring as is.
I tried all sort of alternatives including changing the encoding, using HttpWebRequest/HttpWebResponse, copying the response into a MemoryStream, reading the response using a byte buffer, changing the protocol version, etc.
Any ideas on how to get the full response and can explain what is happening? Thanks!
System.Net.WebClient wc = new System.Net.WebClient();
string data = wc.DownloadString("http://static.arcgis.com/attribution/World_Topo_Map?f=json&callback=dojo.io.script.jsonp_dojoIoScript19._jsonpCallback");
Console.Write(data);// String should end with ",-119.2]}]}]});"
Console.Read();
It's a server problem, but I luckily I believe there's a workaround. The server is not returning correct uncompressed results. If you change your client code to accept compressed responses, the problem seems to go away.
What's not clear is why uncompressed responses are broken. It's possible that the server has trouble with responses over 64K (this json is 64.8K). It's possible that the server is has a bug in its length calculations that only shows up in some circumstances or for some maps. I'm not sure. So there's no guarantee that the compression workaround will work in all cases, and I'd recommend testing on your end. And following up the ArcGIS folks so they know about this bug.
Here's the decompression code (adapted from https://stackoverflow.com/a/4567408/126352) that I got to return the correct JSON.
public class GZipWebClient : WebClient
{
protected override WebRequest GetWebRequest(Uri address)
{
HttpWebRequest request = (HttpWebRequest)base.GetWebRequest(address);
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
return request;
}
}
GZipWebClient wc = new GZipWebClient();
string data = wc.DownloadString("http://static.arcgis.com/attribution/World_Topo_Map?f=json&callback=dojo.io.script.jsonp_dojoIoScript19._jsonpCallback");
Here's how I discovered this workaround:
first I reproduced the problem in code
then I changed the request to HTTP 1.0 (which doesn't have chunked encoding or compression) and noticed that the Content-Length header returned by the server was 66338. But the actual content is longer than that. So the server is telling the client that the content is shorter than it actually is, which explains why your WebClient response was truncated.
then I tried to handle chunked encoding manually with the client, but that didn't work-- I got the same truncation.
at that point I wondered if it was an absolute length issue, e.g. the server handled shorter responses OK but underestimated the length of longer ones.
so I used Fiddler to play around with the request headers, and I found that removing the compression header (Accept-Encoding:gzip,deflate,sdch) would reproduce the problem.
this explained why this request works in a browser (which uses compression) but not in your code.
so I tried a compression-aware C# client and everything worked!
Unfortunately, this doesn't tell you the root cause, so the compression workaround may or may not work in all cases. It may break on longer responses. It may break on different maps. Test it to make sure!
It also doesn't explain why #ssut wasn't able to repro. It's possible that only some servers in ArgGIS's cluster are vulnerable? I dunno.
BTW, if you want to report the bug to ArcGIS, here's an HTTP request they can use to repro the problem:
GET http://static.arcgis.com/attribution/World_Topo_Map?f=json&callback=dojo.io.script.jsonp_dojoIoScript19._jsonpCallback HTTP/1.0
Accept: text/html, application/xhtml+xml, */*
Accept-Language: en-US
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
Host: static.arcgis.com
I am tested using your code and this code is working correctly.
p.s. you don't need &callback parameter when you use for just parse json. (callback parameter is need for jsonp callback on javascript)
Leaving an answer because I don't have enough reputation to leave a comment on your answer...
I think this is a bug in mono on the client side, not a server bug. I've been tracking this down in some code I'm using. It works perfectly on Windows using .Net. It only fails when running the code with mono. When I use your GZipWebClient, it runs perfectly on mono as well.
I am trying to handle a website programmatically. Lets say I visit the page www.example.com/something. On the website there is a button which I am pressing. The code of the button looks something like this:
<form action="/something" method="POST" enctype="text/plain">
<input type="submit" class="button" value="Click me" >
</form>
Pressing this button updates the information on the website.
Now I would like to do this procedure programatically to receive the content of the updated website after pressing the button.
Can someone lead me to the right direction on how to do this? preferably in C#.
Thank you in advance!
Edit:
I used Fiddler to capture the HTTP request and response, it looks like this:
POST /something HTTP/1.1
Host: example.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Referer: http://example.com/something
Cookie: cookie1=cookiecontent; cookie2=cookiecontent
Connection: keep-alive
Content-Type: text/plain
Content-Length: 0
HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Encoding: gzip
Vary: Accept-Encoding
Server: Microsoft-IIS/8.0
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Thu, 05 Dec 2013 23:36:31 GMT
Content-Length: 2202
Although the requests includes cookies they don't appear to be relevant. I decompressed the received content with fiddler and found the wanted data to be included in the response.
I am not very experienced in HTTP requests and am therefore hoping that someone can help me convertion this into a C# http request to receive the content.
If the website in question is open and doesn't do any sort of cookie generation to validate requests (there are plenty of sites like this) then you can just use System.Net.WebRequest or similar to post the required form data, then examine the response. See this MSDN page for an example.
If the page does use cookies and so on you'll have to get a bit more creative. In some cases you can issue one web request to get the first page, examine the results for cookies and hidden form values and use those in your POST.
If all else fails then the Selenium WebDriver library will give you almost complete browser emulation with full access to the DOM. It's a bit more complex than using a WebRequest, but will work for pretty much everything you can use a web browser for.
Regardless of which method you use, Fiddler is a good debugging tool. Use it to compare what your C# code is doing to what the web browser is doing to see if there's anything your code isn't getting right.
Since it's a submit button then simulating the resulting HTTP Request would be easier than simulating a click. First, I would use a program like Fiddler to inspect what is being sent when you submit the form. Then I would replicate that request, just changing the values that I need changing, using HTTPWebRequest. You can find an example here.
The resultant HTTPWebResponse can then be parsed for data. Using something like HtmlAgilityPack makes that part easier.
You can do what you want with http://www.seleniumhq.org/projects/webdriver/. It is possible to do web automation with c# in a console program. I am using it for ui integration testing and it works fairly well
I would look into searching for a browser automation framework. I would usually do this in Python and have not used .Net for this, but a quick Google search yields quite a few results.
Included within these:
http://watin.org/
Web automation using .NET
Can we script and automate a browser, preferably with .Net?
I'm trying to hunt down why a POST request from a C# script isn't working, when the same request works fine in Python. I want to be able to have all the data sent by the script, and the response from the sever, to be displayed on the screen so that I can work out what the difference is between what the C# and Python scripts are sending.
In Python I can do this with the standard httplib2 library by just using:
httplib2.debuglevel = 1
This produces the following output (as an example):
reply: 'HTTP/1.1 201 Created\r\n'
header: Date: Tue, 05 Oct 2010 09:25:42 GMT
header: Server: Apache/2.2.9 (Debian) PHP/5.2.6-1+lenny9 with Suhosin-Patch
header: X-Powered-By: PHP/5.2.6-1+lenny9
header: Location: http://example.org/api/2
header: Content-Length: 0
header: Content-Type: text/html
send: 'GET /api/2 HTTP/1.1\r\nHost: example.org\r\naccept-encoding: gzip, deflate\r\nuser-agent: Python-httplib2/$Rev$\r\n\r\n'
Is there a way to produce similar output in C# using the HttpWebRequest class?
I've seen mention of Fiddler in another question, but I'm running Linux and Fiddler appears to be for Windows only.
You can spin through resp.Headers.AllKeys and then dump the key and its value, though there is sometimes a degree of translation going on (most obviously when it is doing auto-redirect-following).
You can use System.Net tracing.
You can use ethereal which has a linux version. I don't use it for such things these days as Fiddler is indeed handier for such cases, but I used to use ethereal in the past, and sometimes seeing what is actually on the wire rather than what the code is saying is the best way to go (as you aren't depending on possibly buggy code to tell you if the code is buggy).
Yes you can and I have used it successfully before. If you are using C# I imagine there is a config file that you acn turn System.Net tracing on. I am not sure if Mono has this - if you are using Mono.
Have a look here:
http://msdn.microsoft.com/en-us/library/ty48b824.aspx
I have a website that works perfectly under built-for specification but with the increased use of SIM card dongles I’ve (I won’t be the first to be pulling his hair out over this) encountered a site-killing issue... (ok ok ok... a days’ work perhaps)
When the mobile provider gets a request for a file it's passed through a very clever application which essentially strips it down to 'just' functional. This is returned and the file works as expected even though it has been modified slightly (or heavily in the case of images).
The clever software which does this re-writes the request header also, which is causing the serialiser to ignore the response as it's not of the correct content type... I think anyway.
normal request : Content-Type application/json; charset=utf-8
SIM proxy request : Content-Type text/xml; charset=utf-8
issue number one with this is the request requires a verb in the web.config enabling GET/POST (easy added)
issue number two with this, the GET request is built like so www.site.com?a.asmx/function?value="a value" the JSON serialiser understand this is a string type so it removes the appended quotation marks and accepts the parameter, when the serialiser isn't used it will actually add more quotation marks "\"a value\"". (Easy fixed)
Issue number three the return value is XML and not JSON. Requires allot of work but can be fixed should i not be able to find another way.
I've just been reading the W3 for the HTTP header 'Cache-Control' and apparently I can send 'no-transform' and (assuming they conform) the proxy will ignore it. That would be fantastic.
Question is... How do I change the header value of a GET request via Sys.Net.WebServiceProxy.invoke()?
Also has any one any experience with this? the internet reveals little discussion on the topic.
Many Thanks in advance of any responses and sorry for the wall of text for a one line question... just fingers crossed someone has encountered this before.
There is one work around.
Edit the file where Sys.Net.WebServiceProxy.invoke function is kept.
and put following line :
request.get_headers()['Cache-Control'] = 'no-transform';
after
var request = new Sys.Net.WebRequest();