Develop Reference

How to download and decompress file from Ebay API

I am attempting to get a file from Ebays API
https://developer.ebay.com/api-docs/sell/feed/resources/task/methods/getResultFile#h2-samples
I am getting data back but it is not decompressing properly or encoding properly. It should be an gziped xml file. The documentation is not very clear on this actually. I am using RestSharp for my http calls (106.15.0).
Exception:
Found invalid data while decoding
My Code:
const string url = "sell/feed/v1/task/task-16-SOMENUMBER/download_result_file";
var restClient = new RestClient(_restApiUrl)
{
Authenticator = new OAuth2AuthorizationRequestHeaderAuthenticator(authentication.AuthorizationToken, "Bearer")
};
var httpRequest = new RestRequest(url, Method.GET);
httpRequest.AddHeader("Accept-Encoding", "application/gzip");
httpRequest.AddHeader("Accept", "*/*");
byte[] myfile = restClient.DownloadData(httpRequest);
var decodedString = Encoding.UTF8.GetString(myfile);
using (var stream = new MemoryStream(myfile))
{
string res;
using (GZipStream zipStream = new GZipStream(stream, CompressionMode.Decompress))
{
using (var sr = new StreamReader(zipStream))
{
res = sr.ReadToEnd(); //ERROR HERE: Found invalid data while decoding
}
}
var result = res;
}
First 30 of returned string (Encoding.Default.GetString(myfile))
PK\u0003\u0004\u0014\0\b\b\b\0\f£†T\0\0\0\0\0\0\0\0\0\0\0\0>\0\0\0ActiveInventoryReport
Hex
50-4B-03-04-14-00-08-08-08-00-0C-A3-86-54-00-00-00-00-00-00-00-00-00-00-00-00-3E-00-00-00-41-63-74-69-76-65-49-6E-76-65-6E-74-6F-72-79-52-65-70-6F-72-74-2D-41-70-72-2D-30-36-2D-32-30-32-32-2D-32-30-3A-32-34-3A-32-30-2D-30-37-30-30-2D-31-33-33-34-39-39-38-35-32-34-2E-78-6D-6C-BD-9D-5B-53-9B-47-B6-86-AF-77-7E-45-CA-F7-32-7D-3E-4C-79-3C-25-09-03-1E-C0-D6-48-38-60-DF-B1-8D-F6-98-0A-01-17-86-4C-3C-BF-7E-F7-E1-

Unsupported decompress method error using GzipStream.Read in c#

I get the following error: The archive entry was compressed using an unsupported compression method.
I got to decode the following gzip compressed base64 string, here is the string:
H4sIAAAAAAAAAD1SwW6bQBAdO0mDrUq9VGoPPWzVSj1ZItiO7aNjk4TI4IRgY7gtMLZxFnBhiYM/oLee+wn8QL+AT+mHVF1y6F5WM+/N07yZaQO0oBG2AaDRhGYYNH424GyS5DFvtOGE000LTjH2t1C/E2jdhgFeM7rJRPi3De3Hp5yx+SHGVIKmFsBX2R8gri96nXXf9zpduT/seArKHcWnA2WoKD30B6LuPk32mPIQsxZIHF94nmL22oYEZ0vKcoTfWNzJ7morB6s75hfapYitR5nNtd1+oMXLwptol1ok8Nur4zwcPgc3y15wuyzclZ57Nstd2ygc25VnUZ8Fk9F/rdnRL3RLlR1rc9CPi64xdY5utCjc3aLvWnrfUK63zo5FztFR3OlDT49UxbAfLvSdGc2nm65+81Dott4V8U63tsxRnK5h+eF6dTESDtpwHoTZntFCzG6WpCi9Jt9X5dDE73kojBKGz8iIIoMksldJwut5fqjKgU3ZUxh/I0lMUhrGXnLIPgvoi4DG+z0rCN+GGUnzGAlPiFdXkiQl6zxD+CRI/JAIYIN8iymhXNCRmHkc+vBWoPcYYMYpqyU/VuVlVbKZeqMa07HpkMn8UVctbSLBqUEjhE5VBn9+/SBV6ZuCS6sSw6qkcVV6XlWOEgEfxF/LI9GEw3fqC0/pmPM09HJer/OsbjQ7gXNzrBlXc7veL0j1ncGpuTBUgCa8mdKIblAcF/wDb9VytI4CAAA\u003d
When I first use the Convert.FromBase64String method I receive this string when I convert it into a string:
a"\u001f�\b\0\0\0\0\0\0\0=R�n�#\u0010\u001d;I��J�Tj\u000f=l�J=Y"؎��c�����c�-0�q\u0016pa��?����\t�#��O�T]r�^V3��Ӽ�i\u0003��\u0011�\u0001�фf\u00184~6�l��1o���M\vN1��P�\u0013h݆\u0001^3��D��\r�ǧ���!�T��\u0016�W�\u001f �/z�u��:]�?�x\n�\u001dŧ\u0003e�(=�\a��>M���\u0010�\u0016H\u001c_x�b�چ\u0004gK�r��X���j+\a�;�\u0017ڥ��G�͵�~���\u009bh�Z$�۫�<\u001c>\a7�^p�,ܕ�{6�]�(\u001cەgQ�\u0005��\u007f���/tK�\u001dksЏ��1u�n�(�ݢ�Zz�P��ΎE��Q��CO�TŰ\u001f.��\u0019ͧ��~�P��\u0015�N���Q��a��zu1\u0012\u000e�p\u001e�ٞ�B�n��(�&�W����y(�\u0012��Ȉ\"�$�WI��y~�ʁM�S\u0018\u007f#ILR\u001a�^r�>\v苀��=+\b߆\u0019I�\u0018\tO�WW�$%�<C�$H��\b�|�)�\Б�y\u001c��V��\u0018`�)�%?V�eU��z�\u001aӱ���QW-m"��A#�NU\u0006\u007f~� U雂K�\u0012ê�qUz^U�\u0012\u0001\u001f�_�#ф�w�\vO��4�r^��n4;�ss�\u0019Ws��/H�����0T�&��҈nP\u001c\u0017�\u0003o�r��\u0002\0\0"
could this have something to do with the problem?
Here is my code:
public static string Decompress(string input)
{
byte[] compressed = Convert.FromBase64String(input);
byte[] decompressed = Decompress(compressed);
return Encoding.UTF8.GetString(decompressed);
}
private static byte[] Decompress(byte[] input)
{
using (var source = new MemoryStream(input))
{
byte[] lengthBytes = new byte[4];
source.Read(lengthBytes, 0, 4);
var length = BitConverter.ToInt32(lengthBytes, 0);
using (var decompressionStream = new GZipStream(source,
CompressionMode.Decompress))
{
var result = new byte[length];
decompressionStream.Read(result, 0, length); Error: The archive entry was compressed using an unsupported compression method.
return result;
}
}
}

There is one little oddity in the base64 string, though it should not result in the error message you are getting. the \u003d should be replaced an equal sign (=), in order for the base64 decoding to work properly. (I can't tell if the string actually has those five characters at the end, or if it is just a representation of a string with an equal sign at the end. In the latter case, I don't know it wouldn't just show an equal sign as opposed to a unicode escaped representation of an equal sign.)
Otherwise, that base64 string decodes to a valid gzip stream that should decompress with no problem.

I solved the issue, I use the GZipStream.CopyTo to a MemoryStream in place of the read function. Here is the code if anyone would need it!
public static string Decompress(string value)
{
byte[] buffer = Convert.FromBase64String(value);
byte[] decompressed;
using (var inputStream = new MemoryStream(buffer))
{
using var outputStream = new MemoryStream();
using (var gzip = new GZipStream(inputStream, CompressionMode.Decompress, leaveOpen: true))
{
gzip.CopyTo(outputStream);
}
decompressed = outputStream.ToArray();
}
return Encoding.UTF8.GetString(decompressed);
}

How to read from Response.OutputStream in C#

I'm using ServiceStack to create a Service. In one of the methods I write some data in response's output stream like this: await response.OutputStream.WriteAsync(Consts.DATA, 0, Consts.DATA.Length); where data is a byte[]. Ok, now the problem is when using this service from a client and GET the HttpResponseMessage for the specified method, I don't know how to get the data out. I wan't to unit test that the actual response contains the data I passed when writing to output content.
(I've tried to ReadAsByteArray but it throws an exception. When reading as Stream or String I don't know how to make it a byte[] to compare it in the test with the excpected Consts.DATA byte array.)
I'm pretty new to the field, excuse my ignorance if i miss something. Any help is appreciated.
Thanks!

If you want to use the typed client, you can do (from the documentation):
As raw bytes:
byte[] responseBytes = client.Get<byte[]>("/poco/World");
var dto = responseBytes.FromUtf8Bytes().FromJson<PocoResponse>();
dto.Result //Hello, World
Or as a Stream:
using (Stream responseStream = client.Get<Stream>("/poco/World")) {
var dto = responseStream.ReadFully()
.FromUtf8Bytes()
.FromJson<PocoResponse>();
dto.Result //Hello, World
}
Or even access the populated HttpWebResponse object:
HttpWebResponse webResponse = client.Get<HttpWebResponse>("/poco/World");
webResponse.Headers["X-Response"] //World
using (var stream = webResponse.GetResponseStream())
using (var sr = new StreamReader(stream)) {
var dto = sr.ReadToEnd().FromJson<PocoResponse>();
dto.Result //Hello, World
}
You can also use untyped ServiceStack client to access raw response, it is described in the documentation with samples.

If all you need is load the data from an URL as a Stream and youre using a HttpClient you can just do this to get a stream back:
var result = await client.GetStreamAsync(URL);
I think you can also use GetResponseStream() on your HttpWebResponse.
var result = message.GetResponseStream();

Run your output stream through this static method:
public static byte[] ReadFullStream(Stream st)
{
var lockTaken = false;
try
{
Monitor.Enter(_lock, ref lockTaken);
var size = 0;
var continueRead = true;
var buffer = (byte[])Array.CreateInstance(typeof(byte), 0x10000);
using (MemoryStream ms = new MemoryStream())
{
while (continueRead)
{
size = st.Read(buffer, 0, buffer.Length);
if (size > 0)
{
ms.Write(buffer, 0, size);
}
else
{
continueRead = false;
}
}
return ms.ToArray();
}
}
finally
{
if (lockTaken) { Monitor.Exit(_lock); }
}
}
EDIT: forgot, you'll need this defined at class scope:
private static readonly object _lock = new object();

Read response stream as follow-
using (var reader = new System.IO.StreamReader(response.GetResponseStream()))
{
responseContent = reader.ReadToEnd();
if (response.StatusCode == HttpStatusCode.OK && response.StatusDescription.ToUpper() == "OK" && !string.IsNullOrEmpty(responseContent))
{
objFSLstGetAgent = JsonConvert.DeserializeObject<YourObjectClass>(responseContent);
}
}

Unzipping a Stream in C#

I'm working in C#, and I'm downloading for the internet a zip file with one XML file in it. and I wish to load this XML file. This is what I have so far:
byte[] data;
WebClient webClient = new WebClient();
try {
data = webClient.DownloadData(downloadUrl);
}
catch (Exception ex) {
Console.WriteLine("Error in DownloadData (Ex:{0})", ex.Message);
throw;
}
if (data == null) {
Console.WriteLine("Bulk data is null");
throw new Exception("Bulk data is null");
}
//Create the stream
MemoryStream stream = new MemoryStream(data);
XmlDocument document = new XmlDocument();
//Gzip
GZipStream gzipStream = new GZipStream(stream, CompressionMode.Decompress);
//Load report straight from the gzip stream
try {
document.Load(gzipStream);
}
catch (Exception ex) {
Console.WriteLine("Error in Load (Ex:{0})", ex.Message);
throw;
}
in document.Load I'm always getting the following exception:
The magic number in GZip header is not correct. Make sure you are passing in a GZip stream.
What I'm doing wrong?

Apparently SharpZipLib is now unmaintained and you probably want to avoid it:
https://stackoverflow.com/a/593030
In .NET 4.5 there is now built in support for zip files, so for your example it would be:
var data = new WebClient().DownloadData(downloadUrl);
//Create the stream
var stream = new MemoryStream(data);
var document = new XmlDocument();
//zip
var zipArchive = new ZipArchive(stream);
//Load report straight from the zip stream
document.Load(zipArchive.Entries[0].Open());

If you have a byte array that contains a zip archive with a single file, you can use the ZipArchive class to get an unzipped byte array with the file's data.
ZipArchive is contained in .NET 4.5, in the assembly System.IO.Compression.FileSystem (you need to reference it explicitly).
The following function, adapted from this answer, works for me:
public static byte[] UnzipSingleEntry(byte[] zipped)
{
using (var memoryStream = new MemoryStream(zipped))
{
using (var archive = new ZipArchive(memoryStream))
{
foreach (ZipArchiveEntry entry in archive.Entries)
{
using (var entryStream = entry.Open())
{
using (var reader = new BinaryReader(entryStream))
{
return reader.ReadBytes((int)entry.Length);
}
}
}
}
}
return null; // To quiet my compiler
}

I am using SharpZipLib and it's working great !
Below is a function that encapsulate the library
public static void Compress(FileInfo sourceFile, string destinationFileName,string destinationTempFileName)
{
Crc32 crc = new Crc32();
string zipFile = Path.Combine(sourceFile.Directory.FullName, destinationTempFileName);
zipFile = Path.ChangeExtension(zipFile, ZIP_EXTENSION);
using (FileStream fs = File.Create(zipFile))
{
using (ZipOutputStream zOut = new ZipOutputStream(fs))
{
zOut.SetLevel(9);
ZipEntry entry = new ZipEntry(ZipEntry.CleanName(destinationFileName));
entry.DateTime = DateTime.Now;
entry.ZipFileIndex = 1;
entry.Size = sourceFile.Length;
using (FileStream sourceStream = sourceFile.OpenRead())
{
crc.Reset();
long len = sourceFile.Length;
byte[] buffer = new byte[bufferSize];
while (len > 0)
{
int readSoFar = sourceStream.Read(buffer, 0, buffer.Length);
crc.Update(buffer, 0, readSoFar);
len -= readSoFar;
}
entry.Crc = crc.Value;
zOut.PutNextEntry(entry);
len = sourceStream.Length;
sourceStream.Seek(0, SeekOrigin.Begin);
while (len > 0)
{
int readSoFar = sourceStream.Read(buffer, 0, buffer.Length);
zOut.Write(buffer, 0, readSoFar);
len -= readSoFar;
}
}
zOut.Finish();
zOut.Close();
}
fs.Close();
}
}

As the others have mentioned GZip and Zip are not the same so you might need to use a zip library. I use a library called: DotNetZip - available from the below site:
http://dotnetzip.codeplex.com/

From GZipStream Class description:
Compressed GZipStream objects written to a file with an extension of .gz can be decompressed using many common compression tools; however, this class does not inherently provide functionality for adding files to or extracting files from .zip archives
So unless you control server-side files, I'd suggest looking for specific zip-targeted library (SharpZipLib for example).

How to read a WebClient response after posting data?

Behold the code:
using (var client = new WebClient())
{
using (var stream = client.OpenWrite("http://localhost/", "POST"))
{
stream.Write(post, 0, post.Length);
}
}
Now, how do I read the HTTP output?

It looks like you have a byte[] of data to post; in which case I expect you'll find it easier to use:
byte[] response = client.UploadData(address, post);
And if the response is text, something like:
string s = client.Encoding.GetString(response);
(or your choice of Encoding - perhaps Encoding.UTF8)

If you want to keep streams everywhere and avoid allocating huge arrays of bytes, which is good practise (for example, if you plan to post big files), you still can do it with a derived version of WebClient. Here is a sample code that does it.
using (var client = new WebClientWithResponse())
{
using (var stream = client.OpenWrite(myUrl))
{
// open a huge local file and send it
using (var file = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
file.CopyTo(stream);
}
}
// get response as an array of bytes. You'll need some encoding to convert to string, etc.
var bytes = client.Response;
}
And here is the customized WebClient:
public class WebClientWithResponse : WebClient
{
// we will store the response here. We could store it elsewhere if needed.
// This presumes the response is not a huge array...
public byte[] Response { get; private set; }
protected override WebResponse GetWebResponse(WebRequest request)
{
var response = base.GetWebResponse(request);
var httpResponse = response as HttpWebResponse;
if (httpResponse != null)
{
using (var stream = httpResponse.GetResponseStream())
{
using (var ms = new MemoryStream())
{
stream.CopyTo(ms);
Response = ms.ToArray();
}
}
}
return response;
}
}