Develop Reference

How to read numpy .npz file directly into C#

I have a load of data files in numpy .npz format written from python.
I want to read them directly into C# for a few reasons.
The data files contain a number of 1D arrays of different types - some will by byte arrays, and other double arrays.
Can anyone give me some advice on how to achieve this? Or otherwise what I might be doing wrong below?
I have tried using Accord.NET.NPZFormat but can't figure out how to make it work. I think probably because you have to give it a type to return, and because the arrays are of different types it fails.
Here is a link to it:
http://accord-framework.net/docs/html/M_Accord_IO_NpzFormat_Load__1.htm
I am struggling with syntax here, unsure of what to use as "T". The closest I have got is with the following, but doesn't seem to have any data in the result. Accord.IO has no example code.
public static void LoadNPZ(string zip_file, string npz_file)
{
byte[] ret = new byte[0];
using (ZipArchive zip = ZipFile.OpenRead(zip_file))
{
foreach (ZipArchiveEntry entry in zip.Entries)
{
if (entry.Name == npz_file + ".npz")
{
Stream fs = entry.Open();
ret = new byte[fs.Length];
fs.Read(ret, 0, (int)fs.Length);
}
}
}
if (ret.Length==0)
{
return;
}
var ret2 = NpzFormat.Load<object[]>(ret);
};

You can use the NumSharp library.
Let say you have this data created in Python.
import numpy as np
arr = np.array([1,2,3,4])
single = arr.astype(np.single)
double = arr.astype(np.double)
np.savez('single.npz', data=single)
np.savez('double.npz', data=double)
The C# code to read them is below.
using NumSharp;
var singleContent = np.Load_Npz<float[]>("single.npz"); // type is NpzDictionary
var singleArray = singleContent["data.npy"]; // type is float[]
var doubleContent = np.Load_Npz<double[]>("double.npz"); // type is NpzDictionary
var doubleArray = doubleContent["data.npy"]; // type is double[]
If you don't specify name for your array then the default name is arr_0, and the C# code would be like this.
var singleArray = singleContent["arr_0.npy"];
var doubleArray = doubleContent["arr_0.npy"];
Note that NumSharp has the following limitation.
The size of each dimension must be smaller than 2,147,483,591 bytes. Example: for integer (4 bytes), each dimension must have less than 536,870,898 elements.
If you are using .NET Framework, the maximum array size is 2GB (all dimensions considered). On 64-bit platform this limit can be avoided by enabling the gcAllowVeryLargeObjects flag.
More information can be found in this answer and this blog post (Disclaimer: I'm the author of both of them).

I work with C# and python quite a bit, and my reccomendation is to create a COM Server
http://timgolden.me.uk/pywin32-docs/html/com/win32com/HTML/QuickStartServerCom.html
then in python you could simply have something like
import numpy as np
class NPtoCSharp:
_reg_clsid_ = "{7CC9F362-486D-11D1-BB48-0000E838A65F}"
_public_methods_ = ['load_file']
_public_attrs_ = ['arr', 'the_file']
_reg_desc_ = "Python NPZ Loader"
_reg_progid_ = "NPtoCSharp"
def __init__(self):
self.arr = None
self.the_file = None
def load_file(self):
self.arr = np.load(self.the_file)
return self.arr
Then in C#
public void init_python()
{
Type NPtoCSharp = Type.GetTypeFromProgID("NPtoCSharp");
NPtoCSharpInst = Activator.CreateInstance(NPtoCSharp);
NPtoCSharpInst.the_file = 'myfile.npz';
}
Not complete but I hope you get the idea.

Determine if array contains all zeroes

I create the following array like this:
array<UInt16>^ temp = gcnew array<UInt16>(1000);
How do I determine if this entire array has been filled with zero or not.
I think I may be able to use TrueForAll(T) but I'm not sure.

var allElementsAreZero = temp.All(o => o == 0);
Simple as that.
It'll return when it finds one that doesn't satisfy the condition, so may not necessarily iterate through your whole collection:
"The enumeration of source is stopped as soon as the result can be determined."
https://msdn.microsoft.com/en-us/library/bb548541(v=vs.110).aspx

This should work properly (here I used LINQ):
IEnumerable<int> values = new List<int>(); // Or use any array type instead of List.
... Add your values here ...
var allAreZero = !values.Any(v => v != 0);
P.S. the array class inherits IEnumerable.
And here is a solution with foreach:
var isAllZero = true;
foreach (var value in values)
{
if (value != 0)
{
isAllZero = false;
break;
}
}
UPDATE
The really difference between TrueForAll, and my LINQ code is: LINQ code uses the fluent (or maybe also query) syntax, where TrueForAll is just a normal function where you send the array as a parameter.

initialize a counter from 0 then use for loop to interate through the array and increment the counter whenever it finds 0, and at the end compare the counter with size of array if its equal, it has all zeros

Reading the C++/CLI specification, it has been filled with
0s because you created it with a "new-expression" and the default value of the element type is 0.
24.2 CLI array creation
CLI array instances are created by new-expressions containing gcnew (§15.4.6) or …
Elements of CLI arrays created by new-expressions are always initialized to their default value.

TestComplete - How can I get the array elements that are stored in their 'var' object when using C#?

I'm trying to write simple automation test using TestComplete in C#.
(Not JScript/C# Script, just C#)
I'm using their libraries as you can see here:
https://support.smartbear.com/viewarticle/74747/
and specifically their 'var' type: https://support.smartbear.com/viewarticle/68791/
I'm trying to identify all the elements on the screen according to specific key and value, using the method "FindAll" (https://support.smartbear.com/viewarticle/71244/)
var a = someProcess["FindAll"]("text", "Simulate", 200, false);
In debug mode I can see that "a" has two encapsulated elements that he found and this line passes successfully.
The problem: I'm trying to get the first element, using the line
var b = a["0"];
and get a 'MissingMethodException'.
If I try to use
var b = a(0);
it says I'm trying to use variable as a function.
I couldn't find any method that can help me to get the elements.

Here is how you can process such an array in C#:
var font = Connect.Sys["Process"]("notepad")["Window"]("#32770", "Font", 1);
var a = font["FindAllChildren"]("WndClass", "ComboBox");
object[] elements = (a.UnWrap() as object[]);
for (int i = 0; i < elements.Count(); i++) {
var element = new var(elements[i]);
MessageBox.Show(element["FullName"].UnWrap() as string);
}

How do I initialize an empty array in C#?

Is it possible to create an empty array without specifying the size?
For example, I created:
String[] a = new String[5];
Can we create the above string array without the size?

If you are going to use a collection that you don't know the size of in advance, there are better options than arrays.
Use a List<string> instead - it will allow you to add as many items as you need and if you need to return an array, call ToArray() on the variable.
var listOfStrings = new List<string>();
// do stuff...
string[] arrayOfStrings = listOfStrings.ToArray();
If you must create an empty array you can do this:
string[] emptyStringArray = new string[0];

In .NET 4.6 the preferred way is to use a new method, Array.Empty:
String[] a = Array.Empty<string>();
The implementation is succinct, using how static members in generic classes behave in .Net:
public static T[] Empty<T>()
{
return EmptyArray<T>.Value;
}
// Useful in number of places that return an empty byte array to avoid
// unnecessary memory allocation.
internal static class EmptyArray<T>
{
public static readonly T[] Value = new T[0];
}
(code contract related code removed for clarity)
See also:
Array.Empty source code on Reference Source
Introduction to Array.Empty<T>()
Marc Gravell - Allocaction, Allocation, Allocation - my favorite post on tiny hidden allocations.

Try this:
string[] a = new string[] { };

You could inititialize it with a size of 0, but you will have to reinitialize it, when you know what the size is, as you cannot append to the array.
string[] a = new string[0];

There is not much point in declaring an array without size. An array is about size. When you declare an array of specific size, you specify the fixed number of slots available in a collection that can hold things, and accordingly memory is allocated. To add something to it, you will need to anyway reinitialize the existing array (even if you're resizing the array, see this thread). One of the rare cases where you would want to initialise an empty array would be to pass array as an argument.
If you want to define a collection when you do not know what size it could be of possibly, array is not your choice, but something like a List<T> or similar.
That said, the only way to declare an array without specifying size is to have an empty array of size 0. hemant and Alex Dn provides two ways. Another simpler alternative is to just:
string[] a = { };
[The elements inside the bracket should be implicitly convertible to type defined, for instance, string[] a = { "a", "b" };]
Or yet another:
var a = Enumerable.Empty<string>().ToArray();
Here is a more declarative way:
public static class Array<T>
{
public static T[] Empty()
{
return Empty(0);
}
public static T[] Empty(int size)
{
return new T[size];
}
}
Now you can call:
var a = Array<string>.Empty();
//or
var a = Array<string>.Empty(5);

string[] a = new string[0];
or short notation:
string[] a = { };
The preferred way now is:
var a = Array.Empty<string>();
I have written a short regular expression that you can use in Visual Studio if you want to replace zero-length allocations e.g. new string[0].
Use Find (search) in Visual Studio with Regular Expression option turned on:
new[ ][a-zA-Z0-9]+\[0\]
Now Find All or F3 (Find Next) and replace all with Array.Empty<…>() !

Simple and elegant!
string[] array = {}

You can define array size at runtime.
This will allow you to do whatever to dynamically compute the array's size. But, once defined the size is immutable.
Array a = Array.CreateInstance(typeof(string), 5);

I had tried:
string[] sample = new string[0];
But I could only insert one string into it, and then I got an exceptionOutOfBound error, so I just simply put a size for it, like
string[] sample = new string[100];
Or another way that work for me:
List<string> sample = new List<string>();
Assigning Value for list:
sample.Add(your input);

As I know you can't make array without size, but you can use
List<string> l = new List<string>()
and then l.ToArray().

Performance Rule CA1825: Avoid allocating zero-length arrays.
Rule discription:
Initializing a zero-length array leads to an unnecessary memory allocation. Instead, use the statically allocated empty array instance by calling the Array.Empty method.
In your case:
var a = Array.Empty<string>();

Combining #nawfal & #Kobi suggestions:
namespace Extensions
{
/// <summary> Useful in number of places that return an empty byte array to avoid unnecessary memory allocation. </summary>
public static class Array<T>
{
public static readonly T[] Empty = new T[0];
}
}
Usage example:
Array<string>.Empty
UPDATE 2019-05-14
(credits to #Jaider ty)
Better use .Net API:
public static T[] Empty<T> ();
https://learn.microsoft.com/en-us/dotnet/api/system.array.empty?view=netframework-4.8
Applies to:
.NET Core: 3.0 Preview 5 2.2 2.1 2.0 1.1 1.0
.NET Framework: 4.8 4.7.2 4.7.1 4.7 4.6.2 4.6.1 4.6
.NET Standard: 2.1 Preview 2.0 1.6 1.5 1.4 1.3
...
HTH

you can use the Array.Empty method (in .Net Core, at least)
string ToCsv(int[] myArr = null) { // null by default
// affect an empty array if the myArr is null
myArr ??= Array.Empty<int>();
//... do stuff
string csv = string.Join(",", myArr);
return csv;
}

You can do:
string[] a = { String.Empty };
Note: OP meant not having to specify a size, not make an array sizeless

Here is a real world example. In this it is necessary to initialize the array foundFiles first to zero length.
(As emphasized in other answers: This initializes not an element and especially not an element with index zero because that would mean the array had length 1. The array has zero length after this line!).
If the part = string[0] is omitted, there is a compiler error!
This is because of the catch block without rethrow. The C# compiler recognizes the code path, that the function Directory.GetFiles() can throw an Exception, so that the array could be uninitialized.
Before anyone says, not rethrowing the exception would be bad error handling: This is not true. Error handling has to fit the requirements.
In this case it is assumed that the program should continue in case of a directory which cannot be read, and not break- the best example is a function traversing through a directory structure. Here the error handling is just logging it. Of course this could be done better, e.g. collecting all directories with failed GetFiles(Dir) calls in a list, but this will lead too far here.
It is enough to state that avoiding throw is a valid scenario, and so the array has to be initialized to length zero. It would be enough to do this in the catch block, but this would be bad style.
The call to GetFiles(Dir) resizes the array.
string[] foundFiles= new string[0];
string dir = #"c:\";
try
{
foundFiles = Directory.GetFiles(dir); // Remark; Array is resized from length zero
}
// Please add appropriate Exception handling yourself
catch (IOException)
{
Console.WriteLine("Log: Warning! IOException while reading directory: " + dir);
// throw; // This would throw Exception to caller and avoid compiler error
}
foreach (string filename in foundFiles)
Console.WriteLine("Filename: " + filename);

turn javascript array into c# array

Hey. I have this javascript file that I'm getting off the web and it consists of basically several large javascript arrays. Since I'm a .net developer I'd like for this array to be accessible through c# so I'm wondering if there are any codeplex contributions or any other methods that I could use to turn the javascript array into a c# array that I could work with from my c# code.
like:
var roomarray = new Array(194);
var modulearray = new Array(2055);
var progarray = new Array(160);
var staffarray = new Array(3040);
var studsetarray = new Array(3221);
function PopulateFilter(strZoneOrDept, cbxFilter) {
var deptarray = new Array(111);
for (var i=0; i<deptarray.length; i++) {
deptarray[i] = new Array(1);
}
deptarray[0] [0] = "a/MPG - Master of Public Governance";
deptarray[0] [1] = "a/MPG - Master of Public Governance";
deptarray[1] [0] = "a/MBA_Flex MBA 1";
deptarray[1] [1] = "a/MBA_Flex MBA 1";
deptarray[2] [0] = "a/MBA_Flex MBA 2";
deptarray[2] [1] = "a/MBA_Flex MBA 2";
deptarray[3] [0] = "a/cand.oecon";
deptarray[3] [1] = "a/cand.oecon";
and so forth
This is what I'm thinking after overlooking the suggestions:
Retrieve the javascript file in my c# code by making an httprequest for it
paste it together with some code i made myself
from c# call an execute on a javascript function selfmade function that will turn the javascript array into json (with help from json.org/json2.js), and output it to a new file
retrieve the new file in c# parsing the json with the DataContractJsonSerializer resulting hopefully resulting in a c# array
does it sound doable to you guys?

I'm not in front of a computer with c# right now so I'm not able to fully try this.
What you're going to need to do #Jakob is the following:
Write a parser that will download the file and store it in memory.
For each section that you want to "parse" into a c# array (for example zonearray), you need to setup bounds to begin searching and end searching the file. Example: We know that zonearray starts building the array the two lines after zonearray[i] = new Array(1); and ends on zonearray.sort().
So with these bounds we can then zip through each line between and parse a C# array. This is simple enough I think that you can figure out. You'll need to keep track of sub-index as well remember.
Repeat this 2-3 for each array you want to parse (zonearray, roomarray..etc).
If you can't quite figure out how to code the bounds or how to parse the line and dump them into arrays, I might be able to write something tomorrow (even though it's a holiday here in Canada).
EDIT: It should be noted that you can't use some JSON parser for this; you have to write your own. It's not really that difficult to do, you just need to break it into small steps (first figure out how to zip through each line and find the right "bounds").
HTH
EDIT: I just spent ~20 minutes writing this up for you. It should parse the file and load each array into a List<string[]>. I've heavily commented it so you can see what's going on. If you have any questions, don't hesitate to ask. Cheers!
private class SearchBound
{
public string ArrayName { get; set; }
public int SubArrayLength { get; set; }
public string StartBound { get; set; }
public int StartOffset { get; set; }
public string EndBound { get; set; }
}
public static void Main(string[] args)
{
//
// NOTE: I used FireFox to determine the encoding that was used.
//
List<string> lines = new List<string>();
// Step 1 - Download the file and dump all the lines of the file to the list.
var request = WebRequest.Create("http://skema.ku.dk/life1011/js/filter.js");
using (var response = request.GetResponse())
using(var stream = response.GetResponseStream())
using(var reader = new StreamReader(stream, Encoding.GetEncoding("ISO-8859-1")))
{
string line = null;
while ((line = reader.ReadLine()) != null)
{
lines.Add(line.Trim());
}
Console.WriteLine("Download Complete.");
}
var deptArrayBounds = new SearchBound
{
ArrayName = "deptarray", // The name of the JS array.
SubArrayLength = 2, // In the JS, the sub array is defined as "new Array(X)" and should always be X+1 here.
StartBound = "deptarray[i] = new Array(1);",// The line that should *start* searching for the array values.
StartOffset = 1, // The StartBound + some number line to start searching the array values.
// For example: the next line might be a '}' so we'd want to skip that line.
EndBound = "deptarray.sort();" // The line to stop searching.
};
var zoneArrayBounds = new SearchBound
{
ArrayName = "zonearray",
SubArrayLength = 2,
StartBound = "zonearray[i] = new Array(1);",
StartOffset = 1,
EndBound = "zonearray.sort();"
};
var staffArrayBounds = new SearchBound
{
ArrayName = "staffarray",
SubArrayLength = 3,
StartBound = "staffarray[i] = new Array(2);",
StartOffset = 1,
EndBound = "staffarray.sort();"
};
List<string[]> deptArray = GetArrayValues(lines, deptArrayBounds);
List<string[]> zoneArray = GetArrayValues(lines, zoneArrayBounds);
List<string[]> staffArray = GetArrayValues(lines, staffArrayBounds);
// ... and so on ...
// You can then use deptArray, zoneArray etc where you want...
Console.WriteLine("Depts: " + deptArray.Count);
Console.WriteLine("Zones: " + zoneArray.Count);
Console.WriteLine("Staff: " + staffArray.Count);
Console.ReadKey();
}
private static List<string[]> GetArrayValues(List<string> lines, SearchBound bound)
{
List<string[]> values = new List<string[]>();
// Get the enumerator for the lines.
var enumerator = lines.GetEnumerator();
string line = null;
// Step 1 - Find the starting bound line.
while (enumerator.MoveNext() && (line = enumerator.Current) != bound.StartBound)
{
// Continue looping until we've found the start bound.
}
// Step 2 - Skip to the right offset (maybe skip a line that has a '}' ).
for (int i = 0; i <= bound.StartOffset; i++)
{
enumerator.MoveNext();
}
// Step 3 - Read each line of the array.
while ((line = enumerator.Current) != bound.EndBound)
{
string[] subArray = new string[bound.SubArrayLength];
// Read each sub-array value.
for (int i = 0; i < bound.SubArrayLength; i++)
{
// Matches everything that is between an equal sign then the value
// wrapped in quotes ending with a semi-colon.
var m = Regex.Matches(line, "^(.* = \")(.*)(\";)$");
// Get the matched value.
subArray[i] = m[0].Groups[2].Value;
// Move to the next sub-item if not the last sub-item.
if (i < bound.SubArrayLength - 1)
{
enumerator.MoveNext();
line = enumerator.Current;
}
}
// Add the sub-array to the list of values.
values.Add(subArray);
// Move to the next line.
if (!enumerator.MoveNext())
{
break;
}
}
return values;
}

If I understand your question right, you are asking whether you can execute JavaScript code from C#, and then pass the result (which in your example would be a JavaScript Array object) into C# code.
The answer is: Of course it’s theoretically possible, but you would need to have an actual JavaScript interpreter to execute the JavaScript. You’ll have to find one or write your own, but given that JavaScript is a full-blown programming language, and writing interpreters for such a large and full-featured programming language is quite an undertaking, I suspect that you won’t find a complete ready-made solution, nor will you be able to write one unless your dedication exceeds that of all other die-hard C#-and-JavaScript fans worldwide.
However, with a bit of trickery, you might be able to coerce an existing JavaScript interpreter to do what you want. For obvious reasons, all browsers have such an interpreter, including Internet Explorer, which you can access using the WinForms WebBrowser control. Thus, you could try the following:
Have your C# code generate an HTML file containing the JavaScript you downloaded plus some JavaScript that turns it into JSON (you appear to have already found something that does this) and outputs it in the browser.
Open that HTML file in the WebBrowser control, have it execute the JavaScript, and then read the contents of the website back, now that it contains the result of the executed JavaScript.
Turn the JSON into a C# array using DataContractJsonSerializer as you suggested.
This is a pretty roundabout way to do it, but it is the best I can think of.
I have to wonder, though, why you are retrieving a JavaScript file from the web in the first place. What generates this JavaScript file? Whatever generates it, surely could generate some properly readable stuff instead (e.g. an XML file)? If it is not generated but written by humans, then why is it written in JavaScript instead of XML, CSV, or some other data format? Hopefully with these thoughts you might be able to find a solution that doesn’t require JavaScript trickery like the above.

Easiest solution is to just execute the Javascript function that makes the array. Include there a function that makes it an JSON (http://www.json.org/js.html). After that make a XMLHttpRequest (AJAX) to the server and from there extract the JSON to a custom class.
If I may use jQuery, here's an example of the needed Javascript:
var myJSONText = JSON.stringify(deptarray);
(function($){
$.ajax({
type: "POST",
url: "some.aspx",
data: myJSONText,
success: function(msg){
alert( "Data Saved: " + msg );
}
});
})(jQuery);
Only now need some code to rip the JSON string to an C# Array.
EDIT:
After looking around a bit, I found Json.NET: http://json.codeplex.com/
There are also a lot of the same questions on Stackoverflow that ask the same.