Develop Reference

How to write millions of double values into a txt file

I've made a neural network and now I need to save the results of the training process into a local file. In total, there are 7,155,264 values. I've tried with a loop like this
string weightsString = "";
string biasesString = "";
for (int l = 1; l < layers.Length; l++)
{
for (int j = 0; j < layers[l].Length; j++)
{
for (int k = 0; k < layers[l - 1].Length; k++)
{
weightsString += weights[l][j, k] + "\n";
}
biasesString += biases[l][j] + "\n";
}
}
File.WriteAllText(#"path", weightsString + "\n" + biasesString);
But it literally takes forever to go through all of the values. Is there no way to write the contents directly without having to write them in a string first?
(Weights is a double[][,] while biases is a double[][])

First of writing down 7 million datasets will obviously take a lot of time.
I'd suggest you split up weights and biases into two files and write them on the fly, no need to store them all in memory until you are done.
using StreamWriter weigthStream = new("weigths.txt", append: true);
using StreamWriter biasStream = new("biases.txt", append: true);
for (int l = 1; l < layers.Length; l++)
{
for (int j = 0; j < layers[l].Length; j++)
{
for (int k = 0; k < layers[l - 1].Length; k++)
{
await weightStream.WriteLineAsync(weights[l][j, k]);
}
await biasStream.WriteLineAsync(biases[l][j]);
}
}

But it literally takes forever to go through all of the values. Is there no way to write the contents directly without having to write them in a string first?
One option would be to save it as binary data. This makes it much harder to read by humans, but for large amount of data this would really be preferable since it will save a lot of time both when reading and writing. For example using BinaryWriter and using unsafe code.
myBinaryWriter.Write(myArray.GetLength(0));
myBinaryWriter.Write(myArray.GetLength(1));
fixed (double* ptr = myArray)
{
var span = new ReadOnlySpan<byte>(ptr, myArray.GetLength(0) *myArray.GetLength(1) * 8);
myBinaryWriter.Write(span);
}
You might also consider using a binary serialization library like protbuf.net that can just take a object an and serialize it to a stream. Note that some libraries may need attributes to be added to classes and properties. Some libraries may also have issues with multidimensional and/or jagged arrays. Because of this it can sometimes be useful to define your own 2D array that uses a 1D array as the backing storage, this can make things like serialization or passing data to other components much simpler.
Another somewhat common practice is to store metadata, like height, width, etc in a simple human readable text-file using something like json or xml. While keeping the actual data in a separate raw binary file.

Bad variant - you can use json serialization
So-so variant - write in file immediately. Use File.AppendText
IMHO the best variant - use DB
IMHO good variant - use BinaryFormatter (you will not be able to read that by yourself, but application will)
Working variant - use StringBuilder

StringBuilder weightsSB = new StringBuilder();
StringBuilder biasesSB = new StringBuilder();
for (int l = 1; l < layers.Length; l++)
{
for (int j = 0; j < layers[l].Length; j++)
{
for (int k = 0; k < layers[l - 1].Length; k++)
{
weightsSB.Append(weights[l][j, k] + "\n");
}
biasesSB.Append(biases[l][j] + "\n");
}
}
As suggested in the comments, I used a StringBuilder instead. Works like a charm.

How can I specify a subset of a 1-D array as the send or receive buffer in MPI.NET for C#?

I am learning C# parallel programming using MPI.NET. In the example given below, I define a 1-D array (x) in each process and then I do some simple calculation on the corresponding part assigned to that process (i.e only the assigned part of x) to obtain its corresponding part in (y). My primary interest is to gather all these assigned parts (part of y calculated on each process) into the y-array on the root process to be able to finally calculate the sum. I mean, I want to copy each assigned part from all processes on the corresponding part on y-array located on root process.However, I could not do it; the only thing I could do was to gather 1-D arrays into a 2-D array or to gather all of them on a new defined 1-D array with the size of "comm.size*y.length". As I searched, using "MPI_Gather ($sendbuf , sendcnt, sendtype, &recbuf, recvcount, root, comm)" keyword in C++ we are able to do this task, HOWEVER, as it seams to me that "MPI.Gather" in C# is different and it does not have the flexibility of MPI_Gather in C++. I need to gather all the calculated parts of y in each process into the corresponding location in the y-array on root process. In other words, how can I specify a subset of an array as the send or receive buffer in MPI.NET for C#. I would appreciate it if you help me in this matter.
using (new MPI.Environment(ref args))
{
double sumSerial = 0;
double sumParallel = 0;
int arraySize = 100000;
double[] x = new double[arraySize];
double[] y = new double[arraySize];
Intracommunicator comm = Communicator.world;
int numProc = comm.Size;
int numItr = arraySize / numProc;
for (int i = 0; i < x.Length; i++)
{
x[i] = i;
sumSerial += i;
}
int firstInx = comm.Rank * numItr;
int lastInx = firstInx + numItr;
for (int i = firstInx; i < lastInx; i++)
{
y[i] = 5.0 * x[i];
}
//double[][] zz=comm.Gather<double[]>(y,0);
double[] z = comm.GatherFlattened(y, 0);
comm.Barrier();
if (comm.Rank==0)
{
//for (int i = 0; i < numProc; i++)
//{
// for (int j = 0; j < zz[0].Length; j++)
// {
// sumParallel += zz[i][j];
// }
//}
for (int i = 0; i < z.Length; i++)
{
sumParallel += z[i];
}
Console.WriteLine("sum_Parallel: {0}; sum_Serial= {1};
Ratio: {2}; z_length: {3}", sumParallel, sumSerial,
sumParallel / sumSerial, z.Length);
}
}

Calling from a public class and from its public method, into 2d double array, my zeros before the fractional point change to number 9

I've been writing a program, to use it as a tool for quick calculations in an online game, and it also helps me a bit to revise C# for my final exam in IT.
Here's my code:
public class ConvertingToArrays
{
public static double[,] CountryVAT(double[,] vat)
{
//I have a table in a .txt file with 20 rows and 6 columns
//and I only need one of the cols.
vat = new double[20, 1];
string[,] convertTableToString = new string[20, 6];
//Here I'm just calling from ReadFromFile public class and its
//public static string[,] Input method and until this point
//everything works fine
convertTableToString = ReadFromFile.Input(convertTableToString);
for (int i = 0; i < 20; i++)
{
for (int j = 0; j < 1; j++)
{
vat[i, j] = double.Parse(convertTableToString[i, 1]);
}
}
return vat;
}
}
With the string to double converting I had no problem, I tested it and it should not be the cause.
class Program
{
public static void Main(string[] args)
{
double[,] vat = new double[20, 1];
vat = ConvertingToArrays.CountryVAT(vat);
Console.WriteLine("Testing ConvertVAT Method Call");
for (int i = 0; i < 20; i++)
{
for (int j = 0; j < 1; j++)
{
Console.Write(vat[i, j] + '\t');
}
Console.WriteLine('\n');
}
Console.ReadKey();
}
}
I'm reading from a .txt file a few numbers like: 0,03; 0,05; 0,4; 0
And for some reason the output for these numbers are: 9,03; 9,05; 9,4; 9
I've tried to look it up on Google but I found nothing. It might be just one subtle and easy thing that I overlooked accidentally (please keep in mind that I have started learning to code by myself just 6 months before and I've practised it only 10-12 hours a week).
Can anyone help with a solution?

Look at this line:
Console.Write(vat[i, j] + '\t');
What is going on here? You add a double with a char value. You think you do some string operation, but that is not what your code is actually doing. Note that no strings are involved in the operation above. Both the double variable and the char literal are numeric data types, thus your code is executing an addition of the two numeric values.
What is the numeric value of the tabulator char '\t'? It is 9. So basically your code is doing Console.Write(vat[i, j] + 9);
There are different ways to you can change the code. One is to make two Console.Write calls like this:
Console.Write(vat[i, j]);
Console.Write('\t');
Alternatively, you could also force a string concatenation by converting the double value or the tab char to a string before "adding" them:
Console.Write(vat[i, j] + "\t");
or, less elegantly:
Console.Write(vat[i, j].ToString() + '\t');
As a third option you could also use format strings:
Console.Write("{0}{1}", vat[i, j], '\t');
or, simplified:
Console.Write("{0}\t", vat[i, j]);

Cannot implicitly convert type 'System.Array' to 'decimal[]'. An explicit conversion exists (are you missing a cast?)

How to convert this in C# code?
//Cannot implicitly convert type 'System.Array' to 'decimal[]'.
//An explicit conversion exists
Array[] s = new Array[10];
for (int i = 0; i < 9; i++)
{
s[i] = average; //average is decimal []
}
for (int i = 0; i < 2; i++)
{
Mean = s[i]; //mean is decimal[] & s[i] system array
}

It looks like you're trying to setup a jagged array. In which case, I suspect the best option is to type s as decimal[][]:
decimal[][] s = new decimal[10][];
Then everything will just work.
You can also cast:
Mean = (decimal[])s[i];
but that is less satisfying, IMO.

I don't know what you are trying to do with this code, as it does not calculate an average or mean,
but you are using the typeless array class, this is primarily intended to serve as a common base class so that all arrays exhibit the same behaviour, it can also be used in some rare situations when you don't know what type the array is
the correct way to defined an array is TypeName[ArraySize] so if you want a decimal array of size 10 then the correct use would be decimal[10]
here is your "working" code:
decimal demo_average = 0;
decimal demo_mean = 0;
decimal[] s = new decimal[10];
for (int i = 0; i < 9; i++)
{
s[i] = demo_average;
}
for (int i = 0; i < 2; i++)
{
demo_mean = s[i];
}

Force method to accept different variables in signature

is there a way to force a method in c# to accept arrays/variables of different types in the same signature slot or to make it ignore one part of the signature?
My code:
private void array_joiner(String[,] newArray, Int32[,] MatrixArray, String[,] RekursionArray, Char[] ArrayX, Char[] ArrayY)
{
for (Int16 i = 0; i < ArrayX.Length + 1; i++)
{
newArray[i, 0] = ArrayX[i].ToString();
}
for (Int16 i = 1; i < ArrayY.Length + 1; i++)
{
newArray[0, i] = ArrayY[i].ToString();
}
for (Int16 y = 1; y < ArrayY.Length + 1; y++)
{
for (Int16 x = 1; x < ArrayX.Length +1; x++)
{
newArray[y, x] = MatrixArray[y, x].ToString();
}
}
}
My problem is basically that I want to parse two different arrays in the slot of Int32[,]MatrixArray (Int32[,] and String[,]) to the method but I just don't know how. Anyone got a better idea than to write two different methods? Thx in advance.

You could just declare them as the Array class and then work out the type of the data in the arrays and how many ranks they have separately.

You basically have two ways:
Make it a generics method http://msdn.microsoft.com/it-it/library/512aeb7t.aspx (being careful to handle multidimensional collections which could be tricky)
Overload the method, declaring two signatures with the same name and different parameters http://msdn.microsoft.com/en-us/library/vstudio/ms229029(v=vs.100).aspx