Develop Reference

Long Running Task Not Having Access to Console?

I'm writing a producer-> Queue -> Consumer -> Queue2 -> Consumer2 application
I have the consumer2 wait for a list to get to a threshold then start another task simulating a long running task (e.g. SQL Multi=insert).
However, when I run the application the 'long running task' (LongRunningTaskInsert()) seems to wait until all of the queues have signaled completion before writing to the console.
When I debug, the List Lists variable shows that some of the tasks are completing in the middle of the application.
Am I doing something wrong/naughty with tasks?
Code:
class Program
{
static void Main(string[] args)
{
BlockingCollection<string> bag1 = new BlockingCollection<string>();
BlockingCollection<string> bag2 = new BlockingCollection<string>();
var Tasks = new List<Task>();
List<string> Container = new List<string>();
Task consumer2 = Task.Factory.StartNew(() =>
{
foreach (var item in bag2.GetConsumingEnumerable())
{
Container.Add(item);
if (bag2.IsCompleted || Container.Count > 5)
{
Console.WriteLine("Container:");
Container.ForEach(y =>
{
Console.Write($"Value: {y}, ");
});
Console.Write("\n");
var newTask = Task.Factory.StartNew(() => {
Thread.Sleep(2000);
LongRunningTaskInsert();
}
);
Tasks.Add(newTask);
Container.Clear();
}
}
Task.WhenAll(Tasks);
});
//this is a task that evaluates all available elements on separate threads.
Task consumer1 = Task.Factory.StartNew(() =>
{
//do something with the consumer
Parallel.ForEach(
bag1.GetConsumingEnumerable(),
(x) =>
{
Console.WriteLine($"Consumer {x} => bag2, thread {Thread.CurrentThread.ManagedThreadId}");
bag2.Add(x);
});
bag2.CompleteAdding();
});
Task producer = Task.Factory.StartNew(() =>
{
//do something to put records into the bad
for (int i = 0; i < 10; i++)
{
System.Threading.Thread.Sleep(500);
bag1.Add(i.ToString());
bag1.Add((i * 10).ToString());
bag1.Add((i + 10).ToString());
Console.WriteLine($"Producer: {i} & { i * 10} & {i + 10}");
}
bag1.CompleteAdding();
});
producer.Wait();
consumer1.Wait();
consumer2.Wait();
Console.Read();
}
private static bool LongRunningTaskInsert()
{
//Thread.Sleep(1000);
Console.WriteLine("Long Running Task Complete");
return true;
}
}
edit:
The output I'm getting is:
Producer: 0 & 0 & 10
Consumer 0 => bag2, thread 4
Consumer 0 => bag2, thread 6
Consumer 10 => bag2, thread 5
Producer: 1 & 10 & 11
Consumer 10 => bag2, thread 8
Consumer 11 => bag2, thread 10
Consumer 1 => bag2, thread 9
Container:
Value: 0, Value: 0, Value: 10, Value: 10, Value: 11, Value: 1,
Producer: 2 & 20 & 12
Consumer 20 => bag2, thread 4
Consumer 2 => bag2, thread 6
Consumer 12 => bag2, thread 5
Producer: 3 & 30 & 13
Consumer 3 => bag2, thread 10
Consumer 30 => bag2, thread 9
Consumer 13 => bag2, thread 8
Container:
Value: 20, Value: 2, Value: 12, Value: 3, Value: 30, Value: 13,
Producer: 4 & 40 & 14
Consumer 4 => bag2, thread 4
Consumer 40 => bag2, thread 6
Consumer 14 => bag2, thread 5
Producer: 5 & 50 & 15
Consumer 5 => bag2, thread 10
Consumer 15 => bag2, thread 8
Consumer 50 => bag2, thread 9
Container:
Value: 4, Value: 40, Value: 14, Value: 5, Value: 15, Value: 50,
Producer: 6 & 60 & 16
Consumer 6 => bag2, thread 6
Consumer 60 => bag2, thread 6
Producer: 7 & 70 & 17
Consumer 16 => bag2, thread 4
Consumer 70 => bag2, thread 5
Consumer 17 => bag2, thread 5
Consumer 7 => bag2, thread 4
Container:
Value: 6, Value: 60, Value: 16, Value: 70, Value: 17, Value: 7,
Producer: 8 & 80 & 18
Consumer 8 => bag2, thread 6
Consumer 80 => bag2, thread 6
Producer: 9 & 90 & 19
Consumer 90 => bag2, thread 4
Consumer 19 => bag2, thread 4
Consumer 18 => bag2, thread 8
Consumer 9 => bag2, thread 8
Container:
Value: 8, Value: 80, Value: 90, Value: 19, Value: 18, Value: 9,
Long Running Task Complete
Long Running Task Complete
Long Running Task Complete
Long Running Task Complete
Long Running Task Complete
I expect the 'Long Running Task Complete' to be mixed in and not all clustered at the end.

The Parallel.Foreach statement is spawning a bunch of threads and my method LongRunningTaskInsert() isn't getting any clock time.. If I change that to a synchronous foreach loop my number of threads is reduced from 8 to 4 and I get the results I expect (where the long running task console calls are mixed in).

What kind of Algorithm am I looking for to combine quantities?

I have been stuck on this problem now for 8 weeks and I think that I almost have a solution however the last bit of math is racking my mind. I will try to explain a simple problem that requires a complex solution. I am programing in C#.net MVC Web Project. Here is the situation.
I have an unknown group of quantities incoming to look for like items. Those like items share a max level to make it a full box. Here is an example of this:
Revision******
This is the real world case
I have many, let say candy, orders coming in to a company.
Qty Item MaxFill Sold-To DeliverNumber
60 candy#14 26 Joe 1
1 candy#12 48 Jim 2
30 candy#11 48 Jo 3
60 candy#15 48 Tom 4
6 candy#8 48 Kat 5
30 candy#61 48 Kim 6
44 candy#12 48 Jan 7
10 candy#12 48 Yai 8
10 candy#91 48 Jun 9
55 candy#14 26 Qin 10
30 candy#14 26 Yo 11
40 candy#14 26 Moe 12
in this list I am looking for like candy items to combine to make all the full boxes of candy that I can based off the MaxFill number. Here we see the like items are:
Qty Item MaxFill Sold-To DeliverNumber
60 candy#14 26 Joe 1
55 candy#14 26 Qin 10
30 candy#14 26 Yo 11
40 candy#14 26 Moe 12
1 candy#12 48 Jim 2
44 candy#12 48 Jan 7
10 candy#12 48 Yai 8
Now lets take the first set of numbers for candy#14.
I know that the total of candy#14 is 185 and I can get 7 full boxes of 26 with one box having only 3 in the last box. So how do I do this with the values that I have without losing the information of the original order. So this is how I am working it out right now
See below
End of Revision******
Like candy#14 max fill level is 26.
Like candy#14 quantities:
60
55
30
40
Now I already have a recursive function to break these down to the 26 level and is working fine. I feel that I need another recursive function to deal with the remainders that come out of this. As you can see most of the time there will be remainders from any given list but those remainders could total up to another full box of 26.
60 = 26+26+8
55 = 26+26+3
30 = 26+4
40 = 26+14
The 8,3,4,14 = 29 so I can get another 26 out of this. But in the real unknown world I could have the remainders come up with a new set of remainders that could repeat the same situation. To make this even more complicated I have to save the data that is originality with the 60,55,30,40 that is carried with it such as who it was sold to and delivery number. This will also be helpful with knowing how the original amount was broken down and combined together.
from the 8,3,4,14 the best way that I was think to add to that value is to take the 8,4,14 this will give me the 26 that I am looking for and I would not have to split any value because 3 is the remainder and I could save all other data without issue. However this just works in this situation only. If I go in a linear motion 8+3+4=15 so I would have to take 11 from the next value 14 with a remainder of 3.
In reading about different algorithms I was thinking that this might fall into the NP,NP-Complete,NP-Hard category. But with all the situations it is very technical and not a lot of real world scenarios are to be found.
Any suggestions would help here if I should go through the list of number to find the best combinations to reach the 26 or if the linear progression and splitting of the next value is the best solution. I know that I can solve to get how many full boxes I could get from the remainders and what the left over amount would be such as 8+3+4+14=29 which would give me 1, 26 and 1, 3 but I have no idea about the math in a recursive way to solve this. I have this much done and I "feel" that this is on the right track but can't see how to adjust to make this work with the linear or "test every possible combination".
public static void Main(string[] args)
{
var numbers = new List<int>() { 8, 3, 4, 14 };
var target = 26;
sum_up(numbers, target);
}
private static void sum_up(List<int> numbers, int target)
{
sum_up_recursive(numbers, target, new List<int>());
}
private static void sum_up_recursive(List<int> numbers, int target, List<int> partial)
{
int s = 0;
foreach (int x in partial) s += x;
if (s == target)
{
var outputtext = "sum(" + string.Join(",", partial.ToArray()) + ")=" + target;
}
if (s >= target)
return;
for (int i = 0; i < numbers.Count; i++)
{
List<int> remaining = new List<int>();
int n = numbers[i];
for (int j = i + 1; j < numbers.Count; j++) remaining.Add(numbers[j]);
List<int> partial_rec = new List<int>(partial);
partial_rec.Add(n);
sum_up_recursive(remaining, target, partial_rec);
}
}

I wrote sample project in javascript.
Please check my repo.
https://github.com/panghea/packaging_sample

Creating all possible arrays without nested for loops [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 6 years ago.
Improve this question
I would like to generate all the possible numbers that have length n and each digit of my number has a value from the set {1,2,...,n-1}, as an array. In other words, I would like to list all the base n numbers with length n that do not include 0.
Right now, the only way I can think to do it is by nesting n for loops, and assigning myArray[i] with the (i+1)th loop, i.e.
int n;
int[] myArray = new int[n];
for (int i1 = 1; i1 < n; i1++)
myArray[0]=i1;
for (int i2 = 1; i2 < n; i2++)
myArray[1]=i2;
// and so on....
for (int in = 1; in < n; in++)
{
myArray[n]=in;
foreach (var item in myArray)
Console.Write(item.ToString());
Console.Write(Environment.NewLine);
}
and then printing each array at the innermost loop. The obvious issue is that for each n, I need to manually write n for loops.
From what I've read, recursion seems to be the best way to replace nested for loops, but I can't seem figure out how to make a general method for recursion either.
EDIT
For example, if n=3, I would like to write out 1 1 1, 1 1 2, 1 2 1, 1 2 2, 2 1 1, 2 1 2, 2 2 1, 2 2 2.
We are not limited to n<11. For example, if n=11, we would output
1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 2
1 1 1 1 1 1 1 1 1 1 3
...
1 1 1 1 1 1 1 1 1 1 10
1 1 1 1 1 1 1 1 1 2 1
1 1 1 1 1 1 1 1 1 2 2
1 1 1 1 1 1 1 1 1 2 3
...
1 1 1 1 1 1 1 1 1 9 10
1 1 1 1 1 1 1 1 1 10 1
1 1 1 1 1 1 1 1 1 10 2
1 1 1 1 1 1 1 1 1 10 3
...
10 10 10 10 10 10 10 10 10 9 10
10 10 10 10 10 10 10 10 10 10 1
10 10 10 10 10 10 10 10 10 10 2
...
10 10 10 10 10 10 10 10 10 10 10
So, a digit of a number may be any value between and including 1 and 10. The array myArray is simply used to get one of these numbers, then we print it, and go on to the next number and repeat.

As always, when thinking in recursive solutions, try to solve the problem using immutable structures; everything is much simpler to understand.
So first of all, lets build ourselves a fast little immutable stack that will help us keep track of the number we are currently generating (while not worrying about what other numbers are being generated in the recursive call...remember, immutable data can't change!):
public class ImmutableStack<T>: IEnumerable<T>
{
public static readonly ImmutableStack<T> Empty = new ImmutableStack<T>();
private readonly T first;
private readonly ImmutableStack<T> rest;
public int Count { get; }
private ImmutableStack()
{
Count = 0;
}
private ImmutableStack(T first, ImmutableStack<T> rest)
{
Debug.Assert(rest != null);
this.first = first;
this.rest = rest;
Count = rest.Count + 1;
}
public IEnumerator<T> GetEnumerator()
{
var current = this;
while (current != Empty)
{
yield return current.first;
current = current.rest;
}
}
public T Peek()
{
if (this == Empty)
throw new InvalidOperationException("Can not peek an empty stack.");
return first;
}
public ImmutableStack<T> Pop()
{
if (this == Empty)
throw new InvalidOperationException("Can not pop an empty stack.");
return rest;
}
public ImmutableStack<T> Push(T item) => new ImmutableStack<T>(item, this);
IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
}
That's easy. Note how the stack reuses data. How many empty immutable structs will there be in our little program? Only one. And stacks containing the sequence 1->2->4? Yup, only one.
Now, we implement a recursive function that just keeps adding numbers to the stack until we reach our "bail out" condition. Which is? When the stack contains n elements. Easy peasy:
private static IEnumerable<int> generateNumbers(ImmutableStack<string> digits, IEnumerable<string> set, int length)
{
if (digits.Count == length)
{
yield return int.Parse(string.Concat(digits));
}
else
{
foreach (var digit in set)
{
var newDigits = digits.Push(digit);
foreach (var newNumber in generateNumbers(newDigits, set, length))
{
yield return newNumber;
}
}
}
}
Ok, and now we just need to tie it alltogether with our public method:
public static IEnumerable<int> GenerateNumbers(int length)
{
if (length < 1)
throw new ArgumentOutOfRangeException(nameof(length));
return generateNumbers(ImmutableStack<string>.Empty,
Enumerable.Range(1, length - 1).Select(d => d.ToString(CultureInfo.InvariantCulture)),
length);
}
And sure enough, if we call this thing:
var ns = GenerateNumbers(3);
Console.WriteLine(string.Join(Environment.NewLine,
ns.Select((n, index) => $"[{index + 1}]\t: {n}")));
We get the expected output:
[1] : 111
[2] : 211
[3] : 121
[4] : 221
[5] : 112
[6] : 212
[7] : 122
[8] : 222
Do note that the total amount of numbers generated of a specified length n is (n - 1) ^ n which means that for relatively small values of length you are going to get quite an amount of numbers generated; n = 10 generates 3 486 784 401...

Using async/await, when do continuations happen?

I am attempting to use async/await in a very large already existing synchronous code base. There is some global state in this code base that works fine, if kludgy, in a synchronous context, but it doesn't work in the asynchronous context of async/await.
So, my two options seem to be to either factor out the global context which woould be a very large and very time consuming task, or do something clever with when continuations run.
In order to better understand async/await and continuations, I made a test program, shown here. Shown here.
// A method to simulate an Async read of the database.
private static Task ReadAsync()
{
return Task.Factory.StartNew(() =>
{
int max = int.MaxValue / 2;
for (int i = 0; i < max; ++i)
{
}
});
}
// An async method that creates several continuations.
private static async Task ProcessMany(int i)
{
Console.WriteLine(string.Format("{0} {1}", i.ToString(), 0));
await ReadAsync();
Console.WriteLine(string.Format("{0} {1}", i.ToString(), 1));
await ReadAsync();
Console.WriteLine(string.Format("{0} {1}", i.ToString(), 2));
await ReadAsync();
Console.WriteLine(string.Format("{0} {1}", i.ToString(), 3));
}
public static void Main(string[] args)
{
Queue<Task> queue = new Queue<Task>();
for (int i = 0; i < 10; ++i)
{
queue.Enqueue(ProcessMany(i));
}
// Do some synchonous processing...
Console.WriteLine("Processing... ");
for (int i = 0; i < int.MaxValue; ++i)
{
}
Console.WriteLine("Done processing... ");
queue.Dequeue().Wait();
}
After reading all about async/await, my understanding would be that none of the continuations would happen between the "Processing.. " and "Done processing... " WriteLines.
Here is some sample output.
0 0
1 0
2 0
3 0
4 0
5 0
6 0
7 0
8 0
9 0
Processing...
3 1
2 1
7 1
6 1
0 1
4 1
5 1
1 1
6 2
3 2
Done processing...
7 2
2 2
0 2
4 2
5 2
1 2
6 3
3 3
7 3
2 3
0 3
I would expect the single Wait() at the end of the program to potentially yield to multiple continuations while the first one finishes, but I don't understand how any continuations could run between the "Processing... " and the "Done Processing... ". I thought there might be a yield or something in the Console.WriteLine method, so I completely replaced it, but that didn't change the output.
There is clearly a gap in my understanding of async/await. How could a continuation happen when we are simply incrementing a variable? Is the compiler or CLR injecting some sort of magic here?
Thank you in advance for any help in better understanding async/await and continuations.
EDIT:
If you edit the sample code this way as per the comment by Stephen, what's going is much more obvious.
// An async method that creates several continuations.
private static async Task ProcessMany(int i)
{
Console.WriteLine(string.Format("{0} {1} {2}", i.ToString(), 0, Thread.CurrentThread.ManagedThreadId));
await ReadAsync();
Console.WriteLine(string.Format("{0} {1} {2}", i.ToString(), 1, Thread.CurrentThread.ManagedThreadId));
await ReadAsync();
Console.WriteLine(string.Format("{0} {1} {2}", i.ToString(), 2, Thread.CurrentThread.ManagedThreadId));
await ReadAsync();
Console.WriteLine(string.Format("{0} {1} {2}", i.ToString(), 3, Thread.CurrentThread.ManagedThreadId));
}
public static void Main(string[] args)
{
Queue<Task> queue = new Queue<Task>();
for (int i = 0; i < 10; ++i)
{
queue.Enqueue(ProcessMany(i));
}
// Do some synchonous processing...
Console.WriteLine("Processing... {0}", Thread.CurrentThread.ManagedThreadId);
for (int i = 0; i < int.MaxValue; ++i)
{
}
Console.WriteLine("Done processing... {0}", Thread.CurrentThread.ManagedThreadId);
queue.Dequeue().Wait();
}
Output:
0 0 9
1 0 9
2 0 9
3 0 9
4 0 9
5 0 9
6 0 9
7 0 9
8 0 9
9 0 9
Processing... 9
4 1 14
3 1 13
2 1 12
5 1 15
0 1 10
6 1 16
1 1 11
7 1 17
4 2 14
3 2 13
0 2 10
6 2 16
2 2 12
5 2 15
Done processing... 9
1 2 11
7 2 17
0 3 10
4 3 14

If you don't have a current SynchronizationContext or TaskScheduler, then the continuations will execute on a thread pool thread (separately from the main thread). This is the case in Console apps but you'll see very different behavior in WinForms/WPF/ASP.NET.
While you could control the continuation scheduling by using a custom TaskScheduler, that would be quite a bit of work with probably very little benefit. I'm not clear on what the problems are with your global state, but consider alternatives such as SemaphoreSlim.

As soon as you call the following line in your ProcessMany, each call to ProcessMany starts executing in a separate thread in a separate thread pool right away.
await ...;
So that's why you see a bunch of calls before your "Processing" printout. So while you have all those 10 ProcessMany calls executing, then you start running your large loop. As that large loop is running on your main thread, the 10 ProcessMany calls continue to execute in their threads, producing the additional printouts. Looks like your ProcessMany calls do not finish executing before your main thread loop, so they continue to spit out more results after your "Done Processing" printout.
I hope that clarifies the order of things for you.

Generate N random and unique numbers within a range

What is an efficient way of generating N unique numbers within a given range using C#? For example, generate 6 unique numbers between 1 and 50. A lazy way would be to simply use Random.Next() in a loop and store that number in an array/list, then repeat and check if it already exists or not etc.. Is there a better way to generate a group of random, but unique, numbers?
To add more context, I would like to select N random items from a collection, using their index.
thanks

Take an array of 50 elements: {1, 2, 3, .... 50}
Shuffle the array using any of the standard algorithms of randomly shuffling arrays. The first six elements of the modified array is what you are looking for. HTH

For 6-from-50, I'm not too sure I'd worry about efficiency since the chance of a duplicate is relatively low (30% overall, from my back-of-the-envelope calculations). You could quite easily just remember the previous numbers you'd generated and throw them away, something like (pseudo-code):
n[0] = rnd(50)
for each i in 1..5:
n[i] = n[0]
while n[1] == n[0]:
n[1] = rnd(50)
while n[2] == any of (n[0], n[1]):
n[2] = rnd(50)
while n[3] == any of (n[0], n[1], n[2]):
n[3] = rnd(50)
while n[4] == any of (n[0], n[1], n[2], n[3]):
n[4] = rnd(50)
while n[5] == any of (n[0], n[1], n[2], n[3], n[4]):
n[5] = rnd(50)
However, this will break down as you move from 6-from-50 to 48-from-50, or 6-from-6, since the duplicates start getting far more probable. That's because the pool of available numbers gets smaller and you end up throwing away more and more.
For a very efficient solution that gives you a subset of your values with zero possibility of duplicates (and no unnecessary up-front sorting), Fisher-Yates is the way to go.
dim n[50] // gives n[0] through n[9]
for each i in 0..49:
n[i] = i // initialise them to their indexes
nsize = 50 // starting pool size
do 6 times:
i = rnd(nsize) // give a number between 0 and nsize-1
print n[i]
nsize = nsize - 1 // these two lines effectively remove the used number
n[i] = n[nsize]
By simply selecting a random number from the pool, replacing it with the top number from that pool, then reducing the size of the pool, you get a shuffle without having to worry about a large number of swaps up front.
This is important if the number is high in that it doesn't introduce an unnecessary startup delay.
For example, examine the following bench-check, choosing 10-from-10:
<------ n[] ------>
0 1 2 3 4 5 6 7 8 9 nsize rnd(nsize) output
------------------- ----- ---------- ------
0 1 2 3 4 5 6 7 8 9 10 4 4
0 1 2 3 9 5 6 7 8 9 7 7
0 1 2 3 9 5 6 8 8 2 2
0 1 8 3 9 5 6 7 6 6
0 1 8 3 9 5 6 0 0
5 1 8 3 9 5 2 8
5 1 9 3 4 1 1
5 3 9 3 0 5
9 3 2 1 3
9 1 0 9
You can see the pool reducing as you go and, because you're always replacing the used one with an unused one, you'll never have a repeat.
Using the results returned from that as indexes into your collection will guarantee that no duplicate items will be selected.

var random = new Random();
var intArray = Enumerable.Range(0, 4).OrderBy(t => random.Next()).ToArray();
This array will contain 5 random numbers from 0 to 4.
or
var intArray = Enumerable.Range(0, 10).OrderBy(t => random.Next()).Take(5).ToArray();
This array will contain 5 random numbers between 0 to 10.
int firstNumber = intArray[0];
int secondNumber = intArray[1];
int thirdNumber = intArray[2];
int fourthNumber = intArray[3];
int fifthNumber = intArray[4];

For large sets of unique numbers, put them in an List..
Random random = new Random();
List<int> uniqueInts = new List<int>(10000);
List<int> ranInts = new List<int>(500);
for (int i = 1; i < 10000; i++) { uniqueInts.Add(i); }
for (int i = 1; i < 500; i++)
{
int index = random.Next(uniqueInts.Count) + 1;
ranInts.Add(uniqueInts[index]);
uniqueInts.RemoveAt(index);
}
Then randomly generate a number from 1 to myInts.Count. Store the myInt value and remove it from the List. No need to shuffle the list nor look to see if the value already exists.

instead of using List use Dictionary!!

In case it helps anyone else, I prefer allocating the minimum number of items necessary. Below, I make use of a HashSet, which ensures that new items are unique. This should work with very large collections as well, up to the limits of what HashSet plays nice with.
public static IEnumerable<int> GetRandomNumbers(int numValues, int maxVal)
{
var rand = new Random();
var yieldedValues = new HashSet<int>();
int counter = 0;
while (counter < numValues)
{
var r = rand.Next(maxVal);
if (yieldedValues.Add(r))
{
counter++;
yield return r;
}
}
}

generate unique random nos from 1 to 40 :
output confirmed :
class Program
{
static int[] a = new int[40];
static Random r = new Random();
static bool b;
static void Main(string[] args)
{
int t;
for (int i = 0; i < 20; i++)
{
lab: t = r.Next(1, 40);
for(int j=0;j<20;j++)
{
if (a[j] == t)
{
goto lab;
}
}
a[i] = t;
Console.WriteLine(a[i]);
}
Console.Read();
}
}
sample output :
7
38
14
18
13
29
28
26
22
8
24
19
35
39
33
32
20
2
15
37