Develop Reference

How to write a generic function that will call a general function with a specific class?

My code:
public class BaseParamsClass
{
public BaseParamsClass(int pBaseParam = 0)
{
baseParam = pBaseParam;
}
public int baseParam;
}
public class Parent1ParamsClass : BaseParamsClass
{
public Parent1ParamsClass(int pBaseParam = 0) : base(pBaseParam)
{
}
public int parentParam1;
}
public class Parent2ParamsClass : BaseParamsClass
{
public Parent2ParamsClass(int pBaseParam = 0) : base(pBaseParam)
{
}
public int parentParam2;
}
public delegate void Parent1Callback(Parent1ParamsClass theParams);
public delegate void Parent2Callback(Parent2ParamsClass theParams);
private IEnumerator CoRFunction1(Parent1Callback parent1Callback)
{
// This demonstrate few actions i do before the call
yield return new WaitForSeconds(1);
// Now i get the result, it can be 0-10, each one should activate different callback
int result = 0;
parent1Callback(new Parent1ParamsClass(0));
}
private IEnumerator CoRFunction2(Parent2Callback parent2Callback)
{
// This demonstrate few actions i do before the call
yield return new WaitForSeconds(1);
// Now i get the result, it can be 0-10, each one should activate different callback
int result = 0;
// Need a generic way to do the next line:
parent2Callback(new Parent2ParamsClass(0));
}
private IEnumerator CoRFunction2(Parent2Callback parent2Callback)
{
// This demonstrate few actions i do before the call
yield return new WaitForSeconds(1);
// Now i get the result, it can be 0-10, each one should activate different callback
int result = 0;
// Need a generic way to do the next line:
parent2Callback(new Parent2ParamsClass(0));
}
And what i need is a way to replace the line after the '// Need a...' in something more generic, this is how the last two functions should look like:
private IEnumerator CoRFunction1(Parent1Callback parent1Callback)
{
// This demonstrate few actions i do before the call
yield return new WaitForSeconds(1);
// Now i get the result, it can be 0-10, each one should activate different callback
int result = 0;
genericFunction<Parent1Callback>(Parent1ParamsClass);
}
private IEnumerator CoRFunction2(Parent2Callback parent2Callback)
{
// This demonstrate few actions i do before the call
yield return new WaitForSeconds(1);
// Now i get the result, it can be 0-10, each one should activate different callback
int result = 0;
genericFunction<Parent2Callback>(Parent2ParamsClass);
}
Any ideas how to create 'genericFunction'?

I commented - it's not too clear exactly what you're trying to achieve and I suspect you can do this in a better way than what you are trying to... but for the record, I think something like the below might work.
Change the base parameter classes so you're not relying on the constructor to set their internal fields, then constrain some generic methods to the base type only, so you'd end up with this sort of thing:
public class BaseParamsClass
{
public virtual void SetParam(int pBaseParam)
{
baseParam = 0;
}
public int baseParam;
}
public class Parent1ParamsClass : BaseParamsClass
{
public override void SetParam(int pBaseParam)
{
base.SetParam(pBaseParam);
//do other stuff specific to this class...
}
public int parentParam1;
}
public class Parent2ParamsClass : BaseParamsClass
{
public override void SetParam(int pBaseParam)
{
base.SetParam(pBaseParam);
//do other stuff specific to this class...
}
public int parentParam2;
}
public delegate void GenericCallback<T>(T theParams) where T : BaseParamsClass, new();
private IEnumerator GenericCorFunction<T>(GenericCallback<T> callback) where T:BaseParamsClass, new()
{
// This demonstrate few actions i do before the call
yield return new WaitForSeconds(1);
// Now i get the result, it can be 0-10, each one should activate different callback
int result = 0;
//I assume you want result here.
//Also note that you can't use the constructor to set the base param as at compile time
//we're not sure which type will be being used. There are ways around this but it's
//probably clearer to use basic constructor then call the SetParam virtual/overridden method
var param = new T();
param.SetParam(result);
callback(param);
}
you could use it something like this:
var newEnumerator = GenericCorFunction<Parent2ParamsClass>(p =>
{
//this is the callback function body. This will only run when
//called at the end of the GenericCorFunction
//Do things with p, which will be a PArent2ParamsClass object
//with its baseParam field set to whatever result was.
if (p.baseParam == 3)
{
throw new NotImplementedException();
}
});
//do stuff with newEnumerator...

Another approach could be to use Activator.CreateInstance. This will allow you to circumvent the new() restriction and to use your already written constructor:
public delegate void ParentCallback<T>(T theParams) where T : BaseParamsClass;
private void CoRFunction<T>(ParentCallback<T> parentCallback) where T : BaseParamsClass
{
// Now i get the result, it can be 0-10, each one should activate different callback
int result = 0;
parentCallback((T)Activator.CreateInstance(typeof(T), 11));
}
(I changed it into void to make it testable for me)
Here is the TestCode and the calls I used to test it:
public void Call_1(Parent1ParamsClass par1)
{
Console.WriteLine("CALL 1 baseParam: " + par1.baseParam);
}
public void Call_2(Parent2ParamsClass par2)
{
Console.WriteLine("CALL 2 baseParam: " + par2.baseParam);
}
Calls:
CoRFunction<Parent1ParamsClass>(Call_1);
CoRFunction<Parent2ParamsClass>(Call_2);
Output:
CALL 1 baseParam: 11
CALL 2 baseParam: 11

C# MongoDB driver only returning 100 results

I am writing a mailing label, and need to print a label for each document.
I have 829 documents on the Collection, but when I retrieve them, I only get 100 documents.
I have this LINQ code:
IMongoCollection Pessoa;
Pessoa = database.GetCollection<Pessoa>(collectionName);
return Pessoa.AsQueryable().ToList();
How to retrieve ALL the documents?

I have 829 documents on the Collection, but when I retrieve them, I only get 100 documents.
I could reproduce the issue on my side, using AsQueryable extension method on IMongoCollection collection.AsQueryable() to find documents in a collection, which always return 100 documents even though I changed Items per page setting to Unlimited on Azure portal.
Setting:
Test code:
Count documents in query explorer:
To query all the documents in a collection, as you mentioned in comment, you could try to call Find method with an empty filter.

You're probably being limited by the default cursor BatchSize.
You can modify this behaviour passing an AggregateOptions object to the AsQueryable extension and setting the BatchSize property to a large enough value.
public static IMongoQueryable<TDocument> AsQueryable<TDocument>(this IMongoCollection<TDocument> collection, AggregateOptions aggregateOptions = null)

I found the question useful, so I wrote a convenient IEnumerator:
private sealed class MongoCollectionEnumerator : IEnumerator<T> {
private IMongoCollection<T> _collection;
private IAsyncCursor<T> _cursor; // outer enumerator
private IEnumerator<T> _currentBatchEnumerator; // inner enumerator
public MongoCollectionEnumerator(IMongoCollection<T> collection) {
_collection = collection;
InternalInit();
}
#region interface implementation
T IEnumerator<T>.Current {
get {
return _currentBatchEnumerator.Current;
}
}
object IEnumerator.Current {
get {
return ThisAsTypedIEnumerator.Current;
}
}
bool IEnumerator.MoveNext() {
if (_currentBatchEnumerator != null) {
if (_currentBatchEnumerator.MoveNext()) {
return true;
}
}
// inner not initialized or already at end
if (_cursor.MoveNext()) {
// advance the outer and defer back to the inner by recursing
_currentBatchEnumerator = _cursor.Current.GetEnumerator();
return ThisAsIEnumerator.MoveNext();
}
else { // outer cannot advance, this is the end
return false;
}
}
void IEnumerator.Reset() {
InternalCleanUp();
InternalInit();
}
#endregion
#region methods private
// helper properties to retrieve an explicit interface-casted this
private IEnumerator ThisAsIEnumerator => this;
private IEnumerator<T> ThisAsTypedIEnumerator => this;
private void InternalInit() {
var filterBuilder = new FilterDefinitionBuilder<T>();
_cursor = _collection.Find(filterBuilder.Empty).ToCursor();
}
private void InternalCleanUp() {
if (_currentBatchEnumerator != null) {
_currentBatchEnumerator.Reset();
_currentBatchEnumerator = null;
}
if (_cursor != null) {
_cursor.Dispose();
_cursor = null;
}
}
#endregion
#region IDisposable implementation
private bool disposedValue = false; // To detect redundant calls
private void InternalDispose(bool disposing) {
if (!disposedValue) {
if (disposing) {
InternalCleanUp();
_collection = null;
}
disposedValue = true;
}
}
void IDisposable.Dispose() {
InternalDispose(true);
}
#endregion
}

implement ienumerable with ienumerable<T>

What is the need to implement IEnumerable (Non-generic) with IEnumerable (Generic) interface in a generic collection class?
A code example on msdn states (Link - http://msdn.microsoft.com/en-us/library/9eekhta0(v=vs.110).aspx)
public class App
{
// Excercise the Iterator and show that it's more
// performant.
public static void Main()
{
TestStreamReaderEnumerable();
TestReadingFile();
}
public static void TestStreamReaderEnumerable()
{
// Check the memory before the iterator is used.
long memoryBefore = GC.GetTotalMemory(true);
// Open a file with the StreamReaderEnumerable and check for a string.
var stringsFound =
from line in new StreamReaderEnumerable(#"c:\\temp\\tempFile.txt")
where line.Contains("string to search for")
select line;
Console.WriteLine("Found: " + stringsFound.Count());
// Check the memory after the iterator and output it to the console.
long memoryAfter = GC.GetTotalMemory(false);
Console.WriteLine("Memory Used With Iterator = \t"
+ string.Format(((memoryAfter - memoryBefore) / 1000).ToString(), "n") + "kb");
}
public static void TestReadingFile()
{
long memoryBefore = GC.GetTotalMemory(true);
StreamReader sr = File.OpenText("c:\\temp\\tempFile.txt");
// Add the file contents to a generic list of strings.
List<string> fileContents = new List<string>();
while (!sr.EndOfStream) {
fileContents.Add(sr.ReadLine());
}
// Check for the string.
var stringsFound =
from line in fileContents
where line.Contains("string to search for")
select line;
sr.Close();
Console.WriteLine("Found: " + stringsFound.Count());
// Check the memory after when the iterator is not used, and output it to the console.
long memoryAfter = GC.GetTotalMemory(false);
Console.WriteLine("Memory Used Without Iterator = \t" +
string.Format(((memoryAfter - memoryBefore) / 1000).ToString(), "n") + "kb");
}
}
// A custom class that implements IEnumerable(T). When you implement IEnumerable(T),
// you must also implement IEnumerable and IEnumerator(T).
public class StreamReaderEnumerable : IEnumerable<string>
{
private string _filePath;
public StreamReaderEnumerable(string filePath)
{
_filePath = filePath;
}
// Must implement GetEnumerator, which returns a new StreamReaderEnumerator.
public IEnumerator<string> GetEnumerator()
{
return new StreamReaderEnumerator(_filePath);
}
// Must also implement IEnumerable.GetEnumerator, but implement as a private method.
private IEnumerator GetEnumerator1()
{
return this.GetEnumerator();
}
IEnumerator IEnumerable.GetEnumerator()
{
return GetEnumerator1();
}
}
// When you implement IEnumerable(T), you must also implement IEnumerator(T),
// which will walk through the contents of the file one line at a time.
// Implementing IEnumerator(T) requires that you implement IEnumerator and IDisposable.
public class StreamReaderEnumerator : IEnumerator<string>
{
private StreamReader _sr;
public StreamReaderEnumerator(string filePath)
{
_sr = new StreamReader(filePath);
}
private string _current;
// Implement the IEnumerator(T).Current publicly, but implement
// IEnumerator.Current, which is also required, privately.
public string Current
{
get
{
if (_sr == null || _current == null)
{
throw new InvalidOperationException();
}
return _current;
}
}
private object Current1
{
get { return this.Current; }
}
object IEnumerator.Current
{
get { return Current1; }
}
// Implement MoveNext and Reset, which are required by IEnumerator.
public bool MoveNext()
{
_current = _sr.ReadLine();
if (_current == null)
return false;
return true;
}
public void Reset()
{
_sr.DiscardBufferedData();
_sr.BaseStream.Seek(0, SeekOrigin.Begin);
_current = null;
}
// Implement IDisposable, which is also implemented by IEnumerator(T).
private bool disposedValue = false;
public void Dispose()
{
Dispose(true);
GC.SuppressFinalize(this);
}
protected virtual void Dispose(bool disposing)
{
if (!this.disposedValue)
{
if (disposing)
{
// Dispose of managed resources.
}
_current = null;
_sr.Close();
_sr.Dispose();
}
this.disposedValue = true;
}
~StreamReaderEnumerator()
{
Dispose(false);
}
// This example displays output similar to the following:
//Found: 2
//Memory Used With Iterator = 33kb
//Found: 2
//Memory Used Without Iterator = 206kb
}

The need is simply because of how the interfaces are implemented. In .NET 1.1, there were no generics, so IEnumerable has no generic support and only exposes object. This introduces boxing (and is why the compiler also supports a pattern-based implementation for foreach, independent of IEnumerable). In C# 2, we got IEnumerable<T>. It is useful to consider all typed iterations as also being untyped, hence:
IEnumerable<T> : IEnumerable
where IEnumerable<T> re-declares GetEnumerator as with the generic type. However, since these are now different methods with different signatures, it needs 2 different implementations.

C# Priority Queue

I'm looking for a priority queue with an interface like this:
class PriorityQueue<T>
{
public void Enqueue(T item, int priority)
{
}
public T Dequeue()
{
}
}
All the implementations I've seen assume that item is an IComparable but I don't like this approach; I want to specify the priority when I'm pushing it onto the queue.
If a ready-made implementation doesn't exist, what's the best way to go about doing this myself? What underlying data structure should I use? Some sort of self-balancing tree, or what? A standard C#.net structure would be nice.

If you have an existing priority queue implementation based on IComparable, you can easily use that to build the structure you need:
public class CustomPriorityQueue<T> // where T need NOT be IComparable
{
private class PriorityQueueItem : IComparable<PriorityQueueItem>
{
private readonly T _item;
private readonly int _priority:
// obvious constructor, CompareTo implementation and Item accessor
}
// the existing PQ implementation where the item *does* need to be IComparable
private readonly PriorityQueue<PriorityQueueItem> _inner = new PriorityQueue<PriorityQueueItem>();
public void Enqueue(T item, int priority)
{
_inner.Enqueue(new PriorityQueueItem(item, priority));
}
public T Dequeue()
{
return _inner.Dequeue().Item;
}
}

You can add safety checks and what not, but here is a very simple implementation using SortedList:
class PriorityQueue<T> {
SortedList<Pair<int>, T> _list;
int count;
public PriorityQueue() {
_list = new SortedList<Pair<int>, T>(new PairComparer<int>());
}
public void Enqueue(T item, int priority) {
_list.Add(new Pair<int>(priority, count), item);
count++;
}
public T Dequeue() {
T item = _list[_list.Keys[0]];
_list.RemoveAt(0);
return item;
}
}
I'm assuming that smaller values of priority correspond to higher priority items (this is easy to modify).
If multiple threads will be accessing the queue you will need to add a locking mechanism too. This is easy, but let me know if you need guidance here.
SortedList is implemented internally as a binary tree.
The above implementation needs the following helper classes. This address Lasse V. Karlsen's comment that items with the same priority can not be added using the naive implementation using a SortedList.
class Pair<T> {
public T First { get; private set; }
public T Second { get; private set; }
public Pair(T first, T second) {
First = first;
Second = second;
}
public override int GetHashCode() {
return First.GetHashCode() ^ Second.GetHashCode();
}
public override bool Equals(object other) {
Pair<T> pair = other as Pair<T>;
if (pair == null) {
return false;
}
return (this.First.Equals(pair.First) && this.Second.Equals(pair.Second));
}
}
class PairComparer<T> : IComparer<Pair<T>> where T : IComparable {
public int Compare(Pair<T> x, Pair<T> y) {
if (x.First.CompareTo(y.First) < 0) {
return -1;
}
else if (x.First.CompareTo(y.First) > 0) {
return 1;
}
else {
return x.Second.CompareTo(y.Second);
}
}
}

You could write a wrapper around one of the existing implementations that modifies the interface to your preference:
using System;
class PriorityQueueThatYouDontLike<T> where T: IComparable<T>
{
public void Enqueue(T item) { throw new NotImplementedException(); }
public T Dequeue() { throw new NotImplementedException(); }
}
class PriorityQueue<T>
{
class ItemWithPriority : IComparable<ItemWithPriority>
{
public ItemWithPriority(T t, int priority)
{
Item = t;
Priority = priority;
}
public T Item {get; private set;}
public int Priority {get; private set;}
public int CompareTo(ItemWithPriority other)
{
return Priority.CompareTo(other.Priority);
}
}
PriorityQueueThatYouDontLike<ItemWithPriority> q = new PriorityQueueThatYouDontLike<ItemWithPriority>();
public void Enqueue(T item, int priority)
{
q.Enqueue(new ItemWithPriority(item, priority));
}
public T Dequeue()
{
return q.Dequeue().Item;
}
}
This is the same as itowlson's suggestion. I just took longer to write mine because I filled out more of the methods. :-s

Here's a very simple lightweight implementation that has O(log(n)) performance for both push and pop. It uses a heap data structure built on top of a List<T>.
/// <summary>Implements a priority queue of T, where T has an ordering.</summary>
/// Elements may be added to the queue in any order, but when we pull
/// elements out of the queue, they will be returned in 'ascending' order.
/// Adding new elements into the queue may be done at any time, so this is
/// useful to implement a dynamically growing and shrinking queue. Both adding
/// an element and removing the first element are log(N) operations.
///
/// The queue is implemented using a priority-heap data structure. For more
/// details on this elegant and simple data structure see "Programming Pearls"
/// in our library. The tree is implemented atop a list, where 2N and 2N+1 are
/// the child nodes of node N. The tree is balanced and left-aligned so there
/// are no 'holes' in this list.
/// <typeparam name="T">Type T, should implement IComparable[T];</typeparam>
public class PriorityQueue<T> where T : IComparable<T> {
/// <summary>Clear all the elements from the priority queue</summary>
public void Clear () {
mA.Clear ();
}
/// <summary>Add an element to the priority queue - O(log(n)) time operation.</summary>
/// <param name="item">The item to be added to the queue</param>
public void Add (T item) {
// We add the item to the end of the list (at the bottom of the
// tree). Then, the heap-property could be violated between this element
// and it's parent. If this is the case, we swap this element with the
// parent (a safe operation to do since the element is known to be less
// than it's parent). Now the element move one level up the tree. We repeat
// this test with the element and it's new parent. The element, if lesser
// than everybody else in the tree will eventually bubble all the way up
// to the root of the tree (or the head of the list). It is easy to see
// this will take log(N) time, since we are working with a balanced binary
// tree.
int n = mA.Count; mA.Add (item);
while (n != 0) {
int p = n / 2; // This is the 'parent' of this item
if (mA[n].CompareTo (mA[p]) >= 0) break; // Item >= parent
T tmp = mA[n]; mA[n] = mA[p]; mA[p] = tmp; // Swap item and parent
n = p; // And continue
}
}
/// <summary>Returns the number of elements in the queue.</summary>
public int Count {
get { return mA.Count; }
}
/// <summary>Returns true if the queue is empty.</summary>
/// Trying to call Peek() or Next() on an empty queue will throw an exception.
/// Check using Empty first before calling these methods.
public bool Empty {
get { return mA.Count == 0; }
}
/// <summary>Allows you to look at the first element waiting in the queue, without removing it.</summary>
/// This element will be the one that will be returned if you subsequently call Next().
public T Peek () {
return mA[0];
}
/// <summary>Removes and returns the first element from the queue (least element)</summary>
/// <returns>The first element in the queue, in ascending order.</returns>
public T Next () {
// The element to return is of course the first element in the array,
// or the root of the tree. However, this will leave a 'hole' there. We
// fill up this hole with the last element from the array. This will
// break the heap property. So we bubble the element downwards by swapping
// it with it's lower child until it reaches it's correct level. The lower
// child (one of the orignal elements with index 1 or 2) will now be at the
// head of the queue (root of the tree).
T val = mA[0];
int nMax = mA.Count - 1;
mA[0] = mA[nMax]; mA.RemoveAt (nMax); // Move the last element to the top
int p = 0;
while (true) {
// c is the child we want to swap with. If there
// is no child at all, then the heap is balanced
int c = p * 2; if (c >= nMax) break;
// If the second child is smaller than the first, that's the one
// we want to swap with this parent.
if (c + 1 < nMax && mA[c + 1].CompareTo (mA[c]) < 0) c++;
// If the parent is already smaller than this smaller child, then
// we are done
if (mA[p].CompareTo (mA[c]) <= 0) break;
// Othewise, swap parent and child, and follow down the parent
T tmp = mA[p]; mA[p] = mA[c]; mA[c] = tmp;
p = c;
}
return val;
}
/// <summary>The List we use for implementation.</summary>
List<T> mA = new List<T> ();
}

That is the exact interface used by my highly optimized C# priority-queue.
It was developed specifically for pathfinding applications (A*, etc.), but should work perfectly for any other application as well.
public class User
{
public string Name { get; private set; }
public User(string name)
{
Name = name;
}
}
...
var priorityQueue = new SimplePriorityQueue<User>();
priorityQueue.Enqueue(new User("Jason"), 1);
priorityQueue.Enqueue(new User("Joseph"), 10);
//Because it's a min-priority queue, the following line will return "Jason"
User user = priorityQueue.Dequeue();

What would be so terrible about something like this?
class PriorityQueue<TItem, TPriority> where TPriority : IComparable
{
private SortedList<TPriority, Queue<TItem>> pq = new SortedList<TPriority, Queue<TItem>>();
public int Count { get; private set; }
public void Enqueue(TItem item, TPriority priority)
{
++Count;
if (!pq.ContainsKey(priority)) pq[priority] = new Queue<TItem>();
pq[priority].Enqueue(item);
}
public TItem Dequeue()
{
--Count;
var queue = pq.ElementAt(0).Value;
if (queue.Count == 1) pq.RemoveAt(0);
return queue.Dequeue();
}
}
class PriorityQueue<TItem> : PriorityQueue<TItem, int> { }

I realise that your question specifically asks for a non-IComparable-based implementation, but I want to point out a recent article from Visual Studio Magazine.
http://visualstudiomagazine.com/articles/2012/11/01/priority-queues-with-c.aspx
This article with #itowlson's can give a complete answer.

A little late but I'll add it here for reference
https://github.com/ERufian/Algs4-CSharp
Key-value-pair priority queues are implemented in Algs4/IndexMaxPQ.cs, Algs4/IndexMinPQ.cs and Algs4/IndexPQDictionary.cs
Notes:
If the Priorities are not IComparable's, an IComparer can be specified in the constructor
Instead of enqueueing the object and its priority, what is enqueued is an index and its priority (and, for the original question, a separate List or T[] would be needed to convert that index to the expected result)

.NET6 finally offers an API for PriorityQueue
See here

Seems like you could roll your own with a seriews of Queues, one for each priority. Dictionary and just add it to the appropriate one.

Which is the best way to improve memory usage when you collect a large data set before processing it? (.NET)

When I have to get GBs of data, save it on a collection and process it, I have memory overflows. So instead of:
public class Program
{
public IEnumerable<SomeClass> GetObjects()
{
var list = new List<SomeClass>();
while( // get implementation
list.Add(object);
}
return list;
}
public void ProcessObjects(IEnumerable<SomeClass> objects)
{
foreach(var object in objects)
// process implementation
}
void Main()
{
var objects = GetObjects();
ProcessObjects(objects);
}
}
I need to:
public class Program
{
void ProcessObject(SomeClass object)
{
// process implementation
}
public void GetAndProcessObjects()
{
var list = new List<SomeClass>();
while( // get implementation
Process(object);
}
return list;
}
void Main()
{
var objects = GetAndProcessObjects();
}
}
There is a better way?

You ought to leverage C#'s iterator blocks and use the yield return statement to do something like this:
public class Program
{
public IEnumerable<SomeClass> GetObjects()
{
while( // get implementation
yield return object;
}
}
public void ProcessObjects(IEnumerable<SomeClass> objects)
{
foreach(var object in objects)
// process implementation
}
void Main()
{
var objects = GetObjects();
ProcessObjects(objects);
}
}
This would allow you to stream each object and not keep the entire sequence in memory - you would only need to keep one object in memory at a time.

Don't use a List, which requires all the data to be present in memory at once. Use IEnumerable<T> and produce the data on demand, or better, use IQueryable<T> and have the entire execution of the query deferred until the data are required.
Alternatively, don't keep the data in memory at all, but rather save the data to a database for processing. When processing is complete, then query the database for the results.

public IEnumerable<SomeClass> GetObjects()
{
foreach( var obj in GetIQueryableObjects
yield return obj
}

You want to yield!
Delay processing of your enumeration. Build a method that returns an IEnumerable but only returns one record at a time using the yield statement.

The best methodology in this case would be to Get and Process in chunks. You will have to find out how big a chunk to Get and Process by trial and error. So the code would be something like :
public class Program
{
public IEnumerable GetObjects(int anchor, int chunkSize)
{
var list = new List();
while( // get implementation for given anchor and chunkSize
list.Add(object);
}
return list;
}
public void ProcessObjects(IEnumerable<SomeClass> objects)
{
foreach(var object in objects)
// process implementation
}
void Main()
{
int chunkSize = 5000;
int totalSize = //Get Total Number of rows;
int anchor = //Get first row to process as anchor;
While (anchor < totalSize)
(
var objects = GetObjects(anchor, chunkSize);
ProcessObjects(objects);
anchor += chunkSize;
}
}
}