Monday, December 15, 2008 11:00 AM

Inline Asynchronous UI Coding using IDisposable in C# .NET

I recently watched this video on the PowerThreading library: http://blogs.msdn.com/charlie/archive/2008/12/03/jeff-richter-video-on-asynchronous-programming-and-his-power-threading-library.aspx

This is very interesting as Jeff Richter utilizes a strange behavior in the way the C# compiler works with IEnumerables to make a useful and easy-to-use async library.  His primary goal was to allow a website\service yield processing during I/O operations, which can provide for better scalability in your app tier. 

While this is quite interesting, I personally would avoid using strange compilation patterns to my advantage in an application.  Specifically because you have no assurance that the compiler will always do this as the language and subsequent compilers evolve.  What I was interested in was the mechanism for including asynchronous calls into code without using a complex series of callbacks. 

Now, to be quite fair, Jeff's implementation was an attempt to limit thread spawning, and context switches.  I am not going to account for this.  I am more interested in parrallel processing.

Let me give you an example.  Take for instance a series of objects that the UI is going to take cues from and make subsequent database calls to retrieve extra data for.  A simple example is a treeview.  If I have a bunch of nodes in a tree and the UI requires that I display the child node or items count for each of these nodes. What you would generally do is loop through the objects and call the DB to retrieve the data, and assign the UI components in one serial block of code.  The problem with this is that the database is more than likely capable of servicing the request for numerous nodes simultaneously, but in the UI we are implementing this serially.  If the tree structure is complex, and you are showing more than just a node count, it is plausible that each of these requests will take upward of 50ms.  If you have several dozen nodes being displayed this can easily add seconds to your response time.

A serial block of code might look like this:

TreeNodeCollection nodes = new TreeNodeCollection();
foreach (TreeNode node in nodes)
{
     int objectCount = database.DoSomething((MyObject)node.Tag); //call the database
     node.Text = String.Format("{0} ({1})", node.Text, objectCount); //update the UI
 }

 Simple and clean, but slow. 

Why not do all of these calls asynchronously, wait for all of the results to filter in, and then update the UI?  This should surely be faster than a serial operation.  Let's say, we spawn Math.Min(nodes.Count, 10) threads to accomplish this task.  So at a maximum, we will use 10 threads to do the work. If we have 20 nodes, it should take at the longest 2 times as long as the longest running call. 

The usual issue with this is that it requires you to use several call-back methods, do complicated thread monitoring, and subsequently put the UI into a wait state until all of the external threads have finished.  Unfortunately there just is no simple way to do it.

So I got to thinking, what construct can I use to signal an asynchronous block, that would prevent execution beyond it's bounds until all the threads were done.  The answer, IDisposable.  If I created an IDisposable object and wrapped it in a "using" I could prevent the Dispose method from exiting until all the threads have finished.

Utilizing the rest of my Threading objects, this should be fairly simple to accomplish, all I had to do was add a done flag to my workerbee:

public class AsyncBlock : IDisposable
{


        long timeoutMS = 0;
        Dispatcher dispatcher = null;
        bool started = false;

        public AsyncBlock() : this(10) { }

        public AsyncBlock(int timeoutSeconds) : this(timeoutSeconds, 5) { }

        public AsyncBlock(int timeoutSeconds, int threads) : this(new Dispatcher(threads), timeoutSeconds) { }

        public AsyncBlock(Dispatcher dispatcher, int timeoutSeconds)
        {
            this.dispatcher = dispatcher;
            this.timeoutMS = timeoutSeconds * 1000;
            this.WorkQueue = new List<WorkerBee>();
        }


        private IList<WorkerBee> WorkQueue
        {
            get;
            set;
        } 


       public IList<WorkerBee> GetWorkers()        
       {
            return new List<WorkerBee>(this.WorkQueue);
        } 


        public void AddToQueue(WorkerBee bee)
        {
            this.WorkQueue.Add(bee);
            this.dispatcher.AddToQueue(bee);
            started = true;
        }


        public void Start()
        {
            foreach (WorkerBee bee in this.WorkQueue)
                if (!bee.Done)
                    this.dispatcher.AddToQueue(bee);


            started = true;
        }


        public void Dispose()
        {
            if (!started && WorkQueue.Count != 0)
            {
                throw new Exception("Work was never started");
            }


            System.Diagnostics.Stopwatch watch = new System.Diagnostics.Stopwatch();
            watch.Start();
            bool done = false;

            while (!done)
            {
                done = true;
                foreach (WorkerBee bee in WorkQueue)
                {
                    if (!(done = done && bee.Done))
                        break;
                }
 


                if (watch.ElapsedMilliseconds > timeoutMS)
                    throw new TimeoutException("Timeout elapsed");
 
                if (!done)
                    Thread.Sleep(1); // wait 1ms to try again
            }
 
            watch.Stop();
        }
}


public class DatabaseWorkerBee<RefType, ReturnType> : AnyWorkerBee<Delegate>
{
        public delegate ReturnType DBDelegate();
        private DBDelegate dbDelegate;

        public ReturnType ReturnValue
        {
            get;
            set;
        } 


        public RefType ReferenceObject
        {
            get;
            set;
        } 


        public DatabaseWorkerBee(RefType refObject, DBDelegate del)
        {
            this.ReferenceObject = refObject;
            dbDelegate = del;
        } 


        protected override void WorkStart()
        {
            base.WorkStart();
 
            if (dbDelegate != null)
            {
                ReturnValue = dbDelegate();
            }
        }
    }


 

Alright, this is now a fairly simple implementation in  the UI:

  

IList<WorkerBee> workers = null;


using (AsyncBlock block = new AsyncBlock(30, Math.Min(nodes.Count, 10))) // do these async with a 30 second timeout
{
      foreach (TreeNode o in nodes)
      {
          TreeNode node = o; //must declare this in the loop, or you get the same object in every worker, this is a documented closure bug in the framework
          block.AddToQueue(
              new DatabaseWorkerBee<TreeNode, int>( node, delegate { /* Do Database Call Here using the node object*/; }));
       }


      workers = block.GetWorkers()


}//using will not return until all workers have been processed.


foreach (WorkerBee worker in workers)
{
      DatabaseWorkerBee<TreeNode, int> dbworker = worker as DatabaseWorkerBee<TreeNode, int>;
      dbWorker.ReferenceObject.Text = String.Format("{0} ({1})", dbWorker.ReferenceObject.Text, dbWorker.ReturnValue); //update the UI
}


 

Woot!  All of the nodes are executed async, and all of the values are returned properly.  One thing to note, notice I am declaring a redundant TreeNode object inside the loop.  This is necessary, since we are using an anonymous method inside of the loop.  If we were to just reference the "o" in the iterator, we would get the same object passed to each anonymous method. 

My Threading Objects can be found here: ThreadLib.zip

[Edit]
After further review and testing this code works quite well, except in the instance of windows authenticated database connections.  Because we are spawning new threads, and your security context is thread based, all of the subsequent database calls were made using the applications default user.  This isn't a problem in Windows UI's but in Web apps this is not good.  It could be as simple as setting the new threads to the same security context and the spawning thread, but I'm not sure I want that to happen for every thread.  I will work this out and do an update.

Posted by Brian Rudolph | with no comments
Filed under: ,
Wednesday, November 26, 2008 2:48 PM

More Threading Goodness!

Threading - I write about it a lot, even though i would consider myself somewhat of a rookie when it comes to threading complexities.

One of the challenges we often find when trying to incorporate threading into applications is managing work units, thread pools, and queues.  What I've ended up doing is writing a few helpers to accommodate just these things.

To start, we need to manage a pool of available thread resources.  This pool needs to have a limit, and needs to be able to generate threads on the fly if necessary.  However, the need for resource pools is not limited to threading.  Consider the scenario where you have a web service of some sort, being WSE, FTP HTTP whatever.  There is a lot of overhead in opening and handshaking for these connections.  If I had say a list of 100k files I needed to upload, and wanted to do that asynchronously, I wouldn't want to throw away the connection after every upload.  So I need to do simple connection pooling.  But I don't want to open 10 connections immediately if I am only going to need 2 either, so this pool needs to be dynamic. 

To even start writing a pool, we need some sort of queuing class for resource storage, and in the threaded world, this needs to be thread safe.

So a simple wrapper may look something like this:

public class ThreadSafeQueue<T>
    {
        //This is the internal queue that we are wrapping
        Queue<T> queue = new Queue<T>();
 
        [NonSerialized]
        ReaderWriterLockSlim objLock = Locks.GetLockInstance(LockRecursionPolicy.NoRecursion); //setup the lock;
 
        public int Count
        {
            get
            {
                using (new ReadOnlyLock(this.objLock))
                {
                    return this.queue.Count;
                }
            }
        }
 
        public void Clear()
        {
            using (new WriteLock(this.objLock))
            {
                this.queue.Clear(); ;
            }
        }
 
        public bool Contains(T item)
        {
            using (new ReadOnlyLock(this.objLock))
            {
                return this.queue.Contains(item);
            }
        }
 
        public bool TryDequeue(out T obj)
        {
            using (new ReadLock(this.objLock))
            {
                if (this.queue.Count != 0)
                {
                    obj = this.Dequeue();
                    return true;
                }
            }
 
            obj = default(T);
            return false;
        }
 
        public T Dequeue()
        {
            using (new WriteLock(this.objLock))
            {
                return this.queue.Dequeue();
            }
        }
 
        public void Enqueue(T item)
        {
            using (new WriteLock(this.objLock))
            {
                this.queue.Enqueue(item);
            }
        }
 
        public T Peek()
        {
            using (new ReadOnlyLock(this.objLock))
            {
                return this.queue.Peek();
            }
        }
    }


 
Then we can move on to the Resource Pool, which we want to be generic so that it can serve up any kind of resource, be it a thread, a web service connection, a widget of any sort.

public class ResourcePool<T> : IDisposable
        where T : class
    {
 
        public event EventHandler CreatedResource;
        public delegate T CreateResourceDelegate();
        public delegate bool CanReuseResourceDelegate(T obj);
 
        public ResourcePool(CreateResourceDelegate creatorDelegate, CanReuseResourceDelegate checkerDelegate)
        {
            this.MaxResourceCount = 10;
            this.IterationTimeout = 5;
            this.MaxRetry = 100;
            this.Creator = creatorDelegate;
            this.CheckResource = checkerDelegate;
        }
 
        private CreateResourceDelegate Creator { get; set; }
 
        private CanReuseResourceDelegate CheckResource { get; set; }
 
        /// <summary>
        /// Retreives a resource from the pool.
        /// </summary>
        /// <returns></returns>
        public T Get()
        {
            T obj = null;
            try
            {
                int tries = 0;
                while (obj == null && (this.MaxRetry == 0 || tries++ <= this.MaxRetry))
                {
                    if (Queue.Count == 0)
                    {
                        if (this.MaxResourceCount == 0 || (this.MaxResourceCount > this.ActiveResourceCount))
                        {
                            if (this.Creator == null)
                                throw new NullReferenceException("Unable to create new Resource Instance");
 
                            obj = this.Creator();
                            FireEvent(ref this.CreatedResource);
                        }
                        else
                        {
                            Thread.Sleep(this.IterationTimeout); // wait for one to become available if we can't create one.
                        }
                    }
                    else
                    {
                        obj = Queue.Dequeue();
                    }
                }
            }
            finally
            {
                lock (_ActiveResourceCountLock)
                    this.ActiveResourceCount++;
            }
 
            if (obj == null)
                throw new NullReferenceException("Unable to retrieve Resource Instance");
 
            return obj;
        }
 
        /// <summary>
        /// Returns a resource to the pool.
        /// </summary>
        /// <param name="obj"></param>
        public void Release(T obj)
        {
            try
            {
                if (this.MaxResourceCount == 0 || Queue.Count < this.MaxResourceCount)
                {
                    if (this.CheckResource == null || this.CheckResource(obj))
                    {
                        Queue.Enqueue(obj);
                    }
                }
            }
            finally
            {
                lock (_ActiveResourceCountLock)
                    this.ActiveResourceCount--; //decrement counter so we know we can reuse the resource.
            }
        }
 
        private int ActiveResourceCount { get; set; }
        object _ActiveResourceCountLock = new Object();
 
        /// <summary>
        /// This is the maximum number of resources the pool will hold and subsequently attempt to create.  Default is 10.
        /// Set to 0 enables no limit.
        /// </summary>
        public int MaxResourceCount { get; set; }
 
        /// <summary>
        /// This is the maximum number of times the pool will look for or attempt to create a resource.  Default is 100.
        /// Set to 0 enables no limit.
        /// </summary>
        public int MaxRetry { get; set; }
 
        /// <summary>
        /// This is the length of time(in milliseconds) that the pool will wait between tries to get a resource. Default is 5ms.
        /// </summary>
        public int IterationTimeout { get; set; }
 
        private ThreadSafeQueue<T> Queue
        {
            get { return this._Queue; }
        }readonly ThreadSafeQueue<T> _Queue = new ThreadSafeQueue<T>();
 
        private void FireEvent(ref EventHandler targetEvent)
        {
            Eventing.FireEvent(ref targetEvent, this);
        }
 
        public void Dispose(bool disposing)
        {
            if (disposing)
            {
                this.Queue.Clear();
            }
        }
 
        public void Dispose()
        {
            this.Dispose(true);
        }
    }


 
 
I know I am tearing through this quickly, but it's a lot to explain.
 
Now we need to have some sort of managed work unit that the thread can work on.  We want it's work unit to be generic and should be easily derived from:
 

public abstract class WorkerBee
    {
        public WorkerBee() { }
 
        public abstract void Start();
    }
 
    public abstract class WorkerBee<T> : WorkerBee
    {
        public T WorkUnit { get; set; }
    }
 
    /// <summary>
    /// This worker can be used as it.  Subscribe to the WorkBegan and WorkFinished events to inject logic.
    /// </summary>
    /// <typeparam name="T"></typeparam>
    public class AnyWorkerBee<T> : WorkerBee<T>
    {
        public delegate void WorkerEventHandler(WorkerBee<T> sender);
        public event WorkerEventHandler WorkBegan;
        public event WorkerEventHandler WorkFinished;
 
        protected virtual void OnWorkBegan()
        {
            if (this.WorkBegan != null)
                this.WorkBegan(this);
        }
 
        protected virtual void OnWorkFinished()
        {
            if (this.WorkFinished != null)
                this.WorkFinished(this);
        }
 
        protected virtual void WorkStart()
        {
            this.OnWorkBegan();
        }
 
        protected virtual void WorkFinish()
        {
            this.OnWorkBegan();
        }
 
        public virtual void DoWork() { }
 
        public override void Start()
        {
            this.WorkStart();
            this.DoWork();
            this.WorkFinish();
        }
    }


 
Now we can look at the actual dispatcher, we need some way to add work to a queue, have it spawn a thread if it has any left, and then execute the work.  One of the tricky things was to make sure the dispatcher thread was not ALWAYS running.  It would start itself up anytime work was added, continue running and waiting until it's work queue was empty, then shut down.  Every subsequent addition to the queue would ensure that the dispatcher was still running.
 

 
public class Dispatcher : IDisposable
    {
        public Dispatcher(int maxThreadCount)
        {
            this.Threads = new ResourcePool<Thread>(
                new ResourcePool<Thread>.CreateResourceDelegate(this.CreateThread),
                new ResourcePool<Thread>.CanReuseResourceDelegate(this.ValidateThread));
 
            this.Threads.MaxRetry = 0;
 
            this.WorkQueue = new ThreadSafeQueue<WorkerBee>();
 
            this.Threads.MaxResourceCount = maxThreadCount;
            this.EnsureDispatch(false); // setup the dispatcher
        }
 
        /// <summary>
        /// Call this to stop the dispatcher
        /// </summary>
        public void Abort()
        {
            this.AbortDispatch = true;
        }
 
        /// <summary>
        /// Call this to pause the dispatcher
        /// </summary>
        public void Pause()
        {
            this.AbortDispatch = true;
        }
 
        /// <summary>
        /// Call this to resume the dispatcher
        /// </summary>
        public void Resume()
        {
            if (this.AbortDispatch)
            {
                this.AbortDispatch = false;
                this.EnsureDispatch();
            }
        }
 
        /// <summary>
        /// Set this to prevent the dispatcher from spawning threads.
        /// </summary>
        private bool AbortDispatch
        {
            get;
            set;
        }
 
        /// <summary>
        /// This is the thread that will dispatch work to other threads.
        /// </summary>
        private Thread DispatchThread
        {
            get;
            set;
        }object _DispatcherLock = new Object();
 
        /// <summary>
        /// This will make sure that the Dispatcher is active and running.
        /// </summary>
        private void EnsureDispatch()
        {
            this.EnsureDispatch(true);
        }
 
        /// <summary>
        /// This will make sure that the Dispatcher is active and running.
        /// </summary>
        private void EnsureDispatch(bool start)
        {
            lock (this._DispatcherLock)
            {
                if (this.DispatchThread == null || this.DispatchThread.ThreadState == ThreadState.Stopped)
                {
                    this.DispatchThread = new Thread(new ThreadStart(this.FireThread));
                    this.DispatchThread.IsBackground = true;
                }
 
                if (start && ((this.DispatchThread.ThreadState | ThreadState.Unstarted) == this.DispatchThread.ThreadState))
                    this.DispatchThread.Start();
            }
        }
 
        /// <summary>
        /// This is a required callback for the Resource pool.
        /// </summary>
        /// <param name="obj"></param>
        /// <returns></returns>
        private bool ValidateThread(Thread obj)
        {
            return false; //never reuse threads
        }
 
        /// <summary>
        /// This is the method that the resource pool will call to get a new thread to work with.
        /// </summary>
        /// <returns></returns>
        private Thread CreateThread()
        {
            return new Thread(new ParameterizedThreadStart(this.DispatchWork));
        }
 
        /// <summary>
        /// The function starts the work on a new thread.
        /// </summary>
        private void DispatchWork(object state)
        {
            try
            {
                WorkerBee work = state as WorkerBee;
 
                if (work != null)
                    work.Start();
            }
            finally
            {
                //Give the thread back.
                this.Threads.Release(Thread.CurrentThread);
            }
        }
 
        /// <summary>
        /// This puts work on the queue and starts the dispatcher
        /// </summary>
        /// <param name="obj"></param>
        public void AddToQueue(WorkerBee obj)
        {
            this.WorkQueue.Enqueue(obj);
            this.EnsureDispatch();
        }
 
        private int GetQueueLength()
        {
            return this.WorkQueue.Count;
        }
 
        /// <summary>
        /// This is the callback that the dispatcher thread calls.
        /// It will initialize a new thread from the pool and start work
        /// </summary>
        private void FireThread()
        {
            WorkerBee work = null;
            while (!AbortDispatch && this.WorkQueue.TryDequeue(out work))
            {
                if (work != null)
                {
                    Thread thread = this.Threads.Get();
                    
                    if (!AbortDispatch) // check again
                        thread.Start(work);
                    else
                        this.Threads.Release(Thread.CurrentThread);
                }
            }
        }
 
        /// <summary>
        /// This is the queue of stuff to be done.
        /// </summary>
        private ThreadSafeQueue<WorkerBee> WorkQueue
        {
            get;
            set;
        }
 
        /// <summary>
        /// Simple resource pool filled with threads
        /// </summary>
        private ResourcePool<Thread> Threads
        {
            get;
            set;
        }
 
        public void Dispose(bool disposing)
        {
            if (disposing)
            {
                this.WorkQueue.Clear();
                this.Threads.Dispose();
            }
        }
 
        public void Dispose()
        {
            this.Dispose(true);
        }
    }

ThreadingDemo.zip


Download the code and test project and try it for yourself.  I will say that this is fairly experimental, although I have been using it for quite a few things and have experienced no issues.  One of the major things I have used it for is the cache scavenger in my Factory caching model and it seems to work great.

 

Posted by Brian Rudolph | with no comments
Filed under: ,
Tuesday, October 21, 2008 10:26 AM

Work Harder

Work Harder!  That is a statement I find myself repeating daily.  People often wonder what makes one person better at a certain task than others.  There is a truly simple answer, practice. 

Natural ability is a myth.  With the exception of physical limitations, all ability is gained through practice.  No infant is born with the innate ability to walk.  No musician picks up an instrument for the first time and performs a flawless number.  Everything ‘great' is preceded by countless not-so-great attempts. 

The human race is a term that to some is a simple definition of our status as homosapiens.  To me, it is a bit more literal.  I recognize that there are billions of people in this world, all of whom want the same things I do.  What sets certain individuals apart from the pack is their willingness to work harder than the rest.  So for me, it's the race of humans.  Who wants to work harder?  Who wants to spend that extra effort day after day to be the best at what they do.  Someone who doesn't work as hard as I do does not deserve the same things I deserve, just as I do not deserve the same benefits of the people who work harder than I am willing to. 

I hear complaints from people about the state of their lives, every argument of which never focuses on their lack of effort.  It is always the result of some innocuous outside influence.  Bunk.  People who work hard, achieve more.  Good things happen to good people who work hard.  Being good isn't enough.  There are lots of "good" people, if the definition of "good" can be simplified to "not being bad".  Spend a few more hours with your face buried in a book rather than a pint of beer.  Unless, of course, your goal is to be the best drinker on earth.    

Everything that has gone right in my life happened to me when I went the extra mile.  Every real accomplishment didn't come from the 8 to 5 daily grind.  Good things happened during the hundred-hour weeks of work.  Times when 8 to 5 wouldn't have been enough to get the job done, allowing the opportunity to pass right by. 

There have been many times I have explained to people that epiphanies have come to me in my sleep.  Though I was not lying, this is not a result of me having some abnormal super power.  It is a result of me wrapping my mind so deeply around a problem that I am consumed by it even during my few hours of sleep. 

I like to think that everything that has happened to me in my life, I've earned.  This, of course, includes all of the bad things.  For many years my interests were simply in having fun, so my career took a nose-dive.  Then I focused entirely on work which destroyed countless relationships, as well as my physical health. So while balance has become important, I have recognized that every decision comes with consequences; consequences which I hope to never complain about.

People make choices in their lives.  If your choice is to pursue a career or path in your life that will never financially stabilize, you have no right to complain about finances.  You have chosen to sacrifice financial success for your love or interest in what it is you do.  It is not society's responsibility to fill in for your financial woes.  If you choose to sacrifice your personal life for professional glory, then do not cry about your lack of intimacy.  You chose that path.  Just realize that everything requires hard work.  Even if an individual's goal in life is to be the best parent their child could have, then they must work for it. 

This is not to say that I am better or worse than anyone else in this world.  What I am attempting to say is that I take responsibility for what happens in my life.  I will not give up, I will not rest until I am the best I can be at what I do, regardless of what this life throws at me.  The best I can be at anything will require a lifetime of effort.  So I suppose it is a valid assumption to say that I will only achieve this in the moment immediately before my death. 

I truly wish more people would take responsibility for their lives.

Posted by Brian Rudolph | 5 comment(s)
Monday, September 29, 2008 11:14 AM

Thread Safe Dictionary Update

 

For those of you that have read my post about the Thread Safe Dictionary, I have a few updates. 

In recent months, I have done a ridiculous amount of multi-threaded work.  This has forced me to expand my threading libraries.  One of the primary objects that required changes\fixes was my dictionary. 

 

The dictionary uses ReaderWriterLockSlim locks.  These are clean, lightweight locks that MS has provided.  Most importantly, they allow upgradeable read locks.  I now use simple IDisposable locking wrappers.  This allows us to use the "using" syntax to ensure lock releases.

It is important to note that no safe dictionary can provide safe inserts without the caller forcing a lock.  This is because by the time you check for the existence of an object and then insert it, it may have been inserted by another thread.  The only way to prevent this is by doing a "blind" merge.  The reason I call it blind, is that when you do a merge, we check for existence, and do a replace\insert within the scope of a single write lock.  This means you cannot control which objects get replaced in the dictionary.  This is rarely important in caching situations.

using System;
using System.Collections;
using System.Collections.Generic;
using System.Threading;


public interface IThreadSafeDictionary<TKey, TValue> : IDictionary<TKey, TValue>
{
    /// <summary>
    /// Merge is similar to the SQL merge or upsert statement.  
    /// </summary>
    /// <param name="key">Key to lookup</param>
    /// <param name="newValue">New Value</param>
    void MergeSafe(TKey key, TValue newValue);


    /// <summary>
    /// This is a blind remove. Prevents the need to check for existence first.
    /// </summary>
    /// <param name="key">Key to Remove</param>
    void RemoveSafe(TKey key);
}


[Serializable]
public class ThreadSafeDictionary<TKey, TValue> : IThreadSafeDictionary<TKey, TValue>
{
    //This is the internal dictionary that we are wrapping
    IDictionary<TKey, TValue> dict = new Dictionary<TKey, TValue>();


    [NonSerialized]
    ReaderWriterLockSlim dictionaryLock = Locks.GetLockInstance(LockRecursionPolicy.NoRecursion); //setup the lock;


    /// <summary>
    /// This is a blind remove. Prevents the need to check for existence first.
    /// </summary>
    /// <param name="key">Key to remove</param>
    public void RemoveSafe(TKey key)
    {
        using (new ReadLock(this.dictionaryLock))
        {
            if (this.dict.ContainsKey(key))
            {
                using (new WriteLock(this.dictionaryLock))
                {
                    this.dict.Remove(key);
                }
            }
        }
    }


    /// <summary>
    /// Merge does a blind remove, and then add.  Basically a blind Upsert.  
    /// </summary>