Tag Archives: source code

Easier Complex IComparable Implementations

I know a lot of us are using the LINQ OrderBy() method to get our data shuffled in the right order, but on occasion I still do like implementing IComparable, especially when defining the default, intrinsic sort scheme for a particular class.

What I don’t like is implementing IComparable when I want to compare on more than one thing.

I don’t like this kind of code because the engineer in me balks at essentially doing each comparison more than once, first for equality, and then for direction:

public int CompareTo(SomeObject other)
{
	if (this.Prop1 != other.Prop1)
		return this.Prop1.CompareTo(other.Prop1);
	else if (this.Prop2 != other.Prop2)
		return this.Prop2.CompareTo(other.Prop2);
	// .... etc.
	return 0;
}

However, expanding it to get rid of this double evaluation yields this nastiness:

public int CompareTo(SomeObject other)
{
	int cmp = this.Prop1.CompareTo(other.Prop1);
	if (cmp != 0)
	{
		cmp = this.Prop2.CompareTo(other.Prop2);
		if (cmp != 0)
			return cmp;
	}
	return 0;
}

Yuck. I decided I needed a way to have some of the elegance of LINQ in an IComparable shell, and here it is:

public class ComplexCompare
{
	private int value;

	private ComplexCompare()
	{
	}

	public static ComplexCompare By<T>(T a, T b)
	{
		return By(a, b, true);
	}

	public static ComplexCompare By<T>(T a, T b, bool ascending)
	{
		ComplexCompare cc = new ComplexCompare();
		if (ascending)
			cc.value = Comparer<T>.Default.Compare(a, b);
		else
			cc.value = Comparer<T>.Default.Compare(b, a);
		return cc;
	}

	public static ComplexCompare By<T>(T a, T b, IComparer<T> comparer)
	{
		ComplexCompare cc = new ComplexCompare();
		cc.value = comparer.Compare(a, b);
		return cc;
	}

	public ComplexCompare ThenBy<T>(T a, T b)
	{
		return ThenBy(a, b, true);
	}

	public ComplexCompare ThenBy<T>(T a, T b, bool ascending)
	{
		// Only compare more specific items if the preceding items have been equal
		if (value == 0)
		{
			if (ascending)
				this.value = Comparer<T>.Default.Compare(a, b);
			else
				this.value = Comparer<T>.Default.Compare(b, a);
		}
		return this;
	}

	public int End()
	{
		return value;
	}
}

ComplexCompare can be used like this:

public int CompareTo(SomeObject other)
{
	return ComplexCompare.By(this.Prop1, other.Prop1) // ascending by default
		.ThenBy(this.Prop2, other.Prop2, false) // but easy to change to descending
		.ThenBy(this.Prop3, other.Prop3) // each call can compare a completely different type
		.End(); // stops the fun and returns the int value
}

Now a hardcore computer science person would tell me that all this extra abstraction adds overhead, and in a really tight loop with millions of rows, using this scheme to compare items would surely be catastrophic! However, I’m not usually one to give in to Micro-Optimization Theater; I’m usually comparing 30-40 items, not millions. My primary concern is that code I write can be instantly understood when viewed by one of my peer developers, and I think ComplexCompare (although I’m not in love with the name) will help me to do that.

Subversion to Mercurial Conversion App

On my team, we use FogBugz for case/defect tracking and project management, and we absolutely love it.

So when the folks over at Fog Creek created Kiln for version control with integrated code review, we had to try it out, and once I tried it, I knew we had to have it.

Of course, Kiln is based on Mercurial, and in order to convert, we have a fairly large Subversion repository that we need to convert. At least, at over 4000 revisions and still growing, it feels pretty big to us.

(If you have no idea what Kiln or Mercurial is all about, I encourage you to check out Hg Init, a tutorial on Mercurial written by Joel Spolsky, which is extremely informative and well written.)

Fog Creek provides an import tool that is pretty impressive, since it is able to do import from half a dozen different existing source control systems (including just a pile of files on disk) and does it pretty well. However, it didn’t meet our needs in a few areas:

  • It doesn’t take advantage of author mapping offered by the hg convert utility. Our current Subversion server (VisualSVN Server) uses our Windows credentials, which are based on an archaic corporate naming scheme, so we really would like to map these to our real names.
  • The conversion is imperfect, resulting in some directories and files that aren’t in the head of the Subversion trunk turning up in the converted Mercurial repository. These are directories that existed long ago, before a massive repository reorganization moved them elsewhere. In any case, we don’t want them, and we don’t necessarily want to hunt them down manually.
  • The conversion doesn’t convert Subversion’s svn:ignore properties into an .hgignore file. When you’re developing with Visual Studio 2008 and a large solution, that results in a bunch of untracked bin and obj directories.
  • Although this turned out not to be important for us, the importer does not allow carving a large Subversion repository into a bunch of smaller subrepositories.

I developed a Windows Forms application in C# to handle our conversion, and learned a lot about the command-line svn and hg commands along the way. I offer it to you in the hopes that you might find it useful, because there is no greater tragedy than code that is written and only executed once.

I warn you, the application is NOT what I would call polished source code or UI. It’s REALLY rough around the edges. It was designed experimentally to get a job done that only had to be done once. Niceties were not observed. Sorry! And of course I make no warranties that it will work for you, AT ALL!

That said, here’s the link:

  • HgImport, Version 1.0 - A Visual Studio 2008 project for an SVN to Mercurial import application.
  • Requires the pre-.Net-4.0 Task Parallel Library extensions

Here’s how to use it:

  • You must enable the hg convert extension:
    • Edit your .hgrc file in your user folder … which if you’re using Kiln on Windows, may be called Mercurial.ini instead.
    • Find the [extensions] section of the file, and add the line “hgext.convert=” (without the quotes) underneath.
    • All this does is enables the convert command on the command line. See the Mercurial ConvertExtension documentation for further details.
  • Build and run the application.
  • Switch to the Author Map tab and enter your username mappings. Each line should take this format:
    • oldusername = Firstname Lastname
  • I don’t save that text anywhere. Copy and paste it into a text file for safekeeping. Now switch tabs back.
  • Enter your Subversion repository trunk url in the text box. (Sorry, we are just converting our trunk. All our branches are dead and we want to be rid of them. You’re welcome to change the code if you like!)
  • Click the Load SVN button. Sorry if your repository requires login. My app doesn’t support it. I warned you it was rough around the edges! You can either modify the source or do what I did: use the svnsync command to sync your central SVN repository server onto a mirror repository on your local computer. This has the added benefit of speeding the conversion process by taking the network out of the equation.
  • Check any directories that you want to convert into subrepositories. (Since I abandoned this feature, this is NOT well tested! Especially the .hgignore doesn’t take subrepositories into account.)
  • Click Convert and then go get a snack. I estimate approximately one snack per thousand SVN revisions, or one full meal per 5000 revisions. Just ballpark figures.

Here’s what the conversion process does:

  • When you load the SVN repository, it loads the SVN head file structure into an XML document in memory.
  • When you begin conversion, first it clears out the working directory. (“working” under the execution directory)
  • Next, the app performs a full convert of the SVN repository using “hg convert”, similar to what the Kiln import tool does, although the authors are mapped at this point.
  • Next, the app updates the Mercurial repository with “hg up” to essentially establish the working copy.
  • Next, the Mercurial working directory is compared recursively with the SVN-exported XML file structure. This creates a filtering list that eliminates the phantom directories and files that no longer exist in the SVN working copy.
  • Next, the intermediate Mercurial repository is converted to the final Mercurial repository using “hg convert”, utilizing the previously created filter, and then the final repository is updated with “hg up”.
  • If you selected subrepositories, the filtering file will also exclude those directories, and then the app will attempt to repeat the process of converting each subrepository out of the transitional repository. Like I said, I gave up on this feature and this will definitely not work with .hgignore, if it even works at all!
  • Lastly, the app will query the SVN repository for svn:ignore properties, and translate these into an .hgignore file in the root of your new Mercurial repository. Then this file will be added with “hg add” and committed with “hg commit”.

At this point, {AppWorkingDirectory}\working\FilteredRepo will be a fully functioning Mercurial repository! If you want to stop there, great, or it’s ready to push up to Kiln.

That’s it. It’s not pretty, but it works for us and hopefully it will work for you, or at least serve as a starting point. Enjoy!

NServiceBus TimeoutManager Revisited

In my previous post I attempted to build an NServiceBus Timeout Manager that used timers and events to send back timeout messages when they came due instead of looping through the message queue with thread sleeps until each message was ready to send back that occurs in the timeout manager included with the NServiceBus 2.0 RTM.

That implementation stored timeouts that were set to expire “soon” in memory, and stored everything in a secondary MSMQ queue so that if the application failed, it could recover when it started back up.

I realized that a better implementation would be to separate out the storage using a provider implementation.  This way, the entire implementation could be switched out completely by providing a second class that implements ITimeoutStorageProvider.

Also, I refactored the MsmqTimeoutStorage class into a base class that’s concerned with the in-memory timeout handling, and the superclass that adds in the actual storage implementation with MSMQ.

This way, switching to database storage would be as easy as creating a class that inherits TimeoutStorageProvider<T> (more on the generic parameter in a bit) and implements the following methods:


public abstract void Init();
protected abstract T StoreTimeout(IMessageContext context, TimeoutMessage msg);
protected abstract void RemoveBySagaId(Guid sagaId);
protected abstract void RemoveTimeout(T id);
protected abstract List&lt;TimeoutEntry&lt;T&gt;&gt; GetTimeouts(DateTime cutoff);

The generic parameter T is the ID unique to the storage provider, used later to remove it from storage after the timeout is complete. For the MSMQ implementation, this is a string because a MSMQ message’s Id parameter is a string. For a database implementation with Microsoft SQL it would probably be advisable to use a Guid.

Here is the source:

To build, simply unzip the project, include it in a solution, fix the references to the NServiceBus components and Log4Net, build, and go.

Feel free to take the code and use it however you like under the terms of the Apache License, Version 2.0.  I offer it without warranty, express or implied, and as always your mileage may vary.  I only ask that you share any improvements you make to it so that everyone can benefit.

Building a better NServiceBus TimeoutManager

Update 4/30/2010: I updated this code to use a provider implementation so that it is easy to switch out the MSMQ storage for database storage or any other storage medium.  Check it out!

Udi Dahan himself has stated that the TimeoutManager included with NServiceBus 2.0 is “not generically suitable for production” purposes, and since I needed to create a system that used a TimeoutManager for multiple sagas in production, I set about the task to create a better one.

For a complete discussion, check out the message thread Timeout Manager process causes heavy disk IO, or for a quicker read, here are some of the limitations of the TimeoutManager included with NServiceBus 2.0:

  • Operates by doing a short thread sleep, and then resending the message back to the same queue until it expires and can be sent back.  This creates a lot of disk I/O in the Microsoft Message Queue system from the message being written back to persistent storage many times per second.
  • In order to achieve better performance, you must increase the number of milliseconds sleep, but this has the side effect of decreasing time resolution.  It might be OK to sleep for 10 seconds if you’re dealing with a timeout of 20 minutes, but it’s unacceptable for a timeout of less than a minute.  What if the Timeout Manager needs to handle both?

Here are some of the considerations I made for a new design:

  • Obviously a timer is needed to raise an event when a timeout is expired instead of looping and sleeping.
  • The messages must still be stored in persistent storage.  Removing the dependency on Message Queue and storing the timeout messages in memory would not be acceptable because a failure in the Timeout Manager would result in the loss of timeout messages.
  • Replacing the MSMQ dependency with a database is not an acceptable solution because it would require excessive pinging of the database.  This would also inject a lot of necessary configuration.
  • While use of a distributed cache would probably solve some issues, I’m unwilling to use one just for this.  I want to get as much bang for my buck as possible using just the requirements that NServiceBus already has (MSMQ, DTC, etc.) without introducing new dependencies.

What I designed is a system that receives TimeoutMessage through a handler on a queue similar to the bundled timeout manager, and then stores that message on a second MSMQ queue.  At the same time, timeouts that are due “soon” (perhaps within 30 minutes) are also kept in memory.

An instance timer is set to go off when the first timeout in memory is due, at which point the corresponding message is removed from the queue and the timeout is sent back to the saga, and the instance timer is reset.

At startup, after a failure, or periodically via another timer, all the messages in the storage queue are rescanned and the in-memory queue is rebuilt.

I don’t believe that this is by any means a perfect system, in fact I believe it has the following weaknesses:

  • There is quite a bit of thread locking any time the in-memory queue is changed.  For a high-traffic system this could be problematic.
  • I have no idea how well this could be utilized in a failover environment.
  • I’m sure the code probably has several weaknesses I haven’t thought of.  I’m far from a messaging expert.

The last point is why I invite you to download the code and bang on it yourself.

To build, simply unzip the project, include it in a solution, fix the references to the NServiceBus components and Log4Net, build, and go.

Feel free to take the code and use it however you like under the terms of the Apache License, Version 2.0.  I offer it without warranty, express or implied, and as always your mileage may vary.  I only ask that you share any improvements you make to it so that everyone can benefit.