I’ve been wanting to blog about this for a while, but haven’t as it seems it might be time consuming to explain my thoughts. I’m going to dip into it a bit here and then follow up as I have time.
The company I work for provides data analysis for retail stores (read grocery) found throughout the country. The companies we work for are fairly small which is to say chains like Albertsons, King Soopers, Safeway, etc. are much larger than what we handle. We would like to see that change and I think it will in the future, however, the largest chain we work with currently has fewer than 30 stores total.
Part of being able to use the same data analysis tools for all of these different companies has meant being able to take transaction data (data from scans at the cash register stored in log files) and convert it to a format that can be easily loaded into the database schema we use when running our analysis tools.
I was tasked with writing the extraction processes for each of the different Point of Sale (POS) systems that we support and converting that data from raw log files to the format (XML) that can be loaded into our database schema.
As you might have guessed from the title of this post, I use a pluggable framework that allows me to select which ‘extractor’ to use at run time based upon the type of POS system the store is using. Keep in mind that we need to support numerous POS systems since these smaller companies all use a broader range of systems than what you might find at the large corporate site I mentioned earlier. Also, though it doesn’t happen too much, there may even be cases where within a single company or single site (store), there might be a mix of POS systems in use. That’s why it’s important to be able to accommodate multiple types of POS systems.
I am using reflection for my pluggable framework. I’ll get to the details of that in a minute, but first let me talk about how to organize pluggable components.
Namespace Organization
In order to adhere to good programming practices, I created an abstract base class that contains some common functionality that all transaction log extraction modules will benefit from. I placed that class in it’s own assembly, though, which allows me to create new extractor any time I need to without having to have a reference to the other modules. Organized into namespaces, it looks like this:
CompanyName.ApplicationName.Extractors
CompanyName.ApplicationName.Extractors.SuperPOS
CompanyName.ApplicationName.Extractors.HyperPOS
CompanyName.ApplicationName.Extractors.WonderPOS
Each of these are in their own projects and therefore each of them create their own assembly. Each of the specific extractors has a reference to the CompanyName.ApplicationName.Extractor namespace and inherit from a class in there called “ExtractorBase”. Keep in mind that all of my namespaces here are bogus. In the real application they have their actual names, but I didn’t really want to give away anything here about the real POS systems we support.
The reason I’ve done things this way is because I want to be able to create new projects in the future that adhere to the interface (abstract class) without requiring me to recompile anything but the new assembly. So, say in the future I decided to add a new extractor with the namespace:
CompanyName.ApplicationName.Extractors.CoolPOS
I can simply create a new project that has a reference to CompanyName.ApplicationName.Extractors, inherit from CompanyName.ApplicationName.Extractors.ExtractorBase and implement the interface specified. Then, when I’ve compiled the assembly, I can simply move CompanyName.ApplicationName.Extractors.CoolPOS.dll into my extractors directory and then the existing system will be able to simply load the new assembly using reflection based on some parameters I will add to a configuration XML file.
Reflections Of the Way Life Used to Be
Do you remember life before we had reflection? RTTI (RunTime Type Identification) was the closest I recall in C++. These days I can’t imagine how I could have done anything like this so easily without it. In the code that actually calls each extractor, I use reflection to find and invoke the method. As part of my interface (abstract class), I’ve provided a single entry point that has multiple implementations (e.g. polymorphism). The method name is called Execute and it provides multiple signatures in case some parameters are not needed or desired. The method invocation code is able to determine based on the number of parameters passed in which Execute method should be called.
The method invoker uses strings that specify the full namespace of the object that is currently needed and able to load the proper assembly. It then calls the execute method on the extractor module found in that assembly. The code looks something like this:
[csharp]
string extractorAssemblyName =
configManager.GetValue( “ApplicationPath” ) +
“CompanyName.ApplicationName.Extractors.” +
this.extractorName + “.dll”;
string extractorObjectName =
“CompanyName.ApplicationName.Extractors.” +
this.extractorName + “.” +
this.extractorClassName;
DynamicMethodInvoker invoker = new DynamicMethodInvoker(
extractorAssemblyName,
extractorObjectName,
“Execute” );
[/csharp]
The strings extractorName and extractorClassName were populated from an XML configuration file. In the case of my new POS system I referred to above as CompanyName.ApplicationName.Extractors.CoolPOS, the string extractorAssemblyName would now contain “C:\path_to_application\CompanyName.ApplicationName.Extractors.CoolPOS.dll” and extractorObjectName would now contain “CompanyName.ApplicationName.Extractors.CoolPOS.CoolPOSExtractor” (which is the actual name of the class we are going to use for this POS).
All I need to do at this point is pass in the parameters that the Execute() method is expecting and call Invoke() like this:
[csharp]
invoker.AddParameter( param1 );
invoker.AddParameter( param2 );
invoker.AddParameter( param3 );
invoker.Invoke();
[/csharp]
And my Execute method for the new extractor, CompanyName.ApplicationName.Extractors.CoolPOS is now running without having to recompile my entire calling application.
At this point, I can’t remember where I got the DynamicMethodInvoker class from, but if you are interested in taking a look at it and using it, just let me know and I’ll forward it on to you (I believe I got it off of MSDN, but heck if I can find a reference to it now). DynamicMethodInvoker provides a simple interface for calling methods and examining reflected types and methods. It makes things a little bit simpler to implement.
Conclusion
I haven’t yet done any profiling to see if things would be much faster if I didn’t use reflection (e.g. just create a hard coded class hierarchy), however, the system does seem to work well and fast enough to accomplish what we need. I had thought about using a provider model, but I realized later that the provider model is really intended for creating an abstraction in case you need to change the underlying system at a later date. A data provider for instance provides access to different databases based upon a specific implementation of a provider interface. If you ever needed to switch from a SQL Server database to a MySQL database, for example, you would simply need to implement the provider interface for the MySQL database and the specify in a config file that you want to use that provider instead.
The approach I’ve outlined here works similarly, however, in contrast to the provider model this pluggable framework is intended to swap out the current extractor in use during a single session where the data provider model is simply a design choice you make in order to support a different underlying data store should you need to in the future.
Well, that’s it for now. I will revisit this again as more comes to me. Meanwhile, good luck and have fun making pluggable frameworks with reflection.