While searching for information and ideas on configuration stores, I came across an article on better WMI scripting. Defiantly something I will need to review in the future as I get into automated BizTalk management.

It seems as if there is someone at Microsoft, who's sole purpose in life is to figure out how I might want to use something (WCF), and then make sure I can't use it in the way I want to. Today's issue, programmatically configuring a WCF client application.

Background

I have created a central repository store in the form of a SQL database, that put simply, contains key/value pairs that I can query. All of my applications (or each machine.config) only need to specify a single database connection string to this configuration database.  All other configuration settings are retrieved from the database (and cached), making it easy to keep track of configuration, view all current configuration settings, and deploy the applications on new servers.

Expanding upon my simple definition, there is a second "key", which corresponds to what environment (production, staging, development) the application is deployed. So my single configuration database controls the configuration of all applications in all environments. For those settings which are the same across all environments, no deployment key is specified.

So far this has worked great, and I wanted to add configuration information about a WCF client service.

WCF

WCF needs two properties set, a binding, and an endpoint, and each of these 2 properties have sub properties which may or may not have values that need to be set. Out of the box, WCF is most easily configured using application or web configuration files. All of the tools support the configuration files and it's easy to see what's going on (well sort of).

I basically wanted to take the XML sections that define the binding and endpoint, and place it into my configuration database. For some reason, I got it in my head that this would be really easy to do, but found out that it's not. Searching thru Google yielded no direct solution. While you can programmatically set the Binding and Endpoint properties of a proxy class client object, I would either have to have 1 setting per property (and sub property) in my configuration database, or come up with an XML schema, parse the schema and set the values. This was the path I was pursing, but instead of making my own schema, I just used the XML configuration schema provided out of the box, in hopes that one day I could just pass those sections to binding or endpoint objects and have it parse for me.

After creating classes to parse a single binding (WsHttpBinding), I decided there must be a better way.  The code to parse the configuration files must exist somewhere in System.ServiceModel, so I decided to go spelunking using reflector to see what was going on, and determine if I could hack something together.

Implementation

Overview

After spending a couple of hours in reflector, I realized the main roadblock was the use of the System.Configuration namespace to handle the configuration data for System.ServiceModel. System.Configuration only supports the file system, and provides no extension methods at this time (something I'd like to see from Microsoft in the future).

This left me with one option, which was to create temporary XML file with my configuration data from my database, and then read that configuration file using the proper classes/methods in System.Configuration. It took me awhile to accept this option, as writing a temporary file seemed messy, but then I realized, temporary files are a fact of life in .Net, programming, operating systems, etc.

Configuration Store

My configuration store was already defined as I mentioned previously. The 2 XML samples below are stored as separate entries in my configuration database. Below is the XML I stored for handling the bindings:

<bindings>
 <wsHttpBinding>
  <binding name="WSHttpBinding_ITwoWayAsyncVoid" closeTimeout="00:01:00"
          openTimeout="00:00:30" receiveTimeout="00:02:00" sendTimeout="00:00:15"
          bypassProxyOnLocal="false" transactionFlow="false" hostNameComparisonMode="StrongWildcard"
          maxBufferPoolSize="524288" maxReceivedMessageSize="65536" messageEncoding="Text"
          textEncoding="utf-8" useDefaultWebProxy="true" allowCookies="false">
          <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"
            maxBytesPerRead="4096" maxNameTableCharCount="16384" />
          <reliableSession ordered="true" inactivityTimeout="00:10:00"
            enabled="false" />
          <security mode="Message">
            <transport clientCredentialType="Windows" proxyCredentialType="None"
              realm="" />
            <message clientCredentialType="Windows" negotiateServiceCredential="true"
              algorithmSuite="Default" establishSecurityContext="true" />
          </security>
  </binding>
 </wsHttpBinding>
</bindings>
 

I am able to store multiple bindings, and binding configurations in a single entry in my configuration database (see below). In this example, I only have a single binding and binding configuration. I was originally planning to have 1 entry for each binding, but I don't think that will be necessary, if anything I will have a single entry for all necessary bindings per environment, and even that might be overkill.

This is the XML I stored for the end point:

<bindings>
 <wsHttpBinding>
  <binding name="WSHttpBinding_ITwoWayAsyncVoid" />
 </wsHttpBinding>
</bindings>
<client>
 <endpoint address="http://tersodemodev1/TsiCbsEsbReciever/WcfService_TersoSolutions_CBS_Fusion_BizTalk_InitialProcess.svc"
        binding="wsHttpBinding" bindingConfiguration="WSHttpBinding_ITwoWayAsyncVoid"
        contract="TsiCbsEsbReceiver.WcfService_TersoSolutions_CBS_Fusion_BizTalk_InitialProcess"
        name="WSHttpBinding_ITwoWayAsyncVoid">
        <identity>
          <userPrincipalName value="TERSODEMODEV1\BizTalkWebServices" />
        </identity>
 </endpoint>
</client>

Notice the repeat of the <bindings> section. This was required if I wanted to use the the classes in System.ServiceModel for parsing the endpoint, I needed to define a skeleton for the bindings.

I separated the endpoints from the bindings, as that is what is going to change between environments the most for me. The endpoint specifies which server (test, production, staging), and identity (for single server test scenarios, the account will be local, for multi server production enviroments, the identity will be a domain account).

Code

I created 2 classes under a namespace called ConfigSystem. The first class, called ConfigMgr contains code that allows me to go from my configuration database to a System.Configuration.Configuration object by way of a temporary file. The second class called Wcf, contains code specific to my Wcf configuration implementation, and contains code inferred from reflector.

All of the code in ConfigMgr uses public classes and is pretty simple.

public static class ConfigMgr
 {
  #region MemberVars
  private const string configFileHeader = "<?xml version=\"1.0\"?><configuration>";
  private const string configFileFooter = "</configuration>";
  
  #endregion

  #region Methods - Public

  public static ConfigurationSection GetSection(string configKey,
   string xmlConfigFileHeaderExtra, string xmlConfigFileFooterExtra, string sectionName)
  {
   //Get an xml document from the configuration database to store as a tempory file.
   System.Xml.XmlDocument doc = PrepareXmlDocument(configKey, xmlConfigFileHeaderExtra, xmlConfigFileFooterExtra);

   //Now write out to the temp file.
   string path = System.IO.Path.GetTempFileName();
   using (System.Xml.XmlWriter writer = System.Xml.XmlWriter.Create(path))
   {
    doc.WriteTo(writer);
    writer.Close();
   }
   
   //Setup configuration file map so that we can use standard file based config file
   ExeConfigurationFileMap map = new ExeConfigurationFileMap();
   map.ExeConfigFilename = path;

   System.Configuration.Configuration config = ConfigurationManager.OpenMappedExeConfiguration(map, ConfigurationUserLevel.None);
   
   //Now return the specified section
   return config.GetSection(sectionName);
  }

  private static System.Xml.XmlDocument PrepareXmlDocument(string configKey, string xmlConfigFileHeaderExtra, string xmlConfigFileFooterExtra)
  {
   System.Text.StringBuilder sb = new System.Text.StringBuilder();
   
   //Append header
   sb.Append(configFileHeader);
   sb.Append(xmlConfigFileHeaderExtra);

   sb.Append(ConfigHandler.GetValue(configKey));

   //Append Footer
   sb.Append(xmlConfigFileFooterExtra);
   sb.Append(configFileFooter);

   System.Xml.XmlDocument doc = new System.Xml.XmlDocument();
   doc.LoadXml(sb.ToString());
   return doc;
  }

  #endregion
 }

The code in the Wcf class uses some methods I copied via reflector, and because I don't know the legal ramifications of that, I am only providing an outline of what I did.

GetBinding:

• Calls ConfigMgr.Get section and returns the <bindings> section. This is cast to a BindingsSection object.
• Code taken from System.ServiceModel.Configuration creates a BindingCollectionElement which is the correct type associated with the binding stored in the XML config.
• A new binding object is created using Activator.CreateInstance(BindingCollectionElement.BindingType)
• Additional code taken from System.ServiceModel.Configuration loops thru BindingCollectionElement.ConfiguredBindings and initializes the binding object previously created. A check is made to make sure multiple bindings of the same type are not created.
• Finally, the binding object is returned to the caller.

GetChannelEndpointElement: This class uses only public classes and methods so I am including it. EnpointElement contains the configuration data for an endpoint, including the URI, identity, binding, and  binding configuration. The binding and binding configuration data is actually passed to my GetBinding method to return the correct binding with the correct binding configuration.

public static ChannelEndpointElement GetChannelEndpointElement(string configKey)
  {
   ClientSection section = (ClientSection)ConfigMgr.GetSection(configKey, system_ServiceModelAsOpenXml, system_ServiceModelAsCloseXml, clientSection);

   if (section == null)
    throw new ArgumentException(string.Format("Client section returned from config db for key {0} was null", configKey));

   if (section.Endpoints.Count != 1)
    throw new ArgumentException(string.Format("There must be exactly 1 endpoint returned from the configuration. {0} endpoints were returned for key {1}", section.Endpoints.Count, configKey));

   return section.Endpoints[0];
   
  }

GetEndpointAddress: Another method I created which almost uses all public methods. It takes a ChannelEnpointElement and returns a EndpointAddress which can be assigned to a wcf proxy class.

public static EndpointAddress GetEndpointAddress(ChannelEndpointElement element)
  {
   //Create a new builder, as the endpoint address is an immutable class
   EndpointAddressBuilder builder = new EndpointAddressBuilder();

   builder.Identity = LoadIdentity(element.Identity);
   builder.Uri = element.Address;

   return builder.ToEndpointAddress();
  }

LoadIdentity: This final method was taken from System.ServiceModel.Description.Configloader. This method takes a IdentityElement (a property of ChannelEnpointElement) and returns a EndpointIdentity which is passed to the EndpointAddressBuilder in my GetEndpointAddress method. The method basically checks the the existence of certain properties, and if one exists, it returns the appropriate identity using public factory methods.

• EndpointIdentity.CreateUpnIdentity
• EndpointIdentity.CreateSpnIdentity
• EndpointIdentity.CreateDnsIdentity
• EndpointIdentity.CreateRsaIdentity
• EndpointIdentity.CreateX509CertificateIdentity

There is a identity type of certificate reference which I was unable to add support for, because it referenced additional internal classes which I didn't want to duplicate at this time since I had no need for that identity type.

Issues

Since this is completely unsupported by Microsoft, and uses code that was never meant for external use, in a future version of the framework, something could break. I'm also not an legal expert when it comes with what someone can do with MSIL code discovered thru reflector, so for now, the actual code I wrote is not available publicly. Hopefully I provided enough information to allow someone to reproduce my steps and implement something on their own.

Another somewhat disappointing realization I came to, was that this might make it hard to support write operations to my configuration database. Currently the configuration database is managed thru scripts and direct editing. If write support to the config database became a requirement, I'd probably try to do the reverse and use the built in file save functionality in System.Configuration to save a temporary xml file, read it and then store it into my config database.

Conclusion

This was my first attempt at doing something like this, digging into framework code to see how it really works and trying to come up with a solution. I'm going to go forward using the code I wrote outlined in this post, as I have full read support for WCF XML configuration. The amount of code I had to write and test is significantly less then if I would have wanted to get the same amount of functionality (configuration support for all bindings and endpoint configurations).

I finally got nMap working under Vista. I don't know if it was the latest version of nMap I installed (4.23RC3) or winPCap (4.02), or some other change to Vista (update). You do need to run it as an administrator to get access to the network card at the low level required by the tool.

I don't know when they added this, but in the 4.23RC3 the GUI is included and works well. The command line is still there in the background available for the power users. One nice thing about the GUI is that as you make changes, you see the command line that is going to be execute.

In a recent email from MSDN Flash, I first learned of the new Parallel Extensions to the .Net Framework 3.5. This is an out of band release that is currently available as a CTP on Microsoft Downloads.

The description in the MSDN Flash newsletter has me pretty excited about what this offers.

[Parallel Extensions] is a managed programming model for data parallelism, task parallelism, and coordination on parallel hardware unified by a common work scheduler.

...

Parallel Extensions provides library-based support for introducing concurrency into applications written in with any .NET language, including but not limited to C# and Visual Basic.

It is specifically designed to take advantage of multi-core processors (among other things), which is important due to the recent (last 2 years) shift in the industry from raw clock speed to multi-core.

The computational power of multi-core processors, new programming models and platforms, and advanced research in usability all promise to change the way people interact with computers.

While excited, I do have some reservations. Encapsulating the lower level knowledge needed to take advantage of multiple cores will speed up development, but it will reduce the number of programmers who know how that lower level stuff needs to be written. The concepts used to take advantage of multi-cores, should be very similar to those required to take advantage of multi-threading, which is more "widely available" on the compact framework (although I am seeing more and more embedded computers with Core 2 Duo's). If this framework is only available for the full framework, the pool of skilled workers available for the compact framework could diminish (even more so).

I actually have more of a need to take advantage of multi-threading, and parallel processing in compact framework applications. As far as asp.net applications, I would like to see if this is something that would be recommended for them. Long running processes that would take advantage of this new framework, are usually best left outside the asp.net application. Speaking of long running processes, that is the 3rd area I work in, and in a traditional environment, would love to take advantage of this. However, the applications I am writing are all running on VMWare ESX, and IT has it set so that each VM only has a single CPU, and puts the burden of scheduling the 8 physical cores on ESX itself.

This will probably end up being something I try to work into personal projects. Unfortunately, I do not have the time to play with the CTP at this moment, but it is something I defiantly want to look into in the future.

Additional Information:

• MSDN Developer Center

From Dave Northey's blog, comes a link to an Active Directory tool previously only available to MS Premier Support. The tool discovers information about your Active Directory and Exchange infrastructure and exports it to Visio. The tool is available free of charge from Microsoft downloads.

Avoiding Development Disasters is the tile of an article I came across today and talks about how and why software projects fail. Here's what I got out of the article.

FBI Virtual Case File

The article opens up talking about the monumental failure that is/was the VCF and touches 6 factors which lead to the projects failure.

• Lack of an Enterprise Architecture - Unfortunately the article doesn't go into what they did have, have not.
• Poor management of developers, including a lack of management or micro-management
• Unqualified persons were placed in critical roles
• Constantly changing requirements
• Scope Creep
• Throwing more developers at the project in a last ditch effort to meet deadlines.

I am personally experiencing 3 of the listed items on a current development project, although it's far from a $100 million dollar system, I bet there are some striking similarities. I would imagine that every project at every company experiences 1 or more of the things in that list to some extent or another.

What is Success?

The VCF was eventually scrapped, but the author claims, that had it gone to production, it would have been deemed a success, even with all it's flaws. The author goes on to say that this is the general practice in the software industry (at least enterprise applications). As much as I like Microsoft, I think Vista is a shining example of this (although perhaps it is deemed less of a success inside of Microsoft).

I have to agree, and would be willing to say it's common in software projects in general, from small to large. How many times is a game released, and the day of, a patch is released? Even Epic Games, creators of the Unreal platform, and who coined the phrase "When it's done", still manage to release a product with known issues.

I'm not trying to criticize any one company or developer, other then myself, as I have created some less then magnificent code over my short life as a developer. I think part of the problem is that there are no points for creating good code. At the end of the day, "Well it works doesn't it?" pays the bills.

Why is it so hard to write code, almost 30-40 years since the first programs were written? We have better tools, faster computers, and years of other people's failures to learn from, and here we are, still producing less then our potential.

The Code to Ruin

The diagram "The Code to Ruin" presented in the article is so true, it's scary. You pretty much know what's going to happen before you start the project, but you still can't avoid it. That's depressing at best.

Maintainability

The article spends the last half, talking about the maintainability of code. Without code that is maintainable, while you launch may be a success, you next point release is probably going to be a failure. The article states that enterprise software should have a 15 year life span, that's longer then I have been coding.

I think ideas such as software as a service might help us reach that goal, and have more maintainable code overall. I'm not taking about providing software as a service, I mean, that the internal make up of your application is constructed from (loosely coupled) services. Breaking stuff down into more manageable pieces seems to be the way to go. We already do it with proper OOD design, we should also be applying it to the system in general. Of course there is a trade off, the most notable to me being one of performance, but that's what these faster computers are for ;)

All in all, it was a good article, and made me really think about software projects, both past and current that I am working on. I have to imagine that there are people out there that would be the perfect compliment to a talented programmer such as myself (well at least I think I'm talented). Or, does it mean, that I need to spend less time with technology and programming, and learn more skills like project management, documentation, business analysis, etc.? To specialize, or generalize, that is the question?

Sometimes a file gets locked in TFS, and for whatever reason, you need to unlock it. You can use the tf command line utility to accomplish this.

tf lock /lock:none $/Project/AnyFile.extension /workspace:ComputerName;User /s:http://TfsServer.Com:8080

Server

Today I tackled upgrading my development server from BizTalk 2006 (Developer Edition) to R2 (Developer Edition). You can read my previous post on installing R2 on a clean machine. Detailed instructions (for all supported install scenarios) can be downloaded from Microsoft Downloads (See below for another link).

Upgrading was extremely easy, and I encountered no errors or warnings. I used the same cab file I downloaded for my clean install, since it was the same platform (Windows Server 2003 32 bit). You need to stop all BizTalk related services, and IIS (this is detailed in the instructions) during the install, and then restart them after the upgrade is finished. There wasn't even a reboot required.

One thing I'd like to point out, is that upon first glance, it looks like you are still running BizTalk 2006. The banner in the admin console doesn't mention R2, there is no new group in start/programs, even though the upgrade process lists an unistall step of BizTalk 2006 and an install of R2, there is no R2 program listed in Add/Remove programs, and the 2006 is still listed. Re-Running the R2 installer gives you options to repair, modify or remove, so the installer seems to think everything is correctly installed.

Not quite convinced I decided to look into this further. There is a registry key, HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\BizTalk Server\, which lists versions. I had a sub key for 3.0 and 3.5_Migrated. On my clean install, there is no 3.5_Mirgrated key, the admin console, and Add/Remove programs all list 2006 (not R2), so I guess that's just the way it is. If you are in the admin console and go to Help\About Microsoft BizTalk Server Administration, it lists version 3.6.1404 (compared to version 3.5.1602 before the upgrade). I guess I had an expectation that it would say R2 somewhere.

Another gotcha, is that the new WCF adapters are not installed by default. After the upgrade process, you have to re-run the installer and choose modify. Then you can add the WCF adapters, both under the BizTalk runtime, and admin tool. That goes for any optional component. If it wasn't installed before the upgrade, you have to go in and add it after the fact.

Workstation

Figured now was as good as time as ever to upgrade my workstation (development components) as well. The link I posted above didn't have a guide for installing on Vista, but a quick search found another page on Microsoft Downloads that did have a Vista Guide. I wasn't planning on reading the guide, but I couldn't get the install process to even start under Vista, so figured some trickery might be involved.

The first thing I did was find the redistributable components for Vista, so I could start downloading those while reading the install guide. The link to the components in including in the Technical Appendix in the install guide, and at the time of this post, the direct link is: http://go.microsoft.com/fwlink/?LinkId=81432, and weigh in around 30MB.

Since I just want the development tools, I skipped to page 21 in the guide, which are where the install instructions start. There was no special steps listed for Vista. I ended up rebooting my computer, and still couldn't get it to start (clicking install BizTalk from the setup screen didn't seem to kick off the msi process). I tried running the msi directly, and got the expected error saying to run setup. So I went back to setup, and this time it worked, weird.

After I actually got the installer running, everything after that was as easy as the server upgrade. I could have appreciated support for VS 2008, but 2005 isn't all that bad. The only time I would notice a difference is if I am working on a class library instead of a BizTalk artifact.

Rant

I don't like anti-virus software, and I like real time virus scanning even more. It's like trying to fix a paper cut with brain surgery. There are plenty of guidelines out there for how you can configure your computer, and how you can be a smarter user and avoid virus, spyware etc. Those are mainly for workstations and home PCs, but what about servers?

My take on servers is, you shouldn't be doing anything that compromises the security of a server, and that includes browsing the web, using a standard or admin account. So if you cut out browsing, don't install warez, what's left? I know, I know, what about things like Code Red? When Code Red first hit, it didn't matter if you anti-virus installed or not, the attack was so new that your only defense was to pull your web server off line until you got it patched. A virus from last year, should never find it's way onto your server if you are careful, leaving zero-day exploits and black market hacks, that AV isn't going to catch it right away anyway. About the only thing AV software is good for, is finding that bot net Trojan your Mom gets when she installs Super Happy Smiley emoticons for email. 

I know I'm not going to win the argument for not having any anti-virus installed, but let's at least be intelligent about it ok? If you call yourself an IT administrator, Network Administrator, etc, now is the time to earn that title. Anyone can carpet bomb a network with an enterprise anti-virus package. The AV makers even make it easy to auto deploy their poorly written, resource hogging craplets via auto install software and group policy.

A true administrator will take the time to come up with a comprehensive approach to computer and network defense, including the intelligent installation and configuration of AV software. I found an excellent document on folders and files you should seriously consider exempting from real time, and even on demand scans.

Yeah, yeah, I know, just run Linux on the desktop and use a LAMP stack for your servers.

My task (well one of them) for the week was to update one of our Custom BizTalk adapters. Be sure to read my post on how to Install a custom BizTalk Adapter for some background information.

One of the changes I needed to make (which I had been putting off), was changing the namespace. Namespaces are case sensitive in the world of BizTalk. So if you change the namespace, you have to update the registry entry for the adapter with the new case sensitive namespace. Also, it seems like you need to uninstall the adapter from BizTalk in order to use the new assembly, since the TypeName changed.

In order to do this, I used the BizTalk admin tool to delete the adapter (BizTalk Group\Platform Settings\Adapters), update the registry, then re-add the adapter. If you are making calls to resource files where you are specifying the type name, be sure to update the string you are passing in as those calls will now fail. Even better, you should have a single string constant that defines the type name so you only have to make this change once.

Once the namespace change was finished, further changes to the assembly, only required me to copy over the existing assembly and restart the host process. Exceptions in adapters, at least our custom adapter, would cause the host process to keep restarting, so I ended up disabling the windows service for the host process after restarting/starting it, just to keep down on the repetitive errors.

Robust and proper error handling is a must in custom BizTalk artifacts. I find that writing errors to the event log works the best, and to include a domain specific error message, error code, and Exception.ToString(). The error codes are nice because you can go back in search for them in the code. They also are easily referenced in tech support and trouble shooting documentation.

In my specific case, a custom adapter which used the BizTalk PollTask mechanism, uncaught exceptions are written to the event log with the full stack trace. This means, that you don't need to catch general exceptions (and static code analysis says we shouldn't). Starting from exceptions thrown by PollTask, I added specific catch blocks to log specific information useful for trouble shooting, and re-threw the exception using "throw;" which re-throws the original exception. This causes the adapter to error out, and to be honest I do not know if this is the best way to handle things. It results in the host process restarting as mentioned above. When the out of the box adapters have errors, like the File adapter not having permissions to the specified folder, the adapter is eventually shut down.