Not that we use the TFS reports that much, but I noticed the main report on the front page of project site in Share Point, showed the warehouse hadn't been updated in a couple of days. So begins another adventure into getting it to work again.

At first I thought it was going to be easy, the service wasn't running. We had a power failure in the data center this past week, and it looks like on reboot, the service failed to start. Our new data center isn't running at production levels yet, so a power failure due to human error is not surprising. However, after restarting the service, the warehouse still wasn't updating.

I decided to invoke the warehouse manually, and ran the scripts suggested on this familiar MSDN thread to try to see what was going on. The manual warehouse update worked once, then wouldn't work again (Status always returned idle) until I restarted the IIS server. The results of the query did not seem to update, even though the Status values returned from the web service indicated that the warehouse update had ran. I decided to look at the ScheduledServices.xml in Program Files\Microsoft Visual Studio 2008 Team Foundation Server\TFSServerSchedule, and it was showing a wierd value for last run time (lastrun="1:1:1:0:0:0"). I'm guessing that this value indicates the process started, but never finished, so it never updated the date. My final clue to work with was an event log entry stating there was an unexpected exception while calculating code churn (TF53010). The specific details spoke of a incomplete character stream and possibly invalid encoding, in an xml file I had checked in recently.

I came across a MSDN post that mentioned that you can add this (<add key="ProcessingType" value="Full" />) appKey to your warehouse app.config. I could tell the warehouse web service restarted due to the delay in the web page for invoking the warehouse, but it appears like the run command has no effect again. So after "running" once, it won't run again at all. The value in ScheduleServices.xml is not changing at all (I manually changed the value so I could tell), so things are working at all. Event log is clear, so I need to find some other type of information.

I downloaded an installed DebugView from Sysinternals (used it before in TFS 2005) and changed tracing levels as outlined in the MSDN post linked to above, however either something change in 2008, or I setup the debugger wrong cause I got nothing. There were some other settings related to tracing, so I turned those on as well (appSettings.traceWriter = true, and set all switches to level 4). Nothing at all. The changes to the web.config file, which restart the app domain, seem to have some effect in that GetWarehouseStatus returns values to indicated it's trying to process, however, I get no debug or trace information. I take that back, Debug information has finally stared showing up I just don't know why.

Unfortunately there was nothing that pointed to why the warehouse processing was failing in Debug View. It seemed to show that everything was working correctly. I'm thinking that ScheduledServices.xml is only updated when the scheduler runs the warehouse, not manually. Looking back at my reports, them seem to be updating now.

So in summary, things appear to be working for now (need to wait 1 hour to see if the scheduler runs as expected), but I don't know exactly what fixed the problem. I turned off debugging and full processing, and things seemed to continue to work. I think the TfsWarehouse.dbo._WarehouseConfig table has contains rows that provide better information on when the warehouse last ran. You can easily change the RunIntervalSeconds setting to change how often the scheduler runs the task. I restarted the scheduler tasks after changing. Since our database is small, I set this to 2 minutes just to see if it was working, then set it back to 60 minutes.