This post is part of my PDC09 Conference Notes series. These are my raw notes taken while watching the various session videos from PDC09. Refer to my original post for some conventions I tried to use.

This presentation was part 2 of 2 on SQL Server StreamInsight, and was intended as an advanced look at StreamInsight. If I get around to watching part 1, I will update this post.

“Microsoft SQL Server StreamInsight is a powerful platform for developing and deploying complex event processing (CEP) applications. Its high-throughput stream processing architecture and familiar .NET-based development platform enable developers to quickly implement robust and highly efficient event processing applications.

Typical event stream sources include data from manufacturing applications, financial trading applications, Web analytics or operational analytics. StreamInsight enables you to develop CEP applications that derive immediate business value from this raw data by lowering the cost to extract, analyze, and correlate the data and by allowing you to monitor, manage, and mine the data for conditions, opportunities, and defects in close to real time.” –Microsoft TechNet

StreamInsight was designed to handle input sources that operate in the milisecond range, instead of the second, minute and hour+ range. One of the demo’s shown, demonstrated capturing and processing 60 events per second. StreamInsight will be licensed and released with SQL Server.

There was mention of Windows CE and other platform support when they were talking about how the Stream OS provides hardware abstraction for StreamInsight. It seems kind of weird that they didn’t start with Windows CE as the most obvious use (to me) is automation. However, some of the examples shown where at the enterprise server level, which matches up with the definition I pulled from TechNet.

StreamInsight makes use of Native memory to avoid garbage collection, and I would assume there are other native code optimizations used for performance reasons. It looks like they are trying to bridge the gap between native code performance and the ease of .net development to support Complex event processing.

The following challenges facing complex event processing scenarios necessitated a platform:

• Pattern Detection
• Correlate Data
• Aggregate
• Hardware Abstraction

Adapters

• “Get” the data into the application
• Built using the adapter framework
• Default adapters wrap the framework in an Observable Patter and expose IEnumerble collections
• Adapters can be Push or Pull
• Adapters can be In-Order or Out-Of Order
• Work with Native Memory

Query Expressions

• Allows for data manipulation
• Projection
• Filtering
• Correlation (joins)
• Aggregate over Windows of time (Temporal semantics)
  • Time windows can be overlapping or non-overlapping
• Grouping and Aggregation
• Implemented using Linq. Everything except the time aggregation is implemented using out of the box Linq. Linq extensions were created for the time aggregation.