If you write properties on the client you can’t use them in remote LINQ operations.

The problem occurs because these properties can’t be translated and sent to the server as they have been compiled into intermediate language (IL) and not LINQ expression trees that are required for translation by IQueryable implementations. There is nothing available in .NET to let us reverse-engineer the IL back into the methods and syntax that would allow us to translate the intended operation into a remote query.

This means you end up having to write your query in two parts; firstly the part the server can do, a ToList or AsEnumerable call to force that to happen and bring the intermediate results down to the client, and then the operations that can only be evaluated locally. This can hurt performance if you want to reduce or transform the result set significantly.

What we came up (David, Colin Meek and myself) is a provider-independent way of declaring properties just once so they can be used in both scenarios. Computed properties for LINQ to SQL, LINQ to Entities and anything else LINQ enabled with little effort and it works great on .NET 3.5 SP1 :)

Before example

Here we have extended the Employee class to add Age and FullName. We only wanted to people with “da” in their name but we are forced to pull down everything to the client in order to the do the selection.

partial class Employee {
	public string FullName {
		get { return Forename + " " + Surname; }
	}

	public int Age {
		get { return DateTime.Now.Year - BirthDate.Year -
			(DateTime.Now.Month < BirthDate.Now.Month
			|| DateTime.Now.Month == BirthDate.Now.Month && DateTime.Now.Day < BirthDate.Now.Day) ? 1 : 0);
		}
	}
}...var employees = db.Employees.ToList().Where(e => e.FullName.Contains("da")).GroupBy(e => e.Age);

After example

Here using our approach it all happens server side… and works on both LINQ to Entities and LINQ to SQL.

partial class Employee {
    private static readonly CompiledExpression<Employee,string> fullNameExpression =       DefaultTranslationOf<Employee>.Property(e => e.FullName).Is(e => e.Forename + " " + e.Surname);
    private static readonly CompiledExpression<Employee,int> ageExpression =       DefaultTranslationOf<Employee>.Property(e => e.Age).Is(e => DateTime.Now.Year - e.BirthDate.Value.Year - ((DateTime.Now.Month < e.BirthDate.Value.Month || (DateTime.Now.Month == e.BirthDate.Value.Month && DateTime.Now.Day < e.BirthDate.Value.Day)) ? 1 : 0)));

    public string FullName {
        get { return fullNameExpression.Evaluate(this); }
    }

    public int Age {
        get { return ageExpression.Evaluate(this); }
    }
}...

var employees = db.Employees.Where(e => e.FullName.Contains("da")).GroupBy(e => e.Age).WithTranslations();

Getting started

Check out this download which includes the necessary project to drop-in to your solution. The caveat to the usage technique shown above is you need to ensure your class has been initialized before you write queries to it. If this is a problem check out the usage considerations section below.

The other major caveat is obviously the expression you register for a property must be able to be translated to the remote store so you will need to constrain yourself to the methods and operators your IQueryable provider supports.

Usage considerations

There are a few alternative ways to use this rather than the specific examples above.

Registering the expressions

You can register the properties in the class itself as shown in the examples which means the properties themselves can evaluate the expressions without any reflection calls. Alternatively if performance is less critical you can register them elsewhere and have the methods look up their values dynamically via reflection. e.g.

…
DefaultTranslationOf<Employee>.Property(e => e.FullName).Is(e => e.Forename + " " + e.Surname);
var employees = db.Employees.Where(e => e.FullName.Contains("da")).GroupBy(e => e.Age).WithTranslations();
…
partial class Employee {
    public string FullName { get { return DefaultTranslationOf<Employees>.Evaluate<string>(this, MethodInfo.GetCurrentMethod());} }
}

If performance of the client-side properties is critical then you can always have them as regular get properties with the full code in there at the expense of having the calculation duplicated, once in IL in the property and once as an expression for the translation.

Different maps for different scenarios

Sometimes certain parts of your application may want to run with different translations for different scenarios, performance etc. No problem!

The WithTranslations method normally operates against the default translation map (accessed with DefaultTranslationOf) but there is also another overload that takes a TranslationMap you can build for specific scenarios, e.g.

var myTranslationMap = new TranslationMap();
myTranslationMap.Add<Employees, string>(e => e.Name, e => e.FirstName + " " + e.LastName);
var results = (from e in db.Employees where e.Name.Contains("martin") select e).WithTranslations(myTranslationMap).ToList();

Not specifying WithTranslation everywhere

If you are happy to always have the default translation applied simply add the following statement to the top of your file which will bring in a bunch of extensions methods for the usual LINQ query operators that already apply WithTranslation for you :)

using Microsoft.Linq.Translations.Auto;

With .NET 4.0

You should be able to drop the ExpressionVisitor class in the download and reference the built-in one.

How it works

CompiledExpression<T, TResult>

The first thing we needed to do was get the user-written client-side “computed” properties out of IL and back into expression trees so we could translate them. Given that we also want to evaluate them on the client we need to compile them at run time so CompiledExpression exists which just takes an expression of Func<T, TResult>, compiles it and allows evaluation of objects against the compiled version.

ExpressiveExtensions

This little class provides both the WithTranslations extensions methods and the internal TranslatingVisitor that unravels the property accesses into their actual registered Func<T, TResult> expressions via the TranslationMap so that the underlying LINQ provider can deal with that instead.

TranslationMap

We need to have a map of properties to compiled expressions and for that purpose TranslationMap exists. You can create a TranslationMap by hand and pass it in to WithTranslations if you want to programmatically create them at runtime or have different ones for different scenarios but generally you will want to use…

DefaultTranslationOf

This helper class lets you register properties against the default TranslationMap we use when nothing is passed to WithTranslations. It also allows you to lookup what is already registered so you can evaluate to that although there is a small reflection performance penalty for that:

public int Age { get { return DefaultTranslationOf<Employees>.Evaluate<int>(this, MethodInfo.GetCurrentMethod()); } }

AutoTranslation

Remembering to specify WithTranslations everywhere can be a pain so if you always want to go via translation and use the default TranslationMap you can include the Microsoft.Linq.Translations.Auto namespace in your code which includes extension methods for all the regular LINQ operations that saves you having to remember.

Have fun!

[)amien

Templates

• Entity Developer
  Add-in for Visual Studio that provides a replacement designer with code templates. Commercial ($99.95)
• LLBLGen Pro
  This ORM templating and design tool has a set of LINQ to SQL templates available for generating C# code including file per entity that can be used in conjunction with their designer. Templates are free but require LLBLGen Pro licence (€179-249).
• L2ST4
  My templates use Visual Studio's built in T4 engine and provide a great starting point to customizing the code generation process. Everything that SQL Metal/LINQ to SQL designer can generate is handled including C# and VB.NET generation and are freely licensed under the MS-PL.
• PLINQO
  These templates for Eric Smith's ever-popular CodeSmith templating environment include a whole host of extra functionality beyond the standard generation including entity clone & detach, enum's from tables, data annotations, batching, auditing and more. Templates are free but require CodeSmith licence ($79-$299).
• T4 Toolbox
  Oleg Sych has put together a suite of useful T4 templates including ones for producing LINQ to SQL entities, data contexts as well as SQL scripts for altering the schema to reflect changes. C# only, MS-RL.

Note: While the T4 templating language is built-in to Visual Studio 2008/2010 it does not come with syntax highlighting or IntelliSense. Check out either:

• Clarius Visual T4
  Basic ‘community’ version available for free, commercial ‘pro’ version with IntelliSense, sub-templates, preview, user preferences etc. is $99.
• Tangible T4
  Free version available with limited IntelliSense, commercial ‘pro’ version with UML modelling etc. available for $99.

Blogs

• David DeWinter
  David is a dev in test who recently joined our team and hit the ground running with testing, blogging and helping out on the forums.
• Roger Jennings
  Roger over at OakLeaf Systems publishes regular articles and roundups of some of the best .NET data access content from around the web including LINQ to SQL.
• Scott Guthrie
  Our Corporate Vice President for the .NET Developer Platform takes a very active role in getting his hands dirty and publishes a series of useful LINQ to SQL articles.
• Sidar Ok
  Sidar is a regular forum helper and has written a number of great posts on LINQ to SQL including some good POCO coverage.

Tools

• Devart dotConnect
  Devart's dotConnect family are database providers for Oracle, MySQL, Postgres and SQLite that also include LINQ support to enable LINQ to SQL like functionality on other platforms. Some database basic versions are free, professional versions are commercial and vary in price (~$99-$209).
• Hugati DBML/EDMX Tools
  Add-in for Visual Studio that provides comparison/update facilities between the database and the DBML as well as standardising names and generating interfaces. Commercial ($49-$119, free 30 day trial).
• Hugati LINQ to SQL Profiler
  Profiling tool to help optimize your LINQ to SQL based applications. Commercial ($49-$119, free 45 day trial).
• LINQpad
  This invaluable tool helps you write and visualise your LINQ queries in a test-bench without compilation and includes a version for the .NET 4.0 beta. Free to use, auto-completion add-on available ($19).

Official guides

• Samples
  The official samples includes a whopping 101 snippets showing how to use many of the features and syntax.
• Programming Guide
  Includes steps on how to get started, querying, making changes, debugging and background information.
• Whitepaper
  Single document describing the architecture, query capabilities, change tracking and lifecycle, multi-tier entities, external mapping etc.

Books

• Essential LINQ
• LINQ in Action
• Pro ADO.NET 3.5 with LINQ
• Pro LINQ Object Relational Mapping

Support

• Official MSDN forums
  Great way to get access to the product team directly as well as knowledgeable and experienced users if you have a question or a problem.
• Official LINQ to SQL FAQ 
  Coverage is a little thin on the ground but it has some useful tips.
• StackOverflow's LINQ to SQL tag
  StackOverflow has rapidly become a leader in questions and answers for a wide variety of developer topics and covers LINQ to SQL (which the site uses for it's data access too!)

[)amien

Change list

Performance

• Query plans are reused more often by specifically defining text parameter lengths
• Identity cache lookups for primary key with single result now includes query.Where(predicate).Single/SingleOrDefault/First/FirstOrDefault
• Reduced query execution overhead when DataLoadOptions specified (cache lookup considers DataLoadOptions value equivalency – post beta 1)

Usability

• ITable<T> interface for additional mocking possibilities
• Contains with enums automatically casts to int or string depending on column type
• Associations can now specify non-primary-key columns on the other end of the association for updates
• Support list initialization syntax for queries
• LinqDataSource now supports inherited entities
• LinqDataSource support for ASP.NET query extenders added

Query stability

• Contains now detects self-referencing IQueryable and doesn't cause a stack overflow
• Skip(0) no longer prevents eager loading
• GetCommand operates within SQL Compact transactions
• Exposing Link<T> on a property/field is detected and reported correctly
• Compiled queries now correctly detect a change in mapping source and throw
• String.StartsWith, EndsWith and Contains now correctly handles ~ in the search string
• Now detects multiple active result sets (MARS) better
• Associations are properly created between entities when using eager loading with Table-Valued Functions (TVFs)
• Queries that contain sub-queries with scalar projections now work better

Update stability

• SubmitChanges no longer silently consumes transaction rollback exceptions
• SubmitChanges deals with timestamps in a change conflict scenario properly
• IsDbGenerated now honors renamed properties that don't match underlying column name
• Server-generated columns and SQL replication/triggers now work instead of throwing SQL exception

General stability

• Binary types equate correctly after deserialization
• EntitySet.ListChanged fired when adding items to an unloaded entity set
• Dispose our connections upon context disposal (ones passed in are untouched)

SQL Metal

• Foreign key property setter now checks all affected associations not just the first
• Improved error handling when primary key type not supported
• Now skips stored procedures containing table-valued parameters instead of aborting process
• Can now be used against connections that use AttachDbFilename syntax
• No longer crashes when unexpected data types are encountered

LINQ to SQL class designer

• Now handles a single anonymously named column in SQL result set
• Improved error message for associations to nullable unique columns
• No longer fails when using clauses are added to the partial user class
• VarChar(1) now correctly maps to string and not char
• Decimal precision and scale are now emitted correctly in the DbType attributes for stored procedures
• Foreign key changes will be picked up when bringing tables back into the designer without a restart

Code generation (SQL Metal + LINQ to SQL class designer)

• Stored procedures using original values now compiles when the entity and context namespaces differ
• Virtual internal now generates correct syntax
• Mapping attributes are now fully qualified to prevent conflicts with user types
• KnownTypeAttributes are now emitted for DataContractSerializer with inheritance
• Delay-loaded foreign keys now have the correct, compilable, code generated
• Using stored procedures with concurrency no longer gets confused if entities in different namespace to context
• ForeignKeyReferenceAlreadyHasValueException is now thrown if any association is loaded not just the first

Potentially breaking changes

We worked very hard to avoid breaking changes but of course any potential bug fix is a breaking change if your application was depending on the wrong behavior. The ones I specifically want to call out are:

Skip(0) is no longer a no-op

The special-casing of 0 for Skip to be a no-op was causing some subtle issues such as eager loading to fail and we took the decision to stop special casing this. This means if you had syntax that was invalid for a Skip greater than 0 it will now also be invalid for skip with a 0. This makes more sense and means your app would break on the first page now instead of subtlety breaking on the second page. Fail fast :)

ForeignKeyReferenceAlreadyHasValue exception

If you are getting this exception where you weren’t previously it means you have an underlying foreign key with multiple associations based on it and you are trying to change the underlying foreign key even though we have associations loaded.Best thing to do here is to set the associations themselves and if you can’t do that make sure they aren’t loaded when you want to set the foreign key to avoid inconsistencies.

[)amien

Bitmap v scalable fonts

Bitmap fonts are incredibly easy to make. Using a program like Softy or BitFonter you decide the size of your letters and start plotting pixels. You can see exactly how it will look because you draw every glyph (letter/symbol/number) in every size you want to support. This can obviously be very time consuming and doesn’t let you take full advantage of the resolution of the device and the capabilities it offers. A printer can handle in excess of 300 dpi while a display is typically 72 dpi (Mac) or 96 dpi (Windows) with LCD’s supporting sub-pixels due to the individual layout of the red-green and blue elements you can’t feasibly pre-plot every single combination and even if you could the file size would be rather large.

Rather than having specific set of pixels to turn on or off TrueType, OpenType and PostScript fonts contain a series of instructions that tell the computer the shape using a series of points, lines and curves. This means the computer can scale the glyph to the size that is required and then take full advantage of the device being rendered honoring the users preferences for anti-aliasing (smoothing using shades of grey), sub-pixel precision (smoothing using hints of red, green and blue to take advantage of the layout of colour elements in an LCD display), desired contrast and gamma settings etc.

Grid fitting

Such a scaled glyph won’t fit perfectly within a pixel grid and a small sizes and low resolution it can look awful. It is also necessary to ensure that the vertical part of the letter I (known as a stem) looks very similar to the stem of other letters at the same size – we don’t want some letters looking bold – and that the top of the letter o aligns nicely with the top of the i etc. (in most fonts). The glyphs themselves don’t know what is a stem, what should align with other glyphs etc.

Many renders include logic to try and improve un-hinted fonts such as the drop-out control in Windows through to the full auto-hinter in FreeType. If you’ve ever used free fonts from any of the numerous web sites around you’ve probably seen that it doesn’t get it right and it looks like this: 

The first few versions of Envy Code R looked like that because to address this problem you need to learn a process called hinting, which let’s the designer give the renderer “hints” on how to choose the pixels.

Hinting

Font hinting started off as stem and edge identification so that glyphs would maintain the right proportions when sized and rendered on these low-DPI devices. It became apparent that a much more fine-grained level of control was required and so a stack-based byte-code language was developed as part of the TrueType specification to allow designers finer control in how points are adjusted to better take advantage of the display characteristics.

A TrueType font can contain extra blocks which describes, using a sequence of bytes that represent instructions and their arguments, the process by which to align the points and therefore make decisions about how best to fit the letter into the grid by retaining and adjusting various elements.

The important blocks are:

Each instruction (opcode) has a mnemonic that is representative of what it does and these are documented in Chapters 5 through 6 of the TrueType specification (along with much other useful relevant information). Actual per-glyph instructions are stored with each glyph outline in the glyp block.

Rasterizing & rendering

There are many different ways a TrueType font can end up on your screen with a lot of variants between how vendors chose to render the font and what options they expose to developers and users to fine-tune the experience.

• Windows – User choice of 1-bit, 4-bit grey-scale anti-aliasing, ClearType, ClearType tuning and display DPI plus WPF and DirectWrite per-app options
• Mac OS X – User choice of sub-pixel anti-aliasing strength and 1-bit cut-off plus per-app 1-bit option (e.g. Terminal)
• Java – Per-application choice of 1-bit, grey-scale or sub-pixel rendering
• Flash – Per-application choice of 1-bit or grey-scale
• FreeType – Rendering library that exposes a number of runtime and compile-time settings

This is of course ignoring the other rendering engines out there such as the Adobe’s Photoshop, RiscOS, D-Type rendering engine, Font Fusion (used on BeOS) etc. and prior versions of those renderers listed above (Flash and Mac OS changed significantly). Getting it pixel-perfect on every combination is impossible but we can try :)

Tools

Instructing fonts is a painstaking process at the best of times and few people deal directly with the low-level instructions instead relying on tools, stem identification and higher-level languages to achieve the same result. Some tools that have support for hinting instructions are:

• FontLab Studio 5
  Comprehensive font-production package for Windows and Mac that includes auto-hinting and it’s own higher-level link language that it can generate TrueType instructions from but it does not support viewing or modifying existing TrueType instructions and does not handle diagonals well. Rendering preview includes mono, grey-scale and ClearType. (Commercial $649)
• Fontographer 4.1
  Rather dated font-production package for Windows and Mac. (Commercial $349)
• FontForge
  Comprehensive font-production package that runs on X11 that includes auto-hinting and the ability to disassemble and edit existing TrueType instructions as well as debug them with stepping. Includes basic mono/grey-scale rendering options. (Open source)
• Microsoft Visual TrueType
  Hinting instruction tool from Microsoft that uses it’s own higher-level VTT Talk language that compiles down to TrueType instructions that you can further edit. Includes a comprehensive set of preview rendering options but is not capable of disassembling existing instructions. (Commercial, free with signed licence agreement)
• TTX
  Python scripts that can convert a font into an editable XML representation and back including disassembly and assembly of TrueType hinting instructions. (BSD)
• TTIComp
  Command-line tool that provides an alternative C-like hinting language. (GPL)
• Xgridfit
  FontForge scripts to provide an alternative XML-based hinting language. (GPL)
• TrueTypeViewer
  Windows tool for displaying TrueType fonts and glyphs including debugging and descriptive disassembly of instructions. (GPL)
• TTHMachine
  Real-time editing of hinting instruction mnemonics and observing their effects which is useful for learning. (Free, no longer supported)

[)amien

Take full control of  the TSQL

There are times when LINQ to SQL refuses to cook up the TSQL you wanted either because it doesn’t support the feature or because it has a different idea what makes an optimal query.

In either case the Translate method allows you to deliver your own TSQL to LINQ to SQL to process as if it were its own with execution, materialization and identity mapping still honored. For example:

var db = new PeopleContext();
if (db.Connection.State == System.Data.ConnectionState.Closed)
    db.Connection.Open();
var cmd = db.GetCommand(db.Persons.Where(p => p.CountryID == 1));
cmd.CommandText = cmd.CommandText.Replace("[People] AS [t0]", "[People] AS [t0] WITH (NOLOCK)");
var results = db.Translate<Person>(cmd.ExecuteReader());

Complex stored procedures

When working with stored procedures the LINQ to SQL designer and SQLMetal tools need a way of figuring out what the return type will be. In order to do this without actually running the stored procedure itself they use the SET FMTONLY command set to ON so that SQL Server will just parse the stored procedure instead.

Unfortunately this parsing does not extend to tdynamic SQL or temporary tables so you must change the return type from the scalar integer to one of the known entity types by hand. You could use the following command at the start to let it run regardless given the subsequent warning.

SET FMTONLY OFF

If your stored procedure can not safely handle being called at any time with null parameters set the return type by hand instead.

Cloning an entity

There are many reasons you might want to clone an entity – you may want to create many similar ones, you could want to keep it around longer than the DataContext it came from – whatever your reason implementing a Clone method can be a pain but taking advantage of the DataContractSerializer can make light work of this providing your DBML is set to enable serialization.

If you use discriminator subclassing you will need to either ensure your type is cast to its concrete type or use my L2ST4 templates for now as .NET 3.5 SP1 doesn’t emit the necessary KnownType attributes to make this automatically happen (fixed in .NET 4.0). Add a simple method to serialize in-memory like this:

public static T Clone<T>(T source) {
    var dcs = new System.Runtime.Serialization.DataContractSerializer(typeof(T));
    using (var ms = new System.IO.MemoryStream()) {
        dcs.WriteObject(ms, source);
        ms.Seek(0, System.IO.SeekOrigin.Begin);
        return (T)dcs.ReadObject(ms);
    }
}

And then to clone simply:

var source = myQuery.First();var cloned = Clone(source);

Be aware that this comes with a little overhead in the serialization and deserialization process.

If that is a problem for you then why not grab those templates and make your entities implement ICloneable!

[)amien

Loading a delay-loaded property

LINQ to SQL lets you specify that a property is delay-loaded meaning that it is not normally retrieved as part of normal query operations against that entity. This is particularly useful for binary and large text fields such as a photo property on an employee object that is rarely used and would cause a large amount of memory to be consumed on the client not to mention traffic between the SQL and application.

There are times however when you want all these binaries returned in a single query, say for example returning all the photos for the company photo intranet page:

var db = new NorthwindContext();
var loadOptions = new DataLoadOptions();
loadOptions.LoadWith<Employee>(e => e.Photo);
db.LoadOptions = loadOptions;

Multiple entity types from a single stored procedure

It is actually possible to return multiple entity types from a single stored procedure in LINQ to SQL although this is not well known as the LINQ to SQL designer doesn’t actually support it. Indeed to generate the code it is necessary to hand-edit the DBML and then use SQL Metal (or my T4 template) to generate the required method signature.

In fact it is just much easier to write the code yourself and add it to the non-generated portion of your data context. If you imagine a stored procedure GetStaticData that looks like this:

CREATE PROCEDURE GetStaticData AS
SELECT * FROM Region
SELECT * FROM Categories
SELECT * FROM Territories

Then all you need to do is write a method signature that looks like this (the sequence of result type attributes must match the order in the stored procedure):

[Function(Name=@"dbo.DynamicContractsActiveBetween")]
[ResultType(typeof(Region))]
[ResultType(typeof(Category))]
[ResultType(typeof(Territory))]
public IMultipleResults GetStaticData() {
	return (IMultipleResults) ExecuteMethodCall(this, (MethodInfo) MethodInfo.GetCurrentMethod()).ReturnValue;
}

Intercepting create, update and delete operations

There are times it is useful to be able to listen in to when these events happen and perform your own logic, perhaps auditing or logging for some scenarios. The easiest way to do this is to implement some specially-named methods on your data context, perform your action and then to dispatch the call back to LINQ to SQL.

The format of these specially-named methods is [Action][Entity] and then you should pass back control to LINQ to SQL using ExecuteDynamic[Action] where [Action] is either Insert, Update or Delete. One example of such usage might be:

partial class NorthwindContext {
   partial void InsertEmployee(Employee instance) {
      instance.CreatedBy = CurrentUser;
      instance.CreatedAt = DateTime.Now;
      ExecuteDynamicInsert(instance);
   }

   partial void UpdateEmployee(Employee instance) {
      AuditEmployeeOwnerChange(instance);
      instance.LastModifiedAt = DateTime.Now;
      ExecuteDynamicUpdate(instance);
   }

   partial void DeleteEmployee(Employee instance) {
      AuditDelete(instance, CurrentUser);
      ExecuteDynamicDelete(instance);
   }
}

[)amien

I can now exclusively reveal more! You may recognize some from projects you’ve already worked on but didn’t know had a name much less a recognized methodology.

ADD, Agro Driven Development

Developers code features directly proportional to the amount of heat they are getting from users, sales or managers. Results in a jack-of-all-trades, master of none.

CDD, Clone Driven Design

Features and design are achieved by cloning somebody else's product thereby removing the pesky overhead of having to come up with ideas of your own. Sure fire way to stay firmly behind the leader.

EDD, Ego Driven Design

Where the direction of the project is dictated purely on an individuals ego and their ability to shout long and hard until they get their own way. The individual involved is rarely the target audience for the product which is often the ego-bruising trigger in the first place.

GDD, Golf Driven Design

Where features and specifications are agreed on the golf course by people who neither use the software nor are responsible for implementing it but want to write off the whole trip as a business expense. With any luck they won’t check the final product.

LDD, Lunch Driven Development

Where features, goals and APIs are decided over lunch by the developers and users who care enough to meet up over their lunch-time. Lunch must not be provided or you stray into Golf Driven Design instead.

PDD, Psychic Driven Development

Where specifications are not so much decided through real communication but rather obtained via a psychic link with potential users with mixed results. Often seen in conjunction with Ego Driven Design.

QDD, Query Driven Development

Every page or screen starts with a question to the user what they want to see, writing that as a SQL/LINQ query statement and then dumping results out via a simple UI. Would likely be better off in Access or Excel but people involved want to claim they have intraweb experience.

XDD, Xenophobic Driven Development

Where the majority of development time is given over to making sure others can't do anything BUT what the original developers wanted. Typically observed by large sets of exception messages, the absence of hooks and a sprinkling of the sealed keyword.

YDD, Yesterday Driven Development

Features added today are the ones the customers thought they were getting yesterday. Generally used in very-tight rapid methodologies.

ZDD, Zzzzz Driven Development

Every waking hour is given over to the development process at the expense of quality and design ideas that are only appreciated by the sleep-deprived. Normally observed on tight schedules such as those in the gaming industry.

[)amien

One of the things I wanted my LINQ to SQL T4 templates to do was be able to split the output into a file-per-entity. Existing solutions used either a separate set of templates with duplicate code or intrusive handling code throughout the template. Here’s my helper class to abstract the problem away from what is already complicated enough template code.

Using the Manager class

Setup

You’ll need to get the code into your template – either copy the code in or reference it with an include directive. Then declare an instance of the Manager class passing in some environmental options such as the desired default output path.

<#@ template language="C#v3.5" hostspecific="True"
#><#@ include file="Manager.ttinclude"
#><# var manager = new Manager(Host, GenerationEnvironment, true) { OutputPath = Path.GetDirectoryName(Host.TemplateFile) }; #>

Define a block

Then add one line before and one line after each block which could be split out into it’s own file passing in what the filename would be if split.

<# manager.StartBlock("Employee.generated.cs"); #>
public class Employee { … }
<# manager.EndBlock(); #>

Headers and footers

Many templates need to share a common header/footer for such things as comments or using/import statements or turning on/off warnings. Simply use StartHeader/EndHeader and StartFooter/EndFooter. The resulting blocks will be emitted into all split files and left in the original output to.

<# manager.StartHeader(); #>
// Code generated template
using System;

<# manager.EndHeader(); #>

Process

At the end of the template call Process to handle splitting the files (true) or not (false). Anything not included in a specific start/end block will remain in the original output file.

<# manager.Process(true); #>

When processing each block name in the Output path will either be overwritten or deleted to enable proper clean-up. It will also add and remove the files from Visual Studio so make sure your generated names aren’t going to collide with hand-written ones!

Manager classes

Here is the Manger class itself as well as the small ManagementStrategy classes that determines what to do with the files within Visual Studio (add/remove project items) and outside of Visual Studio (create/delete files).

Download Manager.ttinclude (4KB)

<#@ assembly name="System.Core"
#><#@ assembly name="EnvDTE"
#><#@ import namespace="System.Collections.Generic"
#><#@ import namespace="System.IO"
#><#@ import namespace="System.Text"
#><#@ import namespace="Microsoft.VisualStudio.TextTemplating"
#><#+

// T4 Template Block manager for handling multiple file outputs more easily.
// Copyright (c) Microsoft Corporation.  All rights reserved.
// This source code is made available under the terms of the Microsoft Public License (MS-PL)

// Manager class records the various blocks so it can split them up
class Manager
{
	private struct Block {
		public String Name;
		public int Start, Length;
	}

	private List<Block> blocks = new List<Block>();
	private Block currentBlock;
	private Block footerBlock = new Block();
	private Block headerBlock = new Block();
	private ITextTemplatingEngineHost host;
	private ManagementStrategy strategy;
	private StringBuilder template;
	public String OutputPath { get; set; }

	public Manager(ITextTemplatingEngineHost host, StringBuilder template, bool commonHeader) {
		this.host = host;
		this.template = template;
		OutputPath = String.Empty;
		strategy = ManagementStrategy.Create(host);
	}

	public void StartBlock(String name) {
		currentBlock = new Block { Name = name, Start = template.Length };
	}

	public void StartFooter() {
		footerBlock.Start = template.Length;
	}

	public void EndFooter() {
		footerBlock.Length = template.Length - footerBlock.Start;
	}

	public void StartHeader() {
		headerBlock.Start = template.Length;
	}

	public void EndHeader() {
		headerBlock.Length = template.Length - headerBlock.Start;
	}	

	public void EndBlock() {
		currentBlock.Length = template.Length - currentBlock.Start;
		blocks.Add(currentBlock);
	}

	public void Process(bool split) {
		String header = template.ToString(headerBlock.Start, headerBlock.Length);
		String footer = template.ToString(footerBlock.Start, footerBlock.Length);
		blocks.Reverse();
		foreach(Block block in blocks) {
			String fileName = Path.Combine(OutputPath, block.Name);
			if (split) {
				String content = header + template.ToString(block.Start, block.Length) + footer;
				strategy.CreateFile(fileName, content);
				template.Remove(block.Start, block.Length);
			} else {
				strategy.DeleteFile(fileName);
			}
		}
	}
}

class ManagementStrategy
{
	internal static ManagementStrategy Create(ITextTemplatingEngineHost host) {
		return (host is IServiceProvider) ? new VSManagementStrategy(host) : new ManagementStrategy(host);
	}

	internal ManagementStrategy(ITextTemplatingEngineHost host) { }

	internal virtual void CreateFile(String fileName, String content) {
		File.WriteAllText(fileName, content);
	}

	internal virtual void DeleteFile(String fileName) {
		if (File.Exists(fileName))
			File.Delete(fileName);
	}
}

class VSManagementStrategy : ManagementStrategy
{
	private EnvDTE.ProjectItem templateProjectItem;

	internal VSManagementStrategy(ITextTemplatingEngineHost host) : base(host) {
		IServiceProvider hostServiceProvider = (IServiceProvider)host;
		if (hostServiceProvider == null)
			throw new ArgumentNullException("Could not obtain hostServiceProvider");

		EnvDTE.DTE dte = (EnvDTE.DTE)hostServiceProvider.GetService(typeof(EnvDTE.DTE));
		if (dte == null)
			throw new ArgumentNullException("Could not obtain DTE from host");

		templateProjectItem = dte.Solution.FindProjectItem(host.TemplateFile);
	}

	internal override void CreateFile(String fileName, String content) {
		base.CreateFile(fileName, content);
		((EventHandler)delegate { templateProjectItem.ProjectItems.AddFromFile(fileName); }).BeginInvoke(null, null, null, null);
	}

	internal override void DeleteFile(String fileName) {
		((EventHandler)delegate { FindAndDeleteFile(fileName); }).BeginInvoke(null, null, null, null);
	}

	private void FindAndDeleteFile(String fileName) {
		foreach(EnvDTE.ProjectItem projectItem in templateProjectItem.ProjectItems) {
			if (projectItem.get_FileNames(0) == fileName) {
				projectItem.Delete();
				return;
			}
		}
	}
}#>

[)amien

Updates

• Now licensed under the Microsoft Public License and hosted at CodePlex
• User options specified with a var options block at the start of the template
• Option for each class to be a separate file that is reflected in the VS project (EntityPerFile=true)
• Detection and support of IsComposable functions
• General code clean-up and better error handling such as missing DBML file

In action!

To show just how easy the templates are to use (they require nothing more than Visual Studio 2008) here is a short (2m30s) screen-cast of them in action!

Download LINQ to SQL T4 screen-cast (WMV) (5MB).

If you decide you want to go back to the built-in code generation just set the .template.cs/.vb Build Action property to None and the .generated.cs/.vb file back to Compile. You can then remove the .tt and .ttinclude files.

Please remember that these templates are a work-in-progress and you should ensure the generated code is suitably tested with your application.

CodePlex

CodePlex makes it easier for people to be able to see and merge updates in with their own modified versions as well as report issues via the issue tracker etc. There is also an RSS feed that lets you keep track of releases, source updates or whatever else you are interested in.

For now it is a grab-the-source style release but I hope to publish downloadable tested releases wrapped up in a Visual Studio Installer (VSI) package to make getting started easier soon.  Feel free to grab the sources directly via TFS/Subversion to be able to diff them with your own modified versions.

Enjoy!

[)amien

C# and VB.NET don’t allow a class to change its type at runtime and LINQ to SQL specifically doesn’t provide a mechanism for changing the underlying discriminator for this reason.

Discarding the current object and creating a new one is fraught with issues. What do we do about existing references, unsaved data, established associations and caches?

Start with an example

Consider an abstract Account class with SavingsAccount and CurrentAccount sub-classes. Bank accounts don’t change type once created (in my experience) so that’s good so far.

When we get into processing and validation logic its tempting to create ClosedAccount and OpenAccount classes but what happens during execution when closing an account?

A further consideration is how exactly ClosedAccount and OpenAccount fit into the hierarchy given the single-inheritance limitation of C# and VB.NET.

Enter the State Pattern

The ever-resourceful Gang of Four describe the State Pattern as:

Allow an object to alter its behavior when its internal state changes.
The object will appear to change its class.

Taking the bank accounts example and applying the state pattern gives:

Account no longer needs to change type at run-time yet is still able to have clients call Validate and process Methods that end up in discrete methods as if inheritance had been used.

To achieve this we have:

1. Created a State hierarchy that contains the logic we need to change at run-time
2. Introduced a private member in Account that points to the current state instance
3. Ensured the Account methods call the current state class via the private member

Because the state member is private and only accessed by Account itself we can happily create and dispose it as the conditions that affect the state change as much as we like without worrying about references to it.

This is best illustrated with code. Here rather than just calling the state’s validation logic there is a combination of core Account validation (balance), state validation (closed) and CheckingAccount validation (transaction limits):

public abstract class Account {
    private AccountState state;

    public virtual Status Validate(ITransaction tx) {
        Status result = state.Validate(tx);
        if (tx.Amount > Balance)
            result.Add(TransactionFailures.InsufficientFunds);
        return result;
    }
}
public class SavingsAccount : Account {
    public override Status Validate(ITransaction tx) {
        Status result = base.Validate(tx);
        if (Transactions.Count > TransactionLimit)
            result.Add(TransactionFailures.TransactionLimitReached);
        return result;
    }
}
public class ClosedAccountState : AccountState {
    public override Status Validate(ITransaction tx) {
        return new Status(TransactionFailures.InvalidSourceAccount);
    }
}

This is less complex than selectively replacing objects within our application at run-time and can bring additional benefits:

• Single responsibility principle – logic relating to a specific state has its own class to live in
• Favor composition over inheritance – inheritance for OpenSavingsAccount, ClosedCheckingAccount etc. is complex

Like all guidance, patterns and principles do not blindly follow these guidelines or patterns but consider how it affects and fits with your application. For this particular example it not only solves the problem but helps maintainability – at least at this simple stage. Once Validation becomes sufficiently complex it would likely move out entirely into a new set of orchestrated classes just for that.

With LINQ to SQL (and other mappers)

Moving this example into an object-relational mapper requires two - not unexpected - database-mapped properties.

1. The inheritance discriminator (Type)
2. A state indicator (Active)

The only thing we need to ensure is the Account’s state member always refers to either a ClosedAccountState or OpenedAccountState depending upon the Active flag.

Given that LINQ to SQL code-generates the property for Active we could:

1. Make Active private, wrap it in another property and set the state member when it changes and at initialization
2. Make the state member a read-only property instead of an instance variable

The second works well here and given that AccountState is itself stateless (perhaps not the best choice of name) we can use a singleton to avoid multiple instances. The state instance variable in the Account class is replaced with:

private AccountState State {
    get {
        if (Active)
            return OpenAccountState.Instance;
        else
            return ClosedAccountState.Instance;
    }
}

The code continues to work and now changing the Active flag results in different behavior.

Best of all we still have the code in separate classes, no switch/case/if statements relating to validation or account types, a clean inheritance hierarchy and no running around trying to invalidate existing references.

Hitting the discriminator directly

There may be times when claims are made that a type has to change - perhaps data was entered incorrectly.

Before delving into the database or providing a tool to flip the underlying discriminator value consider:

1. Does the new class interpret the same data in a different manner?
   Has a $1,000 credit limit just become a 1,000 transactions per-month limit?
2. Would the object be valid in the new class?
   Did a ProposedCustomer just become ApprovedCustomer without a policy-enforced credit check?
3. Are associations still honored?
   Are 300 unshipped orders for a GameProduct still honored for a BookProduct?

If in doubt don’t do it.

An inconsistent database bleeding through your application isn’t good for anyone and will take a lot longer to sort out than setting up a new entity.

[)amien

The ALT.NET event was particularly interesting. If you didn't already know the name encapsulates:

• the desire to improve the art, process, individual and product
• the understanding that the right tool for the job doesn't always ship with the .NET

The event follows the same minimal up-front planning modern development practices enjoy relying on talented people and simple structure to achieving something great in a short space of time.¹

This open-spaces format involves a small deck of cards to write topics you'd like to talk about on and a board with rooms and times to drop them into. I was surprised at how well the event unfolded (bar one session that veered off-course) given our unplanned attempts at the Guernsey developer group always resulted in five people having dinner in a bar :)

I had an enjoyable day and my thanks go to Brad Wilson (ASP.NET) for driving and listening to my nostalgic 8-bit discussion with fellow brit Ade Miller (p&p) who previously worked at Future Publishing (Your Sinclair magazine). I'm now messing around with a .NET based Spectrum emulator I started years ago and I'm going to blame him for that.

The best news is that there are more events planned for the coming months:

• January 17th normal format, location TBA
• February - provisionally the 7th, regular format, location TBA

ALT.NET Seattle 2009 Conference

This is the big one immediately preceding the MVP Summit by running February 27th to March 1st at Digipen in Redmond. Glenn Block has more details.

Registration will open on Tuesday, 6pm (GMT-8), attendance is free and spaces are limited so get in quick!

Consider joining the Seattle area ALT.NET group or ALT.NET Facebook group for more details or to get involved.

[)amien
¹And as either Glen or Brad said with a nod to the end-of-credits bit in Ferris Bueller - "When it's over it's over. Go home already."

As well as providing the usual blog facilities it also boasts standards compliance, extensibility and a test suite. Nice!

Oxite joins the group of .NET blogging platforms including:

• Subtext
• Dasblog
• BlogEngine.NET
• Graffiti CMS

[)amien

A little background

LINQ was a new feature in .NET 3.5 that provides a store-agnostic query language syntax using a provider model.

As part of that initiative the C# team delivered LINQ to SQL – a LINQ provider to SQL Server with additional update and access management via DataContext and mapping tools such as the designer and SQL Metal. The result is a great lightweight solution that is easy to get started with and a good data access solution where your database and objects are closely aligned.

Meanwhile Data Programmability worked on an initiative to provide abstraction between conceptual and physical models of databases to allow applications to operate independently of the database vendor and underlying physical schema. LINQ support was also added in the form of LINQ to Entities and it shipped with it’s management tools in .NET 3.5 SP1.

Which now means we have two object-relational mapping options that overlap in some areas but with very different backgrounds.

The story so far

Some time before I started in May LINQ to SQL was handed over to Data Programmability.

One of the first things you notice is that the overlap between the two object-relational mapping solutions Microsoft causes confusion and hesitation.

Do I want something lightweight and easy or do I need the extra abstraction with a richer feature set?
If I want to start with LINQ to SQL today but know that the database won’t stay under my control how would I move to Entity Framework?

These are the questions that many developers face and our team spent a lot of time looking at the differences between the two stacks, how they behave and which features were missing if you wanted to migrate from LINQ to SQL to Entity Framework v1. We put together code samples, some helpers and documentation which are now being serialized on the ADO.NET blog and left us with a better understanding of the disparity – some of which the EF team have already addressed in v2.

Where next

The decision has been made that Entity Framework is the recommended solution for LINQ to relational scenarios but we are committed to looking after our users and are approaching this in two ways.

Firstly we are going to make sure LINQ to SQL continues to operate as it should. This doesn’t just mean making sure what we had works in .NET 4.0 but also fixing a number of issues that have arisen as people pick it up for more advanced projects and put it into production environments.

Secondly we will evolve LINQ to Entities to encompass the features and ease of use that people have come to expect from LINQ to SQL. In .NET 4.0 this already includes additional LINQ operators and better persistence-ignorance.

This isn’t to say LINQ to SQL won’t ever get new features. The communities around LINQ to SQL are a continuous source of ideas and we need to consider how they fit the minimalistic lightweight approach LINQ to SQL is already valued for. Where these suggestions fit with this strategy we will be working hard to get them into the product. Some enhancements like the T4 templates can be released independently but runtime elements need to stick to the .NET Framework schedule.

In conclusion

DON’T PANIC
(in large, friendly letters)

LINQ to SQL will continue to work and EF will better address the needs of LINQ to SQL users with each new release.

[)amien

• Stronger aluminium block casing
• NVidia dual graphics for low-power or high-performance
• Glass multi-touch/multi-press trackpad

But the downsides are also worth noting, all of which make me think when I replace my 2.6GHz 17" MBP in a year Apple aren't going to have something I want to replace it with.

• No 17" model
• 4GB RAM limit
• Glossy screen only
• 1440x900 resolution
• Firewire gone

Keith Combs has some similar observations in more depth while AppleInsider is suggesting a January refresh for the 17"

[)amien

1. Settling down in the Redmond area and being with my team are some of the things I'm most looking forward to in October.
2. Sometimes I am so deep in thought when people ask me a question I look dazed and confused, failing to answer them.
3. People grow and situations change and that's why there is a saying, "never say never"!
4. When I'm down, I take a nap, wake up and do something different or creative.
5. Microsoft Building 35 is where you'll find me most often.
6. A rainy day is good for splashing in puddles, getting wet and drying off near something warm with cocoa.
7. And as for the weekend, tonight I’m looking forward to wrapping things up, tomorrow my plans include going out with my Vancouver friends one last time and Sunday, I want to go parkouring and start packing!