Keep your data local

"Keep your data local" - said once a good friend of mine. It didn't mean much to me until some time last week when I was given a task to optimize a .NET application that used way to much memory.

Keeping the story short: if you want to keep your data local use an embedded database like SQLite, SQL Server CE or Firebird Embedded and don't keep all your data in memory at once. Split the process into 3 parts (populate data, process the data, send updates on user's demand) and don't even think about using DataSet for anything that's larger than say 10000 records. It's way to bloated to use it on those amounts of data.

So, if you think that sorting and filtering within a DataView is faster because there's no network overhead and no additional disk IO please be advised that it is way slower than reading records on demand from any of the embedded databases mentioned above and uses a mere fraction of memory by comparison.

And btw - SQLite rules, but its lack of locking on record level is very much annoying...