Random technical stuff

How to use NUnit in Visual Studio 2010

2011-02-22T04:03:00.000-08:00

To run NUnit, you need to add the following to the NUnit.exe.config file:

Under C:\Program Files\NUnit 2.5.9\bin\net-2.0\nunit.exe.config (replace version with whatever you have installed) add the following:

<startup>
   
   <supportedruntime version="v4.0">
</startup>

Also add the following in the <runtime> block:

<loadfromremotesources enabled="true">

The reason is that in .NET v3.5 and below, remote assemblies were loaded partially trusted. Starting with .NET v4, unless you add this directive then you will get a runtime exception.

Once you have done this, add another dll project and add a reference to NUnit.Framework.

What does Vodafone 3G and the Sydney city bus system have in common?

2011-01-11T04:38:00.000-08:00

A note to the NSW government. Adding more buses does not decrease travel times. It actually increases travel times. This is called "bufferbloat" in network systems. Why? Because if you have a limited capacity on a road, and it gets blocked, adding MORE vehicles makes things slower. The only way to fix the issue is to increase the capacity of the roads.

It's the same for train lines. Adding MORE trains won't help anything. You need to add more platforms to stations, and with more platforms you need to lay more rail track. Then you distribute the current number of trains on these tracks and make them go faster than they do. If you then add enough platforms and rail tracks, then you can start adding more trains.

Congestion theory 101. If only the dimwitted politicians would know it. Or perhaps the public? :-)

On that note: Vodafone and 3 - please pay attention! I suspect that this is partly the cause of your incredibly crap service.

My greatest claim to fame

2010-12-28T14:19:00.000-08:00

Sadly, my greatest claim to fame is that I invented the [citation needed] tag. I'll stop sobbing in a corner one day.

P.S. if anyone writes [Citation needed], then [Citation provided].

Markov Chain generator

2010-12-28T01:43:00.001-08:00

As it turns out, with a sufficiently dense input text, a Markov Chain generator is actually reasonably easy to implement.

From page 62 of The Practice of Programming:

set w1 and w2 to the first two words in the text
print w1 and w2
loop:
randomly choose w3, one of the successors of prefix w1 w2 in the text
print w3
print w1 and w2 by w2 and w3
repeat loop

Sargability of monotonic functions in SQL

2010-12-26T07:03:00.000-08:00

An excellent writeup by Quassnoi about search arguments of monotonic functions and how most query engines don't handle them, and how they should.

SQL Server Management Studio 2008 R2

2010-12-01T01:24:00.000-08:00

While it's a great idea to include a debugger to step through T-SQL code, it's not very well done. I frequently get a window handle error, and then I either need to reboot client or restart SSMS.

Microsoft - do some QA testing before release cool new features like this. It's tantamount to baiting me!

Sequence tables in SQL Server 2005

2010-10-21T04:33:00.000-07:00

Interesting post by Paul White about Sequence Tables in SQL Server.

Mistake in Wikipedia article on the Builder pattern

2010-10-19T02:30:00.000-07:00

The following article on the Builder Design Pattern on Wikipedia, is sort of OK, except that the Java example provided is inaccurate.

The abstract PizzaBuilder class uses abstract functions. That's not really recommended - if you subclass this then it means that you have to implement all the functions. But really, the pattern is meant to be more flexible than that - if you don't want to use the function then you shouldn't have to use it. Therefore, as the Gang of Four say in their book on page 101, the build methods should be left intentionally empty and not declared as abstract functions, "letting clients override only the operations they are interested in".

Note that I as I have no interest in having my user pages undeleted, I'm totally unable to correct this mistake. Thanks DuncanHill, you magnificent twat, great contribution to Wikipedia!

ICloneable: Microsoft's big mistake

2010-10-18T03:16:00.000-07:00

So here's the deal: if you implement ICloneable in a public API, you are a making a big mistake. Why? Because there's no way of knowing if the Clone() function will do a deep or shallow copy.

As it turns out, even Microsoft know that they have made a mistake with this interface. Oopsy!

SQL Server indexing...

2010-10-11T02:53:00.000-07:00

There's a series of articles called SQL Server Index Black Ops... and indeed they are!

Posts:

Poor man's LAG() and LEAD()

2010-09-15T06:48:00.001-07:00

How to do LAG() and LEAD() in SQL Server 2008.

Recursive queries vs. nonrecursive queries; and consistent reads vs. currently reads

2010-09-15T04:15:00.000-07:00

Recursive reads vs. non-recursive reads: best to quote Tom Kyte on this one!

non-recursive statements are statements issued by the client to the server.

recursive statements are statements executed by that non-recursive call.

non-recursive statements can be sql, or plsql - they are just "a statement submitted by client to server"

recursive statements can be plsql, or sql - they are just "a statement executed by the statement submitted by the client to the server"

Consistent reads vs. current reads - got the following from the Oracle-l mailing list from Mark Boback:

A 'db block get' is a current mode get. That is, it's the most up-to-date
copy of the data in that block, as it is right now, or currently. There
can only be one current copy of a block in the buffer cache at any time.
Db block gets generally are used when DML changes data in the database.
In that case, row-level locks are implicitly taken on the updated rows.
There is also at least one well-known case where a select statement does
a db block get, and does not take a lock. That is, when it does a full
table scan or fast full index scan, Oracle will read the segment header
in current mode (multiple times, the number varies based on Oracle version).

A 'consistent get' is when Oracle gets the data in a block which is consistent
with a given point in time, or SCN. The consistent get is at the heart of
Oracle's read consistency mechanism. When blocks are fetched in order to
satisfy a query result set, they are fetched in consistent mode. If no
block in the buffer cache is consistent to the correct point in time, Oracle
will (attempt to) reconstruct that block using the information in the rollback
segments. If it fails to do so, that's when a query errors out with the
much dreaded, much feared, and much misunderstood ORA-1555 "snapshot too old".

My app in Ubuntu crashed, but I didn't have debugsymbols...

2010-08-31T07:56:00.001-07:00

Say you've enabled apport on Ubuntu, and you dutifully report segfaults to Launchpad.

Well, normally they won't do you much good with debug symbols. But the bug is already filed! What do you do?

Easy.

$ sudo apt-get install apport-retrace
$ sudo apport-retrace -o /tmp/trash \
> /var/crash/_usr_lib_openoffice_program_soffice.bin.1000.crash

Active Directory trusts

2010-08-14T07:56:00.001-07:00

An excellent article on AD trusts can be found at Tech Republic. Another one can be found at Redmond Magazine.

For posterity

2010-08-13T19:42:00.001-07:00

The following is a very interesting slashdot post on the issues of the NBN and Gigabit speeds...

Expensive, if done right.

Pearson Product Moment Correlation Coefficient

2010-08-12T05:44:00.000-07:00

So finally I understand how the Pearson Product Moment Correlation Coefficient, or also known simply as the correlation coefficient, works.

Firstly, you need to understand the definition of the term correlation. A correlation is a statistical association or relationship between two variables.

A correlation coefficient measures the strength and direction of a linear (think: straight line graph) relationship between two variables using population data.

How do we get the correlation coefficient?

So down to business: the correlation coefficient is the sum of the product of the z-scores over the number of samples.

In other words, it is:

What does this mean?

Well, firstly, the standard deviation allows us to see the variability of data in a dataset. It is the square root of the variance.

Assuming that we know all the values in the sample, then the variance is actually given by taking the deviation of each value from the mean (x - x̄) and squaring it (to ensure that the signs don't cancel themselves out...). Each squared deviation is then summed and the sum is then divided by the number of values in the sample.

In other words, the standard deviation is defined like so:

Now a z score is a way of saying how many standard deviations a value is above or below from the mean value of a particular data set.

You calculate the z score of a value by taking the value (x) and subtract it from the mean (x̄) — this gives you how far from the mean the value is — and then divide by the standard deviation (σ).

Thus, the z-score is:

So if you take the sum of the product of the z scores of the two variables and divide by the number of values, you get:

Now if you expand the Z scores for the two variables, then you get:

This is the same as:

Now if you take the standard deviation of a value x... (also apply this with the variable y)

And then substitute this into the equation (and while doing so also change µ_X to X̄ µ_Y and to Ȳ), then you get:

And down the rabbit hole I go...

2010-08-12T04:08:00.000-07:00

Currently I am reading the book "Statistics for the Utterly Confused". So far it's been excellent - for instance, I now have a good understanding of standard deviation, variability, percentiles and even box-plots.

And so now I start learning about bivariate data, and of course I get to learning about the correlation coefficient. Or more specifically the Pearson product-moment correlation coefficient, of which they present the following spectacular and complex formula:

Of course, there is no attempt to explain why this formula works. So now I'm looking up Wikipedia on this topic. Sadly, Wikipedia does not have a good broad overview of this that I can see, and assumes prior knowledge, so I'm now in the process of following a trail of topics (from bottom to top):

And, as Wikipedia doesn't seem to want to explain Vectors well, I'm also reading the following introduction to vectors.

Update: When I did a Google search, as it turns out there is actually a much simpler version of this formula (who knew?) that is explained here.

Undocumented optimizer feature in SQL Server 2005/2008

2010-08-10T07:21:00.000-07:00

An interesting series of posts has been done by Paul White...

This is the most interesting...

Wow, I got that wrong!

2010-07-31T08:51:00.000-07:00

I wrote the following:

A relational database is so named because the data it stores relates to each other. In SQL, the relationships between data is expressed in a query by what is known as a join.

What I meant was that the tables relate to each other.

That is wrong.

A relational database is so named because of the mathematic concept of the binary relation.

How to debug segfaults in Ubuntu

2010-07-30T07:57:00.000-07:00

Actually, turns out it's not that hard!

You need to do the following:

Firstly, create a build folder.
```
chris@ubuntu:~/src$ mkdir sqlitebuild
```

Now download the source package:

chris@ubuntu:~/src$ cd sqlitebuild
chris@ubuntu:~/src/sqlitebuild$ sudo apt-get source sqlite3

Now you need to build any package dependencies:

chris@ubuntu:~/src/sqlitebuild$ sudo apt-get build-dep sqlite3

Now extract the source files from the dsc file:

chris@ubuntu:~/src/sqlitebuild$ dpkg-source -x foo_version-revision.dsc

We don't want to strip the debug symbols, we don't really need to worry about build tests, and we don't want optimizations (they get in the way of the backtrace), so we need to set theDEB_BUILD_OPTIONS:

chris@ubuntu:~/src/sqlitebuild$ cd sqlite3-3.6.22
chris@ubuntu:~/src/sqlitebuild/sqlite3-3.6.22$ sudo /
> DEB_BUILD_OPTIONS="nocheck noopt nostrip" /
> fakeroot debian/rules binary

sudo DEB_BUILD_OPTIONS="nocheck noopt nostrip" fakeroot debian/rules binary

It will now spit out lots of compilation info.

Now you need to actually install the package, which can be tricky if you've already got it installed.

Most Ubuntu/Debian packages have dependencies. You'll need to force the uninstallation of the package:

chris@ubuntu:~/src/sqlitebuild/sqlite3-3.6.22$ cd ..
chris@ubuntu:~/src/sqlitebuild$ sudo dpkg --purge --force-depends sqlite3
[sudo] password for chris:
(Reading database ... 333636 files and directories currently installed.)
Removing sqlite3 ...
Processing triggers for man-db ...
chris@ubuntu:~/src/sqlitebuild$ sudo dpkg --purge --force-depends libsqlite3
dpkg: warning: ignoring request to remove libsqlite3 which isn't installed.
chris@ubuntu:~/src/sqlitebuild$ sudo dpkg --purge --force-depends libsqlite3-0
dpkg: libsqlite3-0: dependency problems, but removing anyway as you requested:
libsqlite3-0-dbg depends on libsqlite3-0 (= 3.6.22-1).
libmono-sqlite2.0-cil depends on libsqlite3-0 (>= 3.6.13).
libsqlite3-dev depends on libsqlite3-0 (= 3.6.22-1).
evolution-indicator depends on libsqlite3-0 (>= 3.6.22); however:
Package libsqlite3-0 is to be removed.
libmono-sqlite1.0-cil depends on libsqlite3-0 (>= 3.6.13).
evolution-couchdb depends on libsqlite3-0 (>= 3.6.22).
libpackagekit-glib2-12 depends on libsqlite3-0 (>= 3.6.22); however:
Package libsqlite3-0 is to be removed.
evolution depends on libsqlite3-0 (>= 3.6.22); however:
Package libsqlite3-0 is to be removed.
libedata-book1.2-2 depends on libsqlite3-0 (>= 3.6.22).
libwebkit-1.0-2 depends on libsqlite3-0 (>= 3.6.22).
evolution-plugins depends on libsqlite3-0 (>= 3.6.22); however:
Package libsqlite3-0 is to be removed.
evolution-exchange depends on libsqlite3-0 (>= 3.6.22).
libcamel1.2-14 depends on libsqlite3-0 (>= 3.6.22).
libnss3-1d depends on libsqlite3-0 (>= 3.6.22).
libedataserverui1.2-8 depends on libsqlite3-0 (>= 3.6.22).
libopensync0 depends on libsqlite3-0 (>= 3.6.16).
python2.6-dbg depends on libsqlite3-0 (>= 3.6.22).
libqt4-webkit depends on libsqlite3-0 (>= 3.6.22).
python2.6 depends on libsqlite3-0 (>= 3.6.22).
libsvn1 depends on libsqlite3-0 (>= 3.6.16).
packagekit depends on libsqlite3-0 (>= 3.6.22); however:
Package libsqlite3-0 is to be removed.
evolution-data-server depends on libsqlite3-0 (>= 3.6.22).
libebook1.2-9 depends on libsqlite3-0 (>= 3.6.22).
bibledit depends on libsqlite3-0 (>= 3.6.16).
xulrunner-1.9.2 depends on libsqlite3-0 (>= 3.6.22).
libaprutil1-dbd-sqlite3 depends on libsqlite3-0 (>= 3.6.22).
telepathy-gabble depends on libsqlite3-0 (>= 3.6.22).
libgpod4 depends on libsqlite3-0 (>= 3.6.22).
libqt4-sql-sqlite depends on libsqlite3-0 (>= 3.6.22).
libgda-4.0-4 depends on libsqlite3-0 (>= 3.6.22).
libsoup-gnome2.4-1 depends on libsqlite3-0 (>= 3.6.22).
banshee depends on libsqlite3-0 (>= 3.6.22).
python-gpod depends on libsqlite3-0 (>= 3.6.22).
opensyncutils depends on libsqlite3-0 (>= 3.6.16).
(Reading database ... 333630 files and directories currently installed.)
Removing libsqlite3-0 ...
Purging configuration files for libsqlite3-0 ...
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
/sbin/ldconfig.real: /usr/lib/debug/usr/lib/libpython2.6.so.1.0-gdb.py is not an ELF file - it has the wrong magic bytes at the start.

Next, you install the newly built debs:

chris@ubuntu:~/src/sqlitebuild$ sudo dpkg -i libsqlite3-0_3.6.22-1_i386.deb
Selecting previously deselected package libsqlite3-0.
(Reading database ... 333625 files and directories currently installed.)
Unpacking libsqlite3-0 (from libsqlite3-0_3.6.22-1_i386.deb) ...
Setting up libsqlite3-0 (3.6.22-1) ...

Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
/sbin/ldconfig.real: /usr/lib/debug/usr/lib/libpython2.6.so.1.0-gdb.py is not an ELF file - it has the wrong magic bytes at the start.

chris@ubuntu:~/src/sqlitebuild$ sudo dpkg -i libsqlite3-0-dbg_3.6.22-1_i386.deb
(Reading database ... 333632 files and directories currently installed.)
Preparing to replace libsqlite3-0-dbg 3.6.22-1 (using libsqlite3-0-dbg_3.6.22-1_i386.deb) ...
Unpacking replacement libsqlite3-0-dbg ...
Setting up libsqlite3-0-dbg (3.6.22-1) ...
chris@ubuntu:~/src/sqlitebuild$ sudo dpkg -i sqlite3_3.6.22-1_i386.deb
Selecting previously deselected package sqlite3.
(Reading database ... 333627 files and directories currently installed.)
Unpacking sqlite3 (from sqlite3_3.6.22-1_i386.deb) ...
Setting up sqlite3 (3.6.22-1) ...
Processing triggers for man-db ...
chris@ubuntu:~/src/sqlitebuild$ sudo dpkg -i sqlite3-doc_3.6.22-1_all.deb
(Reading database ... 333633 files and directories currently installed.)
Preparing to replace sqlite3-doc 3.6.22-1 (using sqlite3-doc_3.6.22-1_all.deb) ...
Unpacking replacement sqlite3-doc ...
Setting up sqlite3-doc (3.6.22-1) ...

Now to debug the segmentation fault. I'm currently looking into why Firefox is segfaulting on me every one and half minutes or so, and I've worked out that because I have a corrupted places.sqlite (index has gone bad) that it's actually sqlite3 that is having the problem.

I've actually applied this same procedure to Firefox, and I determined that sqlite wasn't happy with the following query:

select h.url, v.visit_date, h.hidden, 0 AS whole_entry FROM moz_places h JOIN moz_historyvisits v ON h.id = v.place_id where v.visit_date < '2000-01-01';

From here, it's pretty easy to hook up gdb to get the full backtrace, or even step through the code.

Update: My efforts have resulted in a bugfix from the sqlite team - yay! Basically, the issue was that Firefox was segfaulting every minute and a half or so. It turns out that my places.sqlite datafile was corrupted, the sqlite guys have now developed a fix.

If anyone else has a similar issue to this one, incidentally, then they can't go wrong with the following:

chris@ubuntu:~/.mozilla/firefox/1u64q3v3.default$ for i in *.sqlite; do echo "Reindexing $i"; echo "reindex;" | sqlite3 $i; done
Reindexing content-prefs.sqlite
Reindexing cookies.sqlite
Reindexing downloads.sqlite
Reindexing formhistory.sqlite
Reindexing permissions.sqlite
Reindexing places.sqlite
Reindexing search.sqlite
Reindexing signons.sqlite
Reindexing urlclassifier3.sqlite
chris@ubuntu:~/.mozilla/firefox/1u64q3v3.default$

Dora!

2010-07-23T17:40:00.001-07:00

http://www.nickjr.com/kids-games/doras-great-big-world-game.html

True geek heaven!

2010-07-02T20:18:00.000-07:00

So I just found a place with a lot of free and legal textbooks.

http://www.freebookcentre.net

It's quite unbelievable!

Yay powersets!

2010-06-28T05:04:00.000-07:00

℘

Boo! Arial can display this correctly, why is blogger doing this? Boo!

Java classpath issues

2010-06-25T07:23:00.000-07:00

So I had a strange issue today. I was getting classpath exceptions. Turns out turning off the JQS on Windows XP fixed it.

But it got me wondering how this works. Here are some links for my own reference:

Dustin's Java development tools blog post
The jps tool.
Jconsole
Actually using Jconsole
ClassNotFoundException vs. NoClassDefFoundError
IBM DeveloperWorks articles on Classpath:

Youtube clips I have posted to my Facebook profile

2010-05-18T08:18:00.000-07:00

My, there are quite a few!

Cross browser code

2010-05-17T06:14:00.000-07:00

Interesting post from Microsoft:

Same Markup: Writing Cross-Browser Code

Update: and damn... look what they've done. meta tags and http headers!

Giano quote file

2010-05-14T11:08:00.001-07:00

o0o

Have you read this [4] are you seriously endorsing it? you thik that is the standards of writing and lower middle class behaviour we should all aspir to? we would be a laughing stock over all of Europe.

Giano on vulgarity

o0o

You have deliberatly and very poorly attempted to misconstrue and portray me in a bad light - but don't wory I expect no less from you - and I think you are the loser for it. However, there is one thing that I do admire about you Mr Wales, your spelling.

Giano on spelling.

o0o

Thanks, I always handle all things to my complete satisfaction, there is only one thing to do with a peanut and I do it. People love to hate me, probably because I have ths iritating habbit of generally being proved right. Sometimes, it takes a while, but sooner or later the editor concerned bites the dust. I put it down to frequent bridge playing and yacht racing, one learns to evaluate one's assets and take few risks.

Giano on his importance and ability

o0o

Not at all, but a reference is a reference and a fact is a fact and a FA is a FA.

Giano explaining references, facts and featured articles

o0o

I suspect this user has an agenda, and I cannot be bothered to discover it.

Giano on suspicious editors

o0o

The civility police behave like a lot of ancient old ladies, sitting knitting mishapen garments, waiting dribbling for someone to use the word "fuck" so they can leap out of their chairs in exitement.

Giano on old ladies, some time before this request.

o0o

Jimbo, sweetheart, this thread has nothing to do with me, please go to the top and start again...

Giano expresses his affection for Jimbo

o0o

Sorry, I find that I have sufficient elderly aunts of my own to protect and keep out of trouble. Anyway, do you think the goat ridden mountains of Sicily a suitable environment for an elderly aunt?"

Giano on aunties

o0o

Running total in SQL Server...

2010-04-27T20:06:00.001-07:00

Not really sure how good this is... but seems to do the job!

;with theData (rowNum, GroupA, GroupRowNum, theValue) as
(
   select row_number() over (order by MajorGroup, GroupOrder),
   DataValue,
   row_number() over (partition by MajorGroup 
      order by MajorGroup, GroupOrder),
   GroupOrder
   from DataTable
)
select X.GroupA, X.GroupRowNum, X.theValue +
   coalesce(
      (select sum(theValue)
      from theData Y
      where Y.GroupA=X.GroupA and Y.GroupRowNum > X.GroupRowNum), 0
   )
from theData X

For some reason I can't get an execution plan as I get an error - I've filed a Connect bug with Microsoft.

How many members should be in the Australian House of Parliament?

2010-03-17T06:49:00.001-07:00

You find this by the following formula:

Quota = Number of people in Australia / (Number of senators in the Senate * 2)

Members of the House of Parliament for each state = Number of people in the state / Quota + (Optional)

Optional is worked out as follows:

If mod(Number of people in the state / Quota) > (Quota / 2) then Optional = 1
Else Optional = 0

However, Members of the House of Parliament for each state must be equal to or greater than 5.

Nasty Oracle ADO bug

2009-11-04T02:58:00.001-08:00

Oracle has a nasty ADO bug that affects 10.2.0.x Windows clients, and I believe 11g R1 clients as well.

When you run a query like select N'X' from dual in an ADO recordset and you are using AL16UTF16 as the national characterset, then if you look at the value of DefinedSize and ActualSize in ADO, you will get a DefinedSize of 1 and an ActualSize of 2.

That's obviously wrong, because DefinedSize is the maximum capacity of the field, and in this case it will be returned as a adVarWChar type. Therefore, it should be being returned as 2, as the letter 'X' in AL16UTF16 is represented as 2 bytes and as it's an adVarWChar type that's all it needs to store the value.

Now if it was the supplementary Han character 𧉧, which is U+27267, the ActualSize will be 4 and DefinedSize will be 4 also. That's because in UTF-16, the actual encoding is 0xD85C-DE67, which is 4 bytes long.

This particular issue has caused my company a great deal of grief, and it took Oracle a long time to acknowledge there was a problem. However, we have been told that this will be fixed in Windows patch bundle 26 for the Oracle 10g client (bug number is 8301952), and even better I believe that it was fixed in Oracle 11g R2.

Great story

2009-10-19T03:00:00.000-07:00

Great story.

Norbert Wiener was perhaps the greatest U.S. mathematician in the first half of the twentieth century, revered among his colleagues for his brilliance. He was also famous for his absent-mindedness.
After a few years at MIT, Norbert Wiener moved to a larger house. His wife, knowing his nature, figured that he would forget his new address and be unable to find his way home after work. So she wrote the address of the new home on a piece of paper which she made him put in his shirt pocket. At lunchtime that day, the professor had a inspiring idea. He pulled the paper out of his pocket and used it to scribble down some calculations. Finding a flaw, he threw the paper away in disgust. At the end of the day he realized he had thrown away his address. He now had no idea where he lived.
Putting his mind to work, he came up with a plan. He would go to his old house and await rescue. His wife would surely realize that he was lost and go to his old house to pick him up. Unfortunately, when he arrived at his old house there was no sign of his wife, only a small girl standing in front of the house. "Excuse me little girl," he said, "but do you happen to know where the people who used to live here moved to?" "It's okay daddy," said the little girl. "Mommy sent me to get you."
P.S. Norbert Wiener's daughter was recently tracked down by a mathematics newsletter. She denies he forgot who she was, but admits he lost the house.
(source)

Planet Gnome

2009-10-12T20:10:00.000-07:00

You know, I really thought that Planet Gnome was all about developing... Gnome applications, frameworks and infrastructure. Silly me. Instead, it's been taken over by a whole lot of PC crap - the latest of which is that Mark Shuttleworth said that he wants to make Ubuntu such that it's "easier to explain to girls". This really put the cat amongst the pigeons, and they are still commenting about it now.

Honestly, I can understand the controversy over RMS's presentation at the Gran Canaria Desktop Summit, which really was sexist, but it appears that it has sparked a wave of self-righteous indignation amongst Gnome people. Seems quite ridiculous.

Evil... but totally cool. Debugging apps remotely via IRC

2009-10-10T04:02:00.000-07:00

The following is so completely evil but completely cool that I just have to tell people about it.

Basically, you get the GNU Debugger (gdb) and hook it up to IRC.

Of course, you'd want to know who you are talking to. Very Insecure.

List of query processing errors in SQL Server 2005

2009-08-21T03:21:00.001-07:00

For my own reference: http://msdn.microsoft.com/en-us/library/ms172350.aspx

Note to myself

2009-08-05T05:49:00.000-07:00

Check the following post for responses:

http://social.msdn.microsoft.com/Forums/en-US/perfmon/thread/fc20f93d-8443-4f62-8d22-556eac66ba71

Update: While somewhat helpful, the MSFT didn't bother looking at my followup question and marked their own answer as the solution. Nice going.

New post to follow up on:

http://social.msdn.microsoft.com/Forums/en-US/perfmon/thread/db3d4e32-1f40-4c95-92a2-02a87649eb56

Well here's something new...

2009-08-05T04:55:00.000-07:00

... a professional sound engineer switched using his Mac to another platform. "Why would anyone use Windows for sound?" I hear you say. Well, they don't. In fact he's switch to... Ubuntu Linux.

Check it out here:

http://createdigitalmusic.com/2009/08/04/linux-music-workflow-switching-from-mac-os-x-to-ubuntu-with-kim-cascone/

Speaking of sound on Linux, I found a good Intro to this topic here:

http://www.osnews.com/story/6720/Introduction_to_Linux_Audio

Guaranteeing order in views

2009-08-03T20:53:00.000-07:00

In SQL Server 2005, using the TOP 100 PERCENT clause with an ORDER BY xxx doesn't gurantee that the results will be ordered by xxx. See this article for more info.

So what to do?

I believe that you can use the following syntax:

create view guaranteedOrderView
select xxx, row_number() over (order by xxx) as OrderNo
from exampleTable

NOTE: I've not tested this assumption. One of the extremely smart developers at my work told me that the optimizer might well... optimize... out this order.

Interesting!

2009-07-22T07:03:00.001-07:00

Discovered a new feature of Windows today.

If you have a service that seems to be playing up, then note down the process ID in the task manager and run the following:

tasklist /svc

Unfortunately, you can have multiple services sharing the one svchost.exe process. If that's the case, then you need to split them off to their own process by doing the following:

sc config <servicename> type=own

You can later make them share the same svchost.exe process by running:

sc config <servicename> type=share

Bah, humbug

2009-07-06T07:11:00.000-07:00

Is there anything more pointless than Twitter? Honestly, it's a technology looking for a solution to a problem that nobody experiences. I don't give a damn whether you just flushed the toilet! And I sure as heck don't care if you just connected your bing bong to your flirp flop through a dongle widget over a Facebook status update by using a new RSS twinkle you just wrote. I say summon the fail whale forever and do an end to this in s anity!

You kids. Get off my lawn!

Bwhahahaha!!!!

2009-06-16T05:19:00.000-07:00

Can't resist posting another blog post. The following folklore is titled "Close encounters of the Steve kind."

Steve had managed to get Don Knuth, the legendary Stanford professor of computer science, to give a lunchtime lecture to the Mac team. Knuth is the author of at least a dozen books, including the massive and somewhat impenetrable trilogy "The Art of Computer Programming." (For an amusing look at Knuth's heady self image, and his $2.56 reward program, see http://www-cs-faculty.stanford.edu/~knuth/books.html )

I was sitting in Steve's office when Lynn Takahashi, Steve's assistant, announced Knuth's arrival. Steve bounced out of his chair, bounded over to the door and extended a welcoming hand.

"It's a pleasure to meet you, Professor Knuth," Steve said. "I've read all of your books."

"You're full of shit," Knuth responded.

Apple story

2009-06-16T04:59:00.001-07:00

One of the best stories about American-Japanese relations I've heard. Better even than the Crazy People ad, interestingly enough also involving Sony. Note that it's talking about how Apply initially needed to get a floppy drive working, and the Apple engineers had to work around the complete looniness of Steve Jobs:

They hatched an alternative plan to continue to work with Sony surreptitiously, against Steve's wishes. Larry Kenyon was given a Sony drive to interface to the Mac, but he was told to keep it hidden, especially from Steve. Bob and George also arranged meetings with Sony, to discuss the customizations that Apple desired and to hammer out the beginnings of a business deal.

This dual strategy entailed frequent meetings with both Alps and Sony, with the added burden of keeping the Sony meetings secret from Steve. It wasn't that hard to do in Japan, since Steve didn't come along, but it got a little awkward when Sony employees had to visit Cupertino. Sony sent a young engineer named Hide Kamoto to work with Larry Kenyon to spec out the modifications that we required. He was sitting in Larry's cubicle with George Crow when we suddenly heard Steve Jobs's voice as he unexpectedly strode into the software area.

George knew that Steve would wonder who Kamoto-san was if he saw him. Thinking quickly, he immediately tapped Kamoto-san on his shoulder, and spoke hurriedly, pointing at the nearby janitorial closet. "Dozo, quick, hide in this closet. Please! Now!"

Kamoto-san looked confused but he got up from his seat and hurried into the dark janitorial closet. He had to stay there for five minutes or so until Steve departed and the coast was clear.

George and Larry apologized to Kamoto-san for their unusual request. "No problem.", he replied, "But American business practices, they are very strange. Very strange."

Ah Apple. When you released the puck mouse, I guess you hadn't learned anything.

Hell just froze over

2009-06-11T06:25:00.000-07:00

Microsoft and the Linux Foundation just presented a joint letter to the American Law Institute. I think you know you've screwed up when such diametrically opposed viewpoints team up together.

Thought I think that CNet said it best - "Finally, Microsoft and the Linux Foundation agree on something. Neither wants to stand behind their products."

Last.fm

2009-06-02T04:17:00.000-07:00

I think that the fact that my favourite track on Last.fm is CD laser lens cleaner speaks volumes for the general quality of today's popular music.

"UAC elevations are not 'security'"

2009-05-31T01:10:00.000-07:00

UAC elevations in Windows Vista are not security.

What are you on about, I hear you say? Of course they are!

Actually, according to Mark Russinovich - of sysinternals fame - UAC elevations are not security.

Signature checks serve as proof-of-origin for trust decisions (e.g. installing an activeX control) and integrity check, not as any indication that the software is non-malicious, free from exploitable defects, or carrying a malicous data payload.
The only code in general checked for signature validity during loading are ActiveX controls, .NET assemblies and device drivers. OS components are not verified except on demand. UAC elevations are not 'security' and the signature verification performed by the consent prompt is intended primarily to encourage ISVs to sign their code.

Excellent post of SSDs

2009-05-28T05:37:00.001-07:00

I seem to be dealing with storage more and more these days. Who would have guessed? Oh well.

Anyway, I found the following excellent article on SSDs.

Method 2's a bit of winner.

2009-05-26T20:55:00.000-07:00

We got an email today from our IT head, subject was "Method 2's a bit of winner."

In the email, it had a link to http://support.microsoft.com/kb/168702

Firstly, the symptom:

When you try to return data from Microsoft Query 97 to a Microsoft Excel 97 worksheet, the spinning globe icon (which signifies that a query is processing) may appear for a long time, and then the query returns no data to your worksheet.

And now, possible solution 2:

Method 2: Move Your Mouse Pointer

If you move your mouse pointer continuously while the data is being returned to Microsoft Excel, the query may not fail. Do not stop moving the mouse until all the data has been returned to Microsoft Excel.

NOTE: Depending on your query, it may take several minutes to return the results of your query to the worksheet.

New storage book

2009-05-21T05:56:00.000-07:00

I bought a new book entitled "Information Storage and Management", which is written by EMC² experts. I'm up to the second chapter, and it's all very dry so far. However... I think it's going to start getting interesting because it goes from the basics of how hard disk drives work and how their components fit together to... 2.4 Fundamental Laws Governing Disk Performance. Which tells me the following about average queue size:

N_Q = N - U
= a × R - U (from eq. 1)
= a × (R_S / (1 - U)) - U (from eq. 5)
= (R_S / R_a) / (1 - U) - U (from eq. 3)
= U / (1 - U) - U (from eq. 4)
= U (1 / (1 - U) - 1)
= U² / (1-U) (6)

Whew! Glad I knew that :-)

Sounds like some interesting reading.

Oracle Dump() statement

2009-05-13T04:10:00.000-07:00

Discovered an interesting Oracle statement today. The statement is dump. The syntax is Dump(value)

This returns the raw value of a row's column.

Here are the possible values:

8 returns result in octal notation.
10 returns result in decimal notation.
16 returns result in hexadecimal notation.
17 returns result as single characters.

If you add 1000 to it, it returns the characterset.

The following website gives some good examples:

dump('Tech') would return 'Typ=96 Len=4: 84,101,99,104'
dump('Tech', 10) would return 'Typ=96 Len=4: 84,101,99,104'
dump('Tech', 16) would return 'Typ=96 Len=4: 54,65,63,68'
dump('Tech', 1016) would return 'Typ=96 Len=4 CharacterSet=US7ASCII: 54,65,63,68'
dump('Tech', 1017) would return 'Typ=96 Len=4 CharacterSet=US7ASCII: T,e,c,h'

Unicode and Oracle

2009-05-12T03:40:00.000-07:00

In my work for a software company, I get a lot of questions and problems relating to Oracle and Unicode. The following series of posts will be a summary of how these two not-particularly-easy-to-use-or-understand components fit together.

Before I get into how Oracle and Unicode fits together, I really feel that it's instructive to understand both what Unicode is, and how it came to be. Therefore I'm starting with...

A potted history of the precursors to Unicode

Even before ASCII... or even proper computers

To really understand Unicode, you first need to understand a little about the background to what came before Unicode. The most detailed history I know actually gives the history of ASCII, and doesn't even touch on Unicode, but if you're into that sort of thing (I am!) then it should be interesting.

Basically, the first non-time based encoding scheme (i.e. not Morse Code) that was invented was used in Telegraphs and was called the Baudot Code. This was created by Emile Baudot in 1876 and it was itself inspired by the Gauss-Weber Telegraph Alphabet. The code was initially represented by 6-bits, but it was later reduced to a 5-bit encoding scheme (32 characters). This patented encoding was the basis of a later code that was called the Murray Code, which was invented by Donald Murray who used it to automatically print telegraphs to a punch-tape based printer he had also invented. The code was also a 5-bit encoding but introduced control characters we still see today - characters such as the Carriage Return (CR), Line Feed (LF), NULL, BLANK and DEL.

Murray's patent was later sold to the Western Union Telegraph Company in 1912. While it was patented, a number of incompatible variations of his code were being used in other Telegraph systems by this time - an issue that not even Murray was too worried about in the beginning. However as reliable communications became more and more important, governments and businesses started to realise that they needed to standardize this code so that they could communicate more easily with each other.

The French actually work with (most of) the rest of the world

Thus in 1925 the French setup the Comité Consultatif International Télégraphique (CCIT, or in English the "International Telegraph Consultative Committee") for the purposes of creating an internationally recognized standardized encoding. This proved to be no mean feat - especially given that they only had 32 letters to work with! As it turns out, standardizing this coding scheme was particularly difficult because the Russians objected to what everyone else initially agreed on. Rather than bore you with what went on, just understand that eventually two encoding schemes were formed - the International Telegraph Alphabet 1 (ITA-1) which the Russians were particularly fond of, and the International Telegraph Alphabet 2 (ITA-2) which everyone else quite liked.

As you can imagine, with almost everyone but the Russians using ITA-2, use of ITA-1 soon fell by the wayside and in 1948 it was adopted (with reservations) by the United States as their standard for Telegraphy. Of course when that happened it was pretty much all over for ITA-1. What I find interesting is that evidently in those days they were more cautious about adopting standards, as it took 19 years to adopt the ITA-2 encoding - enough time for World War I and World War II to have been and gone for some time!

The Americans take over

Now while it was grand that the world had agreed that ITA-2 was the one everyone should use, 5-bits is really quite limiting and so it was inevitable that someone would decide that the encoding set should be expanded. Thus in the 1960s Captain William F. Luebbert of the U.S. Army Signal Research and Development Lab invented FIELDDATA, which was a 6-bit encoding that was used extensively by the U.S. Army in their communications. I only really mention FIELDDATA as it is historically important because it inspired the committee that invented ASCII. More on ASCII later though.

In the meantime, another 100-pound gorilla was also inventing their own encoding scheme. That gorilla was IBM, who in 1962 created the 6-bit Binary Coded Decimal Interchange Code (BCDIC) for use in their punched card machines. When punched cards were replaced with IBM's System/360 mainframes, Charles E. McKenzie of IBM extended the code, which became known as Extended Binary Coded Decimal Interchange Code (EBCDIC). Of course, IBM didn't standardize this on their own machines until a few years went by... but it is still in wide use to this day. In fact, EBCDIC was long the rival of ASCII, but the X3.2 subcommittee rejected it. Don't worry, we'll get to this very shortly!

The birth of ASCII

Given the number of different encodings that had started to proliferate, the American Standards Association (ASA; they are now known as ANSI) decided that it was time to form a new international standard to represent characters. Thus on June 17, 1963 the ASA's X3 Committee for Computer and Processing Standards formed the X3.2 Subcommittee for Coded Character Standards and Data Formats. Within this subcommittee was formed a core group of nerds known as the X3.2.4 taskgroup. To cut a long story short, our small band of heroes spent so long arguing over such things as how many characters they should represent, whether to use control characters and in what position they should put the characters in the code chart that they never met any girls and thus never produced any offspring. Whether that was because they were off the nerd Richter scale, or whether it was because ASCII ruined them is a point hotly debated. Whatever it was, we shall never have a generation of men like them again. The important thing here is that with all that incredibly dull discussion, you would have thought that the resulting standard included such innovations as lowercase letters, umlauts and character grave-marks. Strangely, this was not to be and the world was initially stuck with a bunch of upper case characters, numbers, punctuation symbols and a lot of obscure control characters.

Incredibly, it took another four years for the X3.2.5 working group to decide on a final 7-bit encoding scheme. This time, however, someone must have told them that nobody wants to COMMUNICATE BY SHOUTING. It was only at this point that the members got a clue and decided that the world should actually be allowed to write electronically with proper punctuation and lower-case letters. Incidentally, it appears that they didn't keep their controversies inhouse, and managed to anger another set of (albeit cooler) nerds. What happened was that they were just about to release the final standard that just about every standards body in the world had approved - including the ECMA and ISO - when who should come along but the president of the IBM User Group, otherwise known as SHARE. For their troubles, the X3.2.4 subcommittee working group was broadsided by a vitriolic letter in which the SHARE president threatened that the ASA could go to hell unless changes were made to the draft standard. If they were not, he warned that the programmers of the world would create their own competing standard and ASCII as we know it would be in peril due to a lack of adoption. Faced with this unpalatable situation the X3.2.4 working group decided to pull the wool over the SHARE president's eyes by moving a few characters into different positions as well as changing the form of a few characters - for instance they adding a break in the middle of the "|" character (have a look at your current keyboard to see what I'm talking about).

Thus in 1967, with SHARE suitably triumphant and mollified - not to mention feeling very smug about forcing changes to a standard that every country in the world had agreed to - the ASA officially released the American Standard Code for Information Interchange (ASCII), or more formally X3.4-1967. This was actually a joint release done with the European Computer Manufacturers Association (ECMA) and the International Standards Organization (ISO). The ECMA released ASCII as ECMA-6 and ISO released a slightly modified version of ASCII as ISO-646. The ISO standard differs from ASCII because it replaced the dollars sign with the International symbol for currency, which is ¤. Personally, I think that while this certainly seems a good idea in theory, when you realise that currencies fluctuate against each other then you start to understand that unless you show the actual symbol of the currency being used you could either make quite a bit of money, or lose quite a bit of money. Possibly this is why nobody has ever heard of the universal symbol of currency, which ironically is not used universally. What was the ISO thinking?

As an aside, you'd think that after all that careful consideration and endless argument amongst nations (and even within nations - IBM's EBCDIC was categorically rejected by the X3.2.4 committee) that people wouldn't want to fiddle with the standardized characterset. But no, what we got was ATASCII, PETSKII, the ZX Spectrum characterset, Galaksija and YUSCII. Some people are never happier than when they are buggering up a perfectly good standard.

8-bits of confusion - Extended ASCII

It soon became apparent to almost everyone that while 7-bits certainly gave everyone a lot more characters to play with, it wasn't really enough to represent even a fraction of the world's characters. Computers by this stage all used the byte as the smallest unit of measurement anyway, so really there was no need to only represent characters with 7 bits. Also, by this time the IBM PC had been unleashed on an unsuspecting public, and with it came an 8-bit extended ASCII variant called PC-US (sometimes also known as OEM-US or DOS-US). This extended ASCII characterset was burnt into the ROM of every IBM and IBM compatible PC that was sold, which obviously made it very popular, especially as it included characters that let you do things like this:

╔═══════════════════════════╗
║ SWEET BOX WITH TEXT IN IT ║
╚═══════════════════════════╝

(The box would be even sweeter if Blogger honoured non-breaking spaces).

That one extra bit allowed for a whopping 255 characters, 127 more than were available than before. The rest of the world soon cottoned on, and a large number of ASCII variants appeared; these included two version of the Greek alphabet, many variants of the Cyrillic languages, Arabic, Chinese, etc, etc, etc. Of course they all used the same code points to represent their various languages, so eventually IBM organized these into what are now known as code pages - the original being code page 437. Soon thereafter, Microsoft made it big with Windows, and decided to add to the confusion with their own set of code pages.

For a while, confusion reigned. In particular, BASIC programmers were annoyed because when they switched their graphics card to another mode that used a non-437 code page their sweet graphics would turn into a big mess of characters. And you never want to make a BASIC programmer angry...

ISO to the rescue!

Evidently realising that angry BASIC programmers aren't a good thing, ISO (now known as ISO/IEC) decided to remedy the situation. Unfortunately for the BASIC programmers this is the same organization that nearly caused financial chaos by replacing the dollar symbol with the universal currency symbol, so unfortunately for them they didn't get any ANSI graphic symbols. And hence BASIC died, to give birth to the hell-spawn known as Visual BASIC... sorry, I digress.

Basically, ISO/IEC sat down and decided to use the 8th bit of extended-ASCII to form a proper universal standard. Thus was born the ISO 8859 charactersets, of which there were eventually 15 different versions.

ISO 8859-1; this is the most well known. It's also called Latin-1, and you'll often see reference to it in databases and in formats such as MIME encoded emails. It covers characters from most Western European languages.
ISO 8859-2; (Latin-2) covers characters from Central and Eastern Europe
ISO 8859-3; (Latin-3) covers Esperanto and Maltic languages
ISO 8859-4; (Latin-4) covers Baltic languages
ISO 8859-5; (Cyrillic) covers Bulgarian, Byelorussian, Macedonian, Russian, and Serbian languages
ISO 8859-6; (Arabic) does not cover all of the Persian or Pakistani Urdu languages
ISO 8859-7; (Greek)
ISO 8859-8; (Hebrew)
ISO 8859-9; (Latin-5) has Turkish characters
ISO 8859-10; (Latin-6) covers Nordic languages
ISO 8859-11; (Thai)
ISO 8859-12; sorry, those who use Devanagari missed out - this never eventuated.
ISO 8859-13; (Latin-7) covers languages written in the Baltic Rim
ISO 8859-14; (Latin-8) covers Celtic languages
ISO 8859-15; (Latin-9) Latin-1 on steroids; includes the Euro symbol and a few other obscure characters

Now there are only really two places I know of where you can get a list of ISO 8859 character maps. The first is the following Debian page, and of course the second is Wikipedia which not only lets you copy and paste the characters but gives you an excruciating amount of info on the charactersets. Sort of like this blog post, only without the sarcasm.

Yet more charactersets

Of course, this now meant that there were more charactersets than you could poke a stick at. In actual fact, there are even more charactersets than this because ISO-646 had dozens of variations. Not only this, but there appeared another class of characterset encodings called the Double-Byte Character Set (also known as DBCS). DBCSes were actually the real precursor to Unicode in my mind, because they were the first to attempt to use more than a byte to represent characters. You can read more about DBCSes on Wikipedia. While you are there, if you are interested it's worthwhile reading about another ISO/IEC standard - ISO/IEC 2022 - which uses variable-sized encoding to represent characters and is mainly used to represent characters of East-Asian languages.

More to come later

Well, I'll write more later, as it's quite late here and writing about the precursors to Unicode is actually quite tiring.

Finding what font renders Unicode characters in the PUA

2009-05-08T06:15:00.000-07:00

Unicode 5 allows for 137,468 privately used codepoints. This means that there are codepoints that are reserved for private organizations to do with what they will - which means that no characters have been officially assigned to them.

In practice, this means that you need to have some way of knowing where the document with the embedded Unicode character came from. This is normally left up to a high-level protocol to negotiate how to render the character.

A great example is the character U+F8FF, which is used by a number of fonts. According to Wikipedia, this font is the Apple Computer logo with Apple fonts installed, the Microsoft logo when using Wingdings, and the Luxi font (designed for the X Window System) shows a euro symbol. There are a heck of a lot of other fonts that also use this codepoint.

And this was my challenge the other day. I'm currently in the process of troubleshooting a particularly tricky Unicode issue, but I wanted to see what used U+100084, but finding what uses this codepoint was a bit tricky. Then I discovered that fileformat.info has a Unicode font list and a Unicode font search tool. I had to use the local file list, and for some reason I found that my PC had Palatino Linotype installed on it - a very expensive font. However, it was the only font that rendered the character. Therefore, I came to the conclusion that this came from a PDF document, which had the font embedded.

Very useful!

P.S. Please, please, if you are thinking of using a font that requires a private use codepoint, do everyone a favour. Don't.

Pronouns - oh my!

2009-03-06T03:45:00.000-08:00

Recently I've been editing a lot of technical articles. Poor grammar and the use of passive voice has apparently been a bit of an issue for me!

In the course of understanding the difference between active and passive voice, I stumbled over HyperGrammar, which is an electronic grammar course at the University of Ottawa's Writing Centre. It's an excellent website, one which I'm interested in reading - especially as the Australian school system never taught me grammar as it fell out of fashion with the teaching profession. A terrible mistake we are now all paying for - I understand that grammar is now back in fashion, so I suppose that there is still hope.

I just started reading about what a pronoun is, but as it's almost 11PM, I decided to stop when I discovered you can use subjective personal pronouns, objective personal pronouns, possessive person pronouns, demonstrative pronouns, interrogative pronouns, relative pronouns, indefinite pronouns, relative pronouns and, finally, intensive pronouns!

Richard Simmons

2009-02-26T05:38:00.000-08:00

I have a confession. I'm a bit of a fan of, all people, Richard Simmons. Why? Well, he's a pretty positive guy, he doesn't hold grudges (go watch any of the thousands of clips of him appearing on Letterman), he's happy about himself and he's confident enough that he doesn't mind being totally camp or make himself look a bit foolish in front of millions of people. Most of all, he seems like a fairly kind individual, and that is one of the most important aspects to anyone really.

You see, I think that when most people watch Richard Simmons they don't see a guy who has a very strong will, but I do think this. Richard Simmons is happy in his own skin (literally - *shudder*) that it really doesn't matter what mockers like Letterman think. Simmons goes out and does what he wants to do, and to hell with what others think of him. And in the process, he's actually helped out a reasonable amount of people who do feel bad about themselves.

Therefore, Richard Simmons, I salute you. You may have shocking dress sense, you may act the fool, you might show your 60+ year old legs on Letterman far too often, and you may market ridiculous steamers, but you are still an unlikely role model for our generation.

KFC nutrition

2009-02-19T05:39:00.001-08:00

Figures.

World, meet Emily

2009-02-19T05:07:00.000-08:00

Warning: exceedingly cute baby pictures ahead.

How To Kill Redundancy With a Redundancy

2009-02-19T03:15:00.000-08:00

Terry Childs was a network administrator of the San Francisco FibreWAN. When I say a network admin, I really mean the network admin - given that he was the only one who looked after it. So while I'm impressed he is one of the world's few CCIEs, I do think that this is really a bit too much for anyone to take on by themselves.

So when he went rogue and wouldn't disclose the passwords to the Cisco routers and passwords he administered, I was somewhat gobsmacked. Not at Terry Childs, mind you, but at the dumb-arse morons who left a sole employee the administrator and contact for their entire critical networking infrastructure. Seriously, what would have happened if the man had expired? If he'd dropped off the mortal coil then they would not have been able to recover the passwords at all. Luckily Childs turned them over to Mayor Gavin Newsom, so a big problem was averted.

I'm sure that the San Fran network infrastructure had built-in redundancy. But it looks like they forgot the most important redundancy of all - the people to administer it.

Python takes Youtube

2009-02-15T04:09:00.000-08:00

I notice that there is a "report background image" link down the bottom of the Monty Python YouTube channel. All hail Monty Python!

Coffee printer

2009-02-03T00:35:00.000-08:00

On slashdot I found this awesome printer, that prints with coffee. Hmmmm... if I bought this I think my caffeine addiction would only get worse :-)

This made me look at the website where it was submitted. It was part of the Greener Gadgets design competition, which in its own right is pretty interesting. Go check it out!

On Not Getting It Redux

2009-02-01T03:40:00.000-08:00

I thought John C. Dvorak could not have looked more ridiculous with his "my Windows XP idle process is killing my computer!" line, but I stumbled across the following article today. In it, he rails against CSS.

If your Internet connection happens to lose a bit of CSS data, you get a mess on your screen.

How on earth did this guy get to be a widely known and respected pundit on all things technology?

I pity the fool!

2009-01-20T03:41:00.000-08:00

I pity the fool... who allows users to control text output via a URL.

Nice going TV Guide!

On Not Getting It

2009-01-20T00:56:00.001-08:00

On Slashdot today I read a post entitled "Do Nice Engineers Finish Last?" which had the best first post ever. Which then got better.

First post:

Do Nice Engineers Finish Last In Tough Times?
Why, just the other day, a coworker was in contention for a promotion that was going to a younger engineer. My coworker found the specs to the younger engineer's car online and determined the precise rate it would have to leak coolant to completely drain the reserve tank precisely when he was leaving home to make an important customer meeting the next morning. I saw him on a crawl board attaching the regulator and a valve system in the parking lot and sure enough it overheated at precisely the right time so our customer just sat their waiting.

It's a calculate-or-be-calculated world out there!

Response:

Aside from the fact that your post is a load of horseshit, I suppose that you didn't step up to the plate by telling management what you witnessed.

And, incidentally, once the youngster took his car to the shop to be repaired, the tampering would have been discovered, and your fictional coworker would have been thrown in jail (hmm just where did this after market valve and regulator come from anyway?). In most states tampering with an automobile is a felony.

I think this might have spiked the parent poster's conscience, because he replied:

Alright alright, I need to come clean ... I embellished on this story a little bit. Here's the truth:

I was going to tell my boss but when I walked in, the coworker I was ratting out was on his knees with a mouthful of my boss and I think he said, "Oh hai!" I didn't stick around to clarify, I just left.

And it wasn't a car, it was a hovercraft. And it wasn't a regulator & valve, it was a detonator & C4. And he wasn't late for a meeting, he died. And don't worry about the law, Virginia isn't a state it's a commonwealth.

I feel almost relieved to get that off my chest and to come clean with you. I think I answered all your questions truthfully and fairly. Hopefully, together you and I can keep the internet a sound unbiased source of nothing but the unadulterated truth and historic account of everything.

You've helped me help myself. I love you.

Stupid bash tricks

2009-01-19T04:40:00.001-08:00

The following is a very stupid bash command to run.

:(){:|:};:

This is a fork bomb, so don't run it without having set process limits.

I was trying to understand how this worked, and I found the following blog that set me straight. What was tricking me was the colon... it didn't look like it should be valid but it is. Who knew? Well, certainly not me.

The site I note above helpfully suggests changing the colon to "bomb", which gives us:

bomb() {
bomb | bomb
}; bomb

Stupid slashdot tricks

2009-01-19T04:24:00.000-08:00

Well, only one. It looks like the Slashdot crew are bigger fans of Futurama than I realised!

I got this from a slashdot sig.


chris@ubuntu:~$ echo -e "HEAD / HTTP/1.1\nHost: slashdot.org\n\n" | netcat slashdot.org 80
HTTP/1.1 200 OK
Date: Mon, 19 Jan 2009 12:23:53 GMT
Server: Apache/1.3.41 (Unix) mod_perl/1.31-rc4
SLASH_LOG_DATA: shtml
X-Powered-By: Slash 2.005001237
X-Bender: Farewell, big blue ball of idiots!
Cache-Control: private
Pragma: private
Connection: close
Content-Type: text/html; charset=iso-8859-1

chris@ubuntu:~$ echo -e "HEAD / HTTP/1.1\nHost: slashdot.org\n\n" | netcat slashdot.org 80
HTTP/1.1 200 OK
Date: Mon, 19 Jan 2009 12:24:02 GMT
Server: Apache/1.3.41 (Unix) mod_perl/1.31-rc4
SLASH_LOG_DATA: shtml
X-Powered-By: Slash 2.005001237
X-Fry: People said I was dumb but I proved them!
Cache-Control: private
Pragma: private
Connection: close
Content-Type: text/html; charset=iso-8859-1

chris@ubuntu:~$ echo -e "HEAD / HTTP/1.1\nHost: slashdot.org\n\n" | netcat slashdot.org 80
HTTP/1.1 200 OK
Date: Mon, 19 Jan 2009 12:24:07 GMT
Server: Apache/1.3.41 (Unix) mod_perl/1.31-rc4
SLASH_LOG_DATA: shtml
X-Powered-By: Slash 2.005001237
X-Leela: There's a political debate on. Quick, change the channel!
Cache-Control: private
Pragma: private
Connection: close
Content-Type: text/html; charset=iso-8859-1

chris@ubuntu:~$ echo -e "HEAD / HTTP/1.1\nHost: slashdot.org\n\n" | netcat slashdot.org 80
HTTP/1.1 200 OK
Date: Mon, 19 Jan 2009 12:24:11 GMT
Server: Apache/1.3.41 (Unix) mod_perl/1.31-rc4
SLASH_LOG_DATA: shtml
X-Powered-By: Slash 2.005001237
X-Fry: Stop abducting me!
Cache-Control: private
Pragma: private
Connection: close
Content-Type: text/html; charset=iso-8859-1

chris@ubuntu:~$

Dvorak and the nefarious "idle process"

2009-01-13T00:45:00.000-08:00

While most people know that John Charles Dvorak once famously said that "the Macintosh uses an experimental pointing device called a ‘mouse’. There is no evidence that people want to use these things. I don't want one of these new fangled devices", they might not also know that he said the following in PC World about the Windows XP idle process:

This week's column is about exploring the commonly observed problems that crop up with each new release [of Windows]. Maybe Microsoft should patch the patches once in a while.

Here are a few of my gripes – most of them a result of excessive patching

IDLE-TIME PROCESS. Once in a while the system will go into an idle mode, requiring from five minutes to half an hour to unwind. It's weird, and I almost always have to reboot. When I hit Ctrl-Alt-Delete, I see that the System Idle Process is hogging all the resources and chewing up 95 percent of the processor's cycles. Doing what? Doing nothing? Once in a while, after you've clicked all over the screen trying to get the system to do something other than idle, all your clicks suddenly ignite and the screen goes crazy with activity. This is not right.

Reply-all storms

2009-01-10T20:21:00.000-08:00

Ah yes, the old reply-all storm. Never good, but even worse when it takes out U.S. diplomatic mail servers!

According to this report on Associated Press:

Officials said the storm started when some diplomats used the 'reply all' function to respond to a blank e-mail sent recently to many people on the department's global address list.

Most demanded to be removed from the list while others used 'reply all' to tell their co-workers, in often less than diplomatic language, to stop responding to the entire group, the officials said.

Some then compounded the problem by trying to recall their initial replies, which generated another round of messages to the group, they said.

The best email storm I've heard of though, is the one involving journalists who were accidentally sent a mass email from the Casey Journalism Center at the University of Maryland inviting them to their "Casey Medals" Awards.

According to Editor and Publisher the email snafu caused some interesting effects:

The back-and-forth sparked a circle of never-ending responses that, in some cases, kept hundreds of e-mails filling electronic mailboxes over several hours on Tuesday and Wednesday morning. But, in an unexpected surprise, it also brought many journalists in touch with old colleagues, while forging a number of new industry connections through something of an online cocktail party.

"People started chit-chatting back and forth and inviting themselves to the awards," said Kim Platicha, editor and publisher of Parentwise Austin magazine in Austin, Texas. "It really evolved from there, it was hysterical. I have already started an e-mail conversation with a couple of folks."

What killed the Zune30?

2009-01-03T17:14:00.000-08:00

So we've been hearing a lot about Microsoft Zune 30s crashing. Microsoft have now said that it was a leap year bug.

And indeed it is! From Pastie (start at line 249):


//------------------------------------------------------------------
//
// Function: ConvertDays
//
// Local helper function that split total days since Jan 1, ORIGINYEAR into
// year, month and day
//
// Parameters:
//
// Returns:
//      Returns TRUE if successful, otherwise returns FALSE.
//
//------------------------------------------------------------------
BOOL ConvertDays(UINT32 days, SYSTEMTIME* lpTime)
{
int dayofweek, month, year;
UINT8 *month_tab;

//Calculate current day of the week
dayofweek = GetDayOfWeek(days);

year = ORIGINYEAR;

while (days > 365)
{
 if (IsLeapYear(year))
 {
     if (days > 366)
     {
         days -= 366;
         year += 1;
     }
 }
 else
 {
     days -= 365;
     year += 1;
 }
}


// Determine whether it is a leap year
month_tab = (UINT8 *)((IsLeapYear(year))? monthtable_leap : monthtable);

for (month=0; month<12;>wDay = days;
lpTime->wDayOfWeek = dayofweek;
lpTime->wMonth = month;
lpTime->wYear = year;

return TRUE;
}

Why is this bad? Well, 2008 was a leap year that has 366 days. Let's step through the lines of code that caused the problem.

//Calculate current day of the week
dayofweek = GetDayOfWeek(366);

year = 2008;

while (366 > 365)
{
if (IsLeapYear(2008))
{
if (366 > 366)
{
   days -= 366;
   year += 1;
}
}
else
{
days -= 365;
year += 1;
}
}

As you can see, the while loop condition becomes true - yes, the day is day 366 and that's greater than 365. And yes, 2008 is a leap year. But as you can see, 366 will never be greater than... 366.

Therefore, the loop condition never evaluates to false, hence an infinite loop. Thus your Zune will crash.

Guess Freescale, the makers of the Zune's processor (the MC13783), had a programmer who didn't understand about boundary conditions.

Update: Another blogger has now gone and suggested a few bug fixes for the Zune issue. Nice going :-)

SQL joins

2008-12-27T14:44:00.001-08:00

A relational database is so named because the data it stores relates to each other. In SQL, the relationships between data is expressed in a query by what is known as a join.

There are a number of different types of joins. Most joins can be either defined as an implicit join or an explicit join. An implicit join is one where there is no explicit JOIN statement yet there is a join implicit in the WHERE clause of the query. An explicit join is one where there is actually a JOIN statement in the FROM clause of the query.

An explicit join is also known as a qualified join, and uses the syntax

SELECT <columns>
FROM <table>
<join type> JOIN <table>
ON <join condition>

Cross joins

Given two tables, each row in the inner table joins each row of the outer table. I can think of almost no examples where you'd want to use this, but the implicit join syntax is:

SELECT * FROM
OrgUnit, Person

The explicit join syntax is:

SELECT * FROM OrgUnit
CROSS JOIN Person

A cross join is also known as a Cartesian product or a Cartesian join. I have never needed to use it! Note that this is the only join that does not use the ON syntax, as it's not needed and if you needed to restrict the data set then you'd need to filter in the WHERE clause of the query. Of course, if your filter does a comparison against two related columns of the datasets, then you have an INNER join, described below.

Inner joins

An "inner" join is named because given two tables, the query will return each row in the inner table that has a match in the outer table.

To show an example of this, I'll have another look at my two tables I introduced in the cross join section. They are OrgUnit and Person.

OrgUnit
OrgUnitID	OrgUnitName
1	Manufacturing
2	Accounting
3	Human Resources
4	Tiger team
5	IT
6	Consultancy
7	Special projects

Person
PersonID	FirstName	LastName	PhoneNumber	Address	OrgUnitID
1	Chris	Sasquatch	03 9998 7656	5 Curumbah St, Gladstone, VIC, 3145	5
2	Jill	Anderson	03 9765 3975	23 Sesqatonia Rd, Smallville, VIC, 3087	6
3	Robert	Manaheim	03 7659 8765	18a Xyz St, Whoknowswhere, VIC, 3009	2
4	Mysterious	Guy	02 8765 4321	78 Unknown St, NSW, 2000	1
5	Captain	Obvious	02 1111 1111	54 Duh Lane, NSW, 2119	4
6	Mickey	Mouse	07 7654 6665	1 Disney Rd, QLD, 7654	7
7	Donald	Duck	07 7654 6665	5 Drewery Lane, WA, 8097	NULL
8	Daffy	Duck	08 7654 3214	8 St Kilda Street, WA, 8008	7
9	Bugs	Bunny	08 7654 3214	1004 Killawee Rd, NT, 6543	7

If I wanted to find out all the Organization Units that each invididual resides in, I would run the following explicit query:

SELECT
p.FirstName,
p.LastName,
p.Address,
p.PhoneNumber,
o.OrgUnitName
FROM Person p
INNER JOIN OrgUnit
ON p.OrgUnitID=o.OrgUnitID

The implicit query is:

SELECT
p.FirstName,
p.LastName,
p.Address,
p.PhoneNumber,
o.OrgUnitName
FROM Person p, OrgUnit o
WHERE p.OrgUnitID=o.OrgUnitID

This gives the following results:

FirstName	LastName	PhoneNumber	Address	OrgUnitName
Chris	Sasquatch	03 9998 7656	5 Curumbah St, Gladstone, VIC, 3145	IT
Jill	Anderson	03 9765 3975	23 Sesqatonia Rd, Smallville, VIC, 3087	Consultancy
Robert	Manaheim	03 7659 8765	18a Xyz St, Whoknowswhere, VIC, 3009	Accounting
Mysterious	Guy	02 8765 4321	78 Unknown St, NSW, 2000	Manufacturing
Captain	Obvious	02 1111 1111	54 Duh Lane, NSW, 2119	Tiger team
Mickey	Mouse	07 7654 6665	1 Disney Rd, QLD, 7654	Special projects
Daffy	Duck	08 7654 3214	8 St Kilda Street, WA, 8008	Special projects
Bugs	Bunny	08 7654 3214	1004 Killawee Rd, NT, 6543	Special projects

Outer joins

An "outer" join is so named because given two tables, each row in the outer table is returned even if it does not meet the ON condition. There is no implicit join syntax for this sort of join - you must use the ANSI syntax.

There are a number of forms of outer joins: left outer join, right outer join and full outer join. Let's explore each of these.

Left outer joins

A left outer join returns the "left" rows of an outer join regardless of whether they met the ON condition or not. If the right hand table does not meet the join condition, then a NULL is displayed.

Let's say that I wanted to find all the people in my database and if they are part of an organization unit. I would do this with the following query:

SELECT
p.FirstName,
p.LastName,
p.Address,
p.PhoneNumber,
o.OrgUnitName
FROM Person p
LEFT OUTER JOIN OrgUnit
ON p.OrgUnitID=o.OrgUnitID

This would return the following results:

FirstName	LastName	PhoneNumber	Address	OrgUnitName
Chris	Sasquatch	03 9998 7656	5 Curumbah St, Gladstone, VIC, 3145	IT
Jill	Anderson	03 9765 3975	23 Sesqatonia Rd, Smallville, VIC, 3087	Consultancy
Robert	Manaheim	03 7659 8765	18a Xyz St, Whoknowswhere, VIC, 3009	Accounting
Mysterious	Guy	02 8765 4321	78 Unknown St, NSW, 2000	Manufacturing
Captain	Obvious	02 1111 1111	54 Duh Lane, NSW, 2119	Tiger team
Mickey	Mouse	07 7654 6665	1 Disney Rd, QLD, 7654	Special projects
Daffy	Duck	08 7654 3214	8 St Kilda Street, WA, 8008	Special projects
Bugs	Bunny	08 7654 3214	1004 Killawee Rd, NT, 6543	Special projects
Donald	Duck	07 7654 6665	5 Drewery Lane, WA, 8097	NULL

I have highlighted in bold red the row that is returned in the left outer join that was not returned in the inner join. As you can see, Donald Duck exists in the left hand table, but is not part of a organization unit. With the left outer join he is still returned in the output, however as he is not part of any org unit it displays a NULL value.

Right outer joins

A right outer join is the counterpart of the left outer join. It returns the "right" rows of an outer join regardless of whether they met the ON condition or not. If the left hand table does not meet the join condition, then a NULL is displayed.

Let's say that I wanted to find all org units and if they have any members show who they are. I would run the following query:

SELECT
p.FirstName, p.LastName, o.OrgUnitName
FROM Person p
RIGHT OUTER JOIN OrgUnit
ON p.OrgUnitID=o.OrgUnitID

This returns the following results:

FirstName	LastName	OrgUnitName
Mysterious	Guy	Manufacturing
Robert	Manaheim	Accounting
NULL	NULL	Human Resources
Captain	Obvious	Tiger team
Chris	Sasquatch	IT
Jill	Anderson	Consultancy
Mickey	Mouse	Special projects
Daffy	Duck	Special projects
Bugs	Bunny	Special projects

Again I have highlighted in bold red the important row that is returned by the right outer join. The human resources org unit of this company has nobody in it (silly rabbits!) but we still want to know all the org units of the company. Thus as the table OrgUnits is on the right hand side of the join we return the organization but NULLs for the left hand side table rows.

Full outer joins

A full outer join allows us to display each row in both left and right tables. If there is a row in the right hand table that does not match the ON join condition, then it displays the right hand table row and NULL values for the left hand table. If there is a row in the left hand table that does not match the ON join condition, then it displays the left hand table row and NULL values for the right hand table.

This may seem a bit like a cross join, but it is not. A cross join matches every row for the left and right hand tables regardless of the relationship between the tables. The full outer join maintains the relationships between the tables, it's just if either side does not have a matching row then it displays the row on the right/left hand side regardless.

If I wanted to see all the OrgUnits and each member of the OrgUnit, but I also wanted to see all the people in the database regardless of whether they belong to an OrgUnit, I would run the following query:

SELECT
o.OrgUnitName,
p.FirstName,
p.LastName,
p.PhoneNumber,
p.Address,
o.OrgUnitName
FROM Person p
FULL OUTER JOIN OrgUnit
ON p.OrgUnitID=o.OrgUnitID

FirstName	LastName	PhoneNumber	Address	OrgUnitName
Chris	Sasquatch	03 9998 7656	5 Curumbah St, Gladstone, VIC, 3145	IT
Jill	Anderson	03 9765 3975	23 Sesqatonia Rd, Smallville, VIC, 3087	Consultancy
NULL	NULL	NULL	NULL	Human Resources
Robert	Manaheim	03 7659 8765	18a Xyz St, Whoknowswhere, VIC, 3009	Accounting
Mysterious	Guy	02 8765 4321	78 Unknown St, NSW, 2000	Manufacturing
Captain	Obvious	02 1111 1111	54 Duh Lane, NSW, 2119	Tiger team
Mickey	Mouse	07 7654 6665	1 Disney Rd, QLD, 7654	Special projects
Daffy	Duck	08 7654 3214	8 St Kilda Street, WA, 8008	Special projects
Bugs	Bunny	08 7654 3214	1004 Killawee Rd, NT, 6543	Special projects
Donald	Duck	07 7654 6665	5 Drewery Lane, WA, 8097	NULL

Semi and anti-semi joins

The last sort of joins I want to talk about are semi-joins and anti-semi joins.

A semi-join will return rows from a table when there is one or more corresponding rows in the joining table. Interestingly, there isn't actually any explicit syntax for semi-joins, instead you must use the EXISTS statement.

For instance, let's say that I want to find all OrgUnits that have at least one Person. We could do the following:

SELECT o.OrgUnitName
FROM OrgUnit o
INNER JOIN Person p

However, this will return more than one record for the OrgUnit "Special projects".

Instead, a semi-join will do this for you. The syntax is:

SELECT o.OrgUnitID
FROM OrgUnit AS o
WHERE EXISTS
(
SELECT p.OrgUnitID FROM Person AS p
WHERE p.OrgUnitID = o.OrgUnitID
)

You can also do this by using an IN statement:

SELECT o.OrgUnitName
FROM OrgUnit AS o
WHERE o.OrgUnitID IN
(
SELECT p.OrgUnitID FROM Person AS p
WHERE p.OrgUnitID = o.OrgUnitID
)

An anti-semi join however, does the exact opposite. It will find any rows that do not have at least one matching row in the second table. To do this, use the NOT EXISTS statement.

Going back to our previous example, if we wanted to find all the departments that do not have any members, we would do the following:

SELECT o.OrgUnitID
FROM OrgUnit AS o
WHERE NOT EXISTS
(
SELECT p.OrgUnitID FROM Person AS p
WHERE p.OrgUnitID = o.OrgUnitID
)

With the NOT IN statement it would be:

SELECT o.OrgUnitName
FROM OrgUnit AS o
WHERE o.OrgUnitID NOT IN
(
SELECT p.OrgUnitID FROM Person AS p
WHERE p.OrgUnitID = o.OrgUnitID
)

Note 1: I picked this IN trick up at the following blog.
Note 2: I asked over at SQL Server Central whether they thought it would be a good idea to add a TOP 1 clause to the query. i.e. the correlated subquery becomes:

SELECT TOP 1 p.OrgUnitID FROM Person AS p
WHERE p.OrgUnitID = o.OrgUnitID

However, it was politely suggested I test myself. Fair enough I suppose :-)

Another way of doing a FULL OUTER join

Interestingly, we can get a FULL OUTER join by doing a left outer join of Table1 to Table2, then do a UNION ALL of a left anti-semi join of Table2 to Table1. Or in other words:

Table1 LEFT OUTER JOIN Table2
UNION ALL
Table2 LEFT ANTI-SEMI JOIN Table1

I picked this up from the book Inside Microsoft SQL Server 2005: Query Tuning and Optimization. It's useful because a nested loop can't do a FULL OUTER join, except that you can transform the FULL OUTER join into joins that the nested loop can handle. Which I thought was quite neat, in a totally geeky way of course :-)

Attention span of a gnat on meth

2008-12-26T19:26:00.001-08:00

Thank you, Slate.

C warning flags

2008-12-25T23:16:00.000-08:00

A short blog post today. Just found an excellent post explaining the different gcc warning flags and what they do, by GNOME hacker Benjamin Otte.

Query tuning

2008-12-23T02:55:00.000-08:00

I think that I've found the best blog post on SQL tuning I've ever read.

It's by Jeff Moden, and it's entitled More RBAR and "Tuning" UPDATEs.

I had a similar issue the other day, where I had a query that was taking almost 7 minutes to retrieve 6,000 records. For some reason it was doing a UNION of two SELECT statements, yet the second SELECT statement was the same as the first - so I thought that this was what was causing the issue.

And sure, when we took out the second SELECT statement the query reduced to about 3 and a half minutes.

Yet that was absolutely still too long. When I looked at the statement a bit further, I noticed that it was doing a LEFT OUTER join against a view that used a count function, a correlated subquery and a CASE statement, and not only that but when the actual SELECT statement found a NULL yet another CASE statement was used to convert the NULL into a value. The query plan was hideous - for 6,000 records in the final query output the LEFT OUTER join to the view was bringing back 1.6 million rows.

Hello RBAR!

So I did exactly what Jeff did - read the view and the query to work out what it was trying to do, then I rewrote the view to be more sensible. All of a sudden I went from about 3.5 minutes to run the query to 7 seconds.

So lesson learned - RBAR bad, set-based good.

Memory in Windows

2008-12-12T02:54:00.000-08:00

During memory performance troubleshooting on Windows servers, I've often had to go to the task manager - and at times I've needed to use the Performance Monitor. The biggest problem I had when I first used these tools was understanding the different types of memory counters/columns in these tools. It has taken me a reasonable amount of research to understand the difference between such counters as virtual bytes, private bytes and working set.

In case anyone else has wondered what the different memory counters are in the Performance Monitor, or have been confused by the column labels in the task manager, the following is a brief summary.

Virtual memory

Before I explain, a quick explanation is probably due as to the meaning of the term "Virtual Memory". Probably the best explanation I have ever read is a 1992 Microsoft technical article entitled The Virtual-Memory Manager in Windows NT. However, in a nutshell, Windows uses a 32-bit contiguous virtual address space which can address up to 4 GB of memory. Don't get too excited about this, because by default 2GB of virtual address space is reserved for the operating system, though this can be changed to just 1GB with the /3GB bootup switch.

Each virtual address is actually just a placeholder to a physical location of memory in the machine, and when accessing memory Windows must jump through a few hoops to translate the virtual address to the physical address. In essence it must look up a 3 level btree to get to the address, first looking up a page directory, which refers to an entry in a page table, which in turn refers to a page frame. Note that for faster memory access Windows (and in fact all modern operating systems) relies on the processor's translatable lookaside buffer (TLB) to gain direct access to the memory.

The interesting thing about virtual memory is that each page table entry may either refer to an area of physical memory or to a pagefile on disk. This allows Windows to address more memory than is physically available via the system's RAM. When a program looks up an area of memory that is backed by a pagefile, the process is suspended by the operating system and the data is retrieved from the disk and placed into physical memory. This is known as a page fault. When the page has been retrieved from the disk and into memory, the operating system unsuspends and returns control back to the process.

This is often a point of confusion for people, the best example of which was recently on Slashdot under an "Ask Slashdot" question "Why Use Virtual Memory in Modern Systems?":

"I have a system with Windows Vista Ultimate (64-bit) installed on it, and it has 4GB of RAM. However when I've been watching system performance, my system seems to divide the work between the physical RAM and the virtual memory, so I have 2GB of data in the virtual memory and another 2GB in the physical memory. Is there a reason why my system should even be using the virtual memory anymore? I would think the computer would run better if it based everything off of RAM instead of virtual memory. Any thoughts on this matter or could you explain why the system is acting this way?"

The confusion here is that the poster is asking quite a different question, which is "why use a pagefile in modern systems?", which I think for workstations is actually a fairly reasonable question!

I also find it interesting that in Windows Vista Microsoft have tried to address this in their description of the virtual bytes counter of Performance Monitor, as they now state in the counter's description that "Use of virtual address space does not necessarily imply corresponding use of either disk or main memory pages." Obviously Microsoft tech support got heartily sick of having to explain this concept to the many people who misunderstand it and call them about it!

Performance monitor

The performance monitor (perfmon.exe, I'll call it perfmon from now on) is a tool in all versions of Windows since Windows 2000 that can access the various counters that are built into Windows, as well as other third party counters for products such as SQL Server.

If you had a process that you believed was leaking memory, then you can run perfmon to find out if this is the case. You would simply create a new log, then you'd add a number of process counters for the process you feel is causing the problem. Alternatively, if you didn't know what process was causing issues you could just add the process object to capture all counters for every process, of course you would then need to remove hundreds of counters when you loaded the file... but sometimes you've just got to do these sort of things when it's not clear what's going on. Painful, but true. :-)

The problem, then, becomes a matter of knowing what counters to actually look at. The following are the important ones:

Virtual bytes are the number of bytes of virtual memory that the process is using in total
Private bytes are the number of bytes of virtual memory that is being used by the process alone and that it can't share with other processes. We must distinguish between this counter and the virtual bytes counter because the process can actually load memory-mapped files, EXEs or DLLs into its virtual address space. Private bytes is a counter that does not include these things. As such, you will almost always see that virtual bytes is significantly larger than private bytes.
Working set bytes are the set of virtual addresses used by the application that are actually resident in memory (i.e. are NOT paged to the disk)
Page file bytes are, fairly obviously, the set of virtual addresses that reference a page in the pagefile. When memory is low, incidently, you can also add in the counter Page faults/sec to see how often the memory is being swapped from disk into memory. The more page faults per second, the slower the process will run because suspending a process while a page is retreived from a disk can be quite an expensive operation!
Paged pool bytes/Paged non-pool bytes - I had to look this one up! The best explanation is actually on Microsoft's Ask the Performance Team where they have a whole article on understanding pool resources. Rather than reinvent the wheel, I'll just quote them:
When a machine boots up, the Memory Manager creates two dynamically sized memory pools that kernel-mode components use to allocate system memory. These two pools are known as the Paged Pool and NonPaged Pool. Each of these pools start at an initial size that is based upon the amount of physical memory present in the system. Pool memory is a subset of available memory and is not necessarily contiguous. If necessary, these pools can grow up to a maximum size that is determined by the system at boot time.
For more info on this area of memory, the linked article explains it all very well so I won't explain it any further here.

I should note that after you capture these counters, when you view them again you will probably need to set the scale to something more sensible like 0.000001, or else you just won't see a line on the graph!

Task Manager

When something is going slowly, most people who know enough about Windows look at the processes tab of the Windows Task Manager.

I have to say before I talk about the task manager that though I've really only just started to use Windows Vista (well, I hardly use it at all, I prefer Ubuntu Linux for home use) despite its shortcomings Microsoft really put some care in the ancillary utilities like the Performance Monitor (now part of the Reliability and Performance Monitor) and the task manager. The task manager under Vista is really quite impressive now!

However, I'm afraid that I've only just started looking at the Vista task manager, so I won't talk about it much here. I might edit the article later and rewrite this section. :-)

The main areas to look at in the Task manager are the mem usage column and the VM size column. While it might seem very straight forward to understand what these columns do, given the discussion above, the Task manager programmers must have been on crack as they don't report what you might think.

Mem usage seems very vague for a description - after all, does this refer to the amount of virtual memory being used, the amount of actual unpaged memory, perhaps the total amount of memory exclusively used by the process? Well, it is actually the working set, which if you remember is all the virtual memory addresses that are resident in physical memory.
VM size is really a very odd name to give a column. After all, VM size is actually 4GB on 32-bit CPUs as virtual memory is all the memory that Windows can possibly address. Not surprisingly, confusion reigns supreme about this column and it is often confused as either the virtual memory that is paged to disk or the amount of virtual memory being used by a process. It is neither of these things, instead it is actually the amount of private memory being used by the process (remembering that private memory is memory that is being used exclusively by the process and doesn't take into account things like dlls, etc).

Anyway, I do hope that this is somewhat enlightening. Researching this took me some time and while searching google I found it frustrating seeing so much contradictory and confusing information about how to answer a question that at first glance seem to be straight forward but isn't: "how much memory is my program using?"

varchar(MAX) truncating

2008-12-10T04:08:00.000-08:00

I was reading Karen Delaney's excellent Inside SQL Server 2005 book about the SQL Server 2005 storage engine, and she explained quite a neat feature: varchar(MAX).

varchar(MAX) is a funny sort of varchar, only you aren't restricted to only 8000 bytes (or characters - to varchar it's all the same) but in fact the maximum length of the column is the maximum length of any LOB type (2^31). Basically, if a column is set to varchar(MAX) and you have less than 8000 bytes then SQL Server internally stores the column as a varchar column, and if you exceed 8000 bytes then it stores it as LOB data.

The interesting thing about varchar(max) is that it appears that unlike TEXT columns you can actually apply normal string manipulating SQL on it (i.e. replicate, left, right, etc). Yes, that's right, no more mucking about with TEXTPTR manipulation!

The trap to watch out for is that if you decide to do something like join a varchar(MAX) value to a varchar value in order to create a new varchar(MAX) value, if the new value becomes greater than 8000 characters then the new value will be truncated to 8000 bytes.

e.g.

declare @theText varchar(MAX)
set @theText = replicate('x', 7095) + 'the end bit'
select len(@theText)

(shamelessly stolen from this blog)

While you might think that the length of the string will show as 8006, the length is actually 8000.

Why does this occur?

The answer is actually found in SQL Server books online, under the section Precision, Scale and Length. This says that:

When two char, varchar, binary, or varbinary expressions are concatenated, the length of the resulting expression is the sum of the lengths of the two source expressions or 8,000 characters, whichever is less.
When two nchar or nvarchar expressions are concatenated, the length of the resulting expression is the sum of the lengths of the two source expressions or 4,000 characters, whichever is less.

Therefore, be warned! unless you cast the string to a varchar(MAX), then you might not get the results you expect!

For the above example, incidently, you would do the following:

declare @theText varchar(MAX)
set @theText = replicate(cast 'x' as varchar(MAX)), 7095) + cast('the end bit' as varchar(MAX))
select len(@theText)