Monday’s post on CopyExt 2 turned into more of an advert than a post containing actual information, which I did expect as I was still on that I’ve just finished a project buzz. So I thought I’d do a post-mortem post of sorts, partly for my benefit so that I don’t forget the things I’ve learned, and partly because I think it might be interesting. Before continuing I would like to extend my gratitude to a good friend of mine, Dan Abbatt, who donated these cool new icons to replace those I originally used. Much better!

Now onto the technical stuff; the following sections aren’t really arranged in any particular order and, strangely, only one of them actually focuses on the coding behind CopyExt. This post has grown very large actually, but I hope you find it useful, or at least interesting.

Nullsoft Scriptable Install System

An installer was a key deliverable (is it still a deliverable if you’re your own customer?) that I wanted to provide with this release. The previous version of CopyExt was very programmery, for lack of a real word, and relied on the user manually registering the DLL and massaging Windows Explorer back to health if something went wrong. I decided to use NSIS to build my installer primarily because I’d seen the end result when installing a pretty large amount of the software I regularly use. I didn’t look for any other offerings or even attempt to find alternatives so keep that in mind throughout the rest of this section.

My first impression, based on the wiki and tutorials I found, was that it was a simple tool with an enormous amount of potential. Now that I’ve used it I can say that I think I was right. As the name implies, it uses scripts to generate the install programs. The smallest possible script that will also create an uninstaller is as follows:

OutFile "MyInstaller.exe"

Section "" ; Default section doesn't need a name
    SetOutPath $INSTDIR
    File file1.whatever
    File file2.whatever ; repeat as needed
    WriteUninstaller "Uninstall.exe"
SectionEnd

Section "Uninstall"
    Delete "$INSTDIR\Uninstall.exe"
    Delete "$INSTDIR\file1.whatever"
    Delete "$INSTDIR\file2.whatever" ; repeat as needed
    RMDir  "$INSTDIR"
SectionEnd

For some projects, these commands might even be the only ones you need! It does offer much more functionality than this if you need it (registry access and shortcut creation to name two common things). It even provides instructions to read and write files so that you can actually generate content in the installer itself for configuration variables and stuff (think Apache web server or something).

Places to get information

NSIS has a comprehensive set of documentation that even features a pretty good tutorial, but I also used the following places to get more information (this is by no means an exhaustive list):

Tutorial: Create a NSIS install script is a very basic introduction that is really more of an overview for non-programmers.
Quick Guide to NSIS does exactly what it says on the tin but it does the classic computer science textbook thing of teaching you the baseline and then jumping into advanced topics (in this case, the plug-in architecture and how to make your own).
Using NSIS To Make Installable Java Applications only real focus on Java is that there’s a .class file in it; most of the stuff is actually quite transferable.
Creating an Installer is my personal favourite because it has a good pace and each part is very well explained. The others had a tendency to jump about a bit to show off the cool features of NSIS whereas this is a practical tutorial that prepares you for the cool stuff.

After I’d already written my NSIS script, compiled it, and hosted on my website, I finally got around to searching for comparisons between it and Inno Setup which is another free installer compiler. The search obviously found several articles proclaiming that one package is better than the other because it just is, but this article and this forum thread stood out as being the most impartial.

It seems that Inno Setup has a smaller learning curve for making simple installers (such as the one I needed) but becomes complex (with the introduction of a scripting language) when more sophisticated behaviours are required. NSIS doesn’t suffer from this problem because the scripting language is required from the most basic installer up to the most complex one imaginable. Keep in mind that this paragraph is entirely speculative because I haven’t used Inno Setup.

Some Gotcha’s

Overall I think I only encountered two weird things while making the installers:

Firstly, RegDLL and UnRegDLL are the built-in instructions for [un]registering COM DLLs, but I think they do so by loading the DLL, directly finding the addresses of DllRegisterServer and DllUnregisterServer, and then invoking whichever is required. Now this isn’t an issue if the installer and the DLL are both the same binary format (i.e. 32bit or 64bit) but in my case I’m shipping separate binaries for each platform.

Getting around this problem is quite easy thankfully. The code below invokes the program regsvr32 which, despite the name, will happily work with 32bit and 64bit DLLs regardless of the parent process. The /s argument is only there to suppress the message box that reports the result of the operation.

ExecWait '"$SYSDIR\regsvr32.exe" /s "$INSTDIR\CopyExt.dll"'

Secondly, I learned that internet shortcuts on Windows are not .LNK files! I had always assumed that a .URL file (i.e. internet shortcut) was identical to a .LNK type shortcut file, probably because I a) rarely use them and b) always use the wizard. They are actually text files (open them in Notepad) structured like the (in)famous .INI files:

[InternetShortcut]
URL=http://www.google.co.uk/

There might be some other stuff in there too but this seems to be largely irrelevant to having a working link. So to generate an internet shortcut in the Program Files menu as I have in the CopyExt installer, the following code is used:

WriteINIStr "Google Search.URL" "InternetShortcut" \
    "URL" "http://www.google.co.uk/"

You can have an ordinary .LNK point to a web address but, for whatever reason, the icon defaults to the empty file icon. If you use a .URL shortcut then you get whatever the user has registered as the default browser which looks a lot better.

I first started learning C++ programming by writing Windows programs using its native API so this project should’ve fallen well within my scope of knowledge, but I a learned several new things (which is good, I like learning new things). All of these things actually revolve around menus. After I had a decent version of CopyExt working, I decided I wanted icons on the menus to make them look a bit more interesting. Well, it turns out that this wasn’t as straight forward as I hoped because there’s a compatibility issue.

Windows XP introduced the whole Visual Styles thing which provided its signature look. Every Windows component went through this API (if you had a correctly written manifest, of course). Every component, except menus which kept the strange behaviour from Windows 98 whereby if you add bitmaps to a menu item, they get colour-inverted on highlight and look awful. The solution to this was to set the bitmap to the magic constant HBMMENU_CALLBACK which causes the messages WM_DRAWITEM and WM_MEASUREITEM to be sent to your window.

Context menu shell extensions can handle these messages if they implement the IContextMenu2 or IContextMenu3 interfaces (the latter derives from the former) which provide the functions HandleMenuMsg and HandleMenuMsg2 respectively. The only difference between the two functions is that the latter allows a result to be returned through a pointer to an LRESULT. But it’s optional, so HandleMenuMsg actually just does this in CopyExt:

STDMETHODIMP CCopyShellExt::HandleMenuMsg(
    UINT uiMsg, WPARAM wParam, LPARAM lParam)
{
    return HandleMenuMsg2(uiMsg, wParam, lParam, NULL);
}

So all of the actual processing happens in HandleMenuMsg2, but in reality there’s not much that needs doing! Well, at least there shouldn’t be. I discovered something very weird about the WM_MEASUREITEM message:

LPMEASUREITEMSTRUCT lpMIS = (LPMEASUREITEMSTRUCT)lParam;

// Only interested in handling the menu item
if (lpMIS == NULL || lpMIS->CtlType != ODT_MENU)
    break;

// Make sure the menu is sufficiently large for the icon
UINT cyIcon = GetSystemMetrics(SM_CYMENUCHECK);

if (lpMIS->itemHeight < cyIcon)
    lpMIS->itemHeight = cyIcon;

lpMIS->itemWidth = 0; // hack for XP: prevent enormous gutters

If lpMIS->itemWidth was anything other than zero, then the menu gutters (the bit where the icons and checkboxes appear) grew to twice the size it should’ve been! The hack I used above (i.e. forcing it to be zero) seems to prevent that from happening. I don’t know if this is an Explorer-specific thing or not so it’s something to bear in mind. Thankfully the drawing side of things is much more straight forward:

LPDRAWITEMSTRUCT lpDIS = (LPDRAWITEMSTRUCT)lParam;

// Only interested in handling the menu item
if (lpDIS == NULL || lpDIS->CtlType != ODT_MENU)
    break;

// Determine which icon to load
HICON hIcon = LoadIconForCommand(lpDIS->itemID);
if (hIcon == NULL)
    break;

// Calculate the XY coords of the icon
int cxIcon = GetSystemMetrics(SM_CXMENUCHECK);
int cyIcon = GetSystemMetrics(SM_CYMENUCHECK);
int xIcon  = (lpDIS->rcItem.left - cxIcon) / 2;
int yIcon  = lpDIS->rcItem.top +
             (lpDIS->rcItem.bottom - lpDIS->rcItem.top - cyIcon) / 2;

// Draw the icon in the rectangle provided
DrawIconEx(lpDIS->hDC, xIcon, yIcon, hIcon, cxIcon, cyIcon,
           0, NULL, DI_NORMAL);

// Free the icon resource
DestroyIcon(hIcon);

The only potential issue here is that LoadIconForCommand (which you’ll need to write yourself) must map Explorer’s command ID back onto your own internal representation of commands. I just used an array for this, where the index was the offset from Explorer’s ID (i.e. lpDIS->itemID - <base command ID>) and the value was my own internal command ID. If this works correctly, then the icons will appear as in the following image.

If this exact code is then executed on Windows Vista or Windows 7 then, all of a sudden, the menus look completely different to the regular menus. This is because menus are no longer sacred and are now part of the visual style framework, but when at least one menu item is marked as ownerdrawn (as is automatically the case with HBMMENU_CALLBACK) then the whole menu drops back to the original way of working in the interests of backwards compatibility. The image below shows this happening.

In principle, the fix for this is very easy: 32bit bitmaps are natively supported as of Windows Vista, which means 24bits of colour and 8bits of alpha. So we can specify a bitmap with an alpha channel instead of the callback and it will be drawn correctly for us by Windows. But what if you don’t have an alpha channel? (As was the case before the new icons were donated). Well, then you have a bit of work on your hands as you will clearly see if you look at the code in the MenuBitmap.cpp file. The process I used is quite simple though, it just requires a reasonable amount of code:

Create an off-screen buffer using CreateDIBSection to guarantee that it will have 32bits per pixel.
Draw the icon onto this buffer (using memory device contexts as necessary with CreateCompatibleDC/DeleteDC) using DrawIconEx for best results — DrawIcon is useless here because it will always draw an icon as the system icon size which will probably be bigger than what you want for menus.
Drawing the icon in this manner probably has resulted in transparent pixels having an alpha channel of zero but, unfortunately, almost every GDI function is oblivious to alpha so the opaque pixels will probably also be transparent. To get around this, get the mask bitmap from the icon using GetIconInfo (remember to close the bitmaps when you’re done with them), and then read the bits into a mallocated (or whatever) buffer with GetDIBits.
Now the mask bits and the colour bits (which were already returned from CreateDIBSection) can be walked in lockstep. Masks are monochrome bitmaps that are AND‘d with the destination pixels to clear a dirty great black blob, then the colour data is XOR‘d on top to replace the black with the colour. So this implies that a white pixel in the mask is actually a transparent pixel and hence the corresponding colour pixel should be cleared completely (alpha of 0 is completely transparent), and for each black mask pixel the corresponding colour pixel should be OR‘d with 0xFF000000 to set the alpha to 100% (i.e. completely opaque).
Finally, the 32bpp bitmap returned from CreateDIBSection will now contain an icon with correct alpha channel data that can be used directly on the menu! All of the other bitmaps and stray device contexts and whatever can all be released/deleted/freed.

There is one small caveat with this technique: if an icon already contains alpha information it will be destroyed, so the colour bitmap pixels should be walked in search of alpha data; if any is found then do not do step 4. I do apologise for the long-winded description here but the code is too big to insert. Check out MenuBitmap.cpp to see for yourself!

The final piece to this whole messy jigsaw is a humble if-statement. Use GetVersionEx to find out what version of Windows your program is running under. If the major version number is at least 6 then your program is running on Windows Vista and hence the long-winded conversion technique described above is necessary but you can set the bitmap directly, otherwise force it to be HBMMENU_CALLBACK to invoke the message handlers.

Embarrassing “bug”

After I’d done all of the guff above and I recompiled my code I was faced with a rather hellish situation: only the settings icon actually appeared! This caused me a rather large amount of confusion because all of the image-handling code was identical between the two menu items as they called the same functions! Sadly, after far too much time was wasted on unnecessary debugging, I stumbled on a forum response in the seventh part of Michael Dunn’s awesome series on shell extensions.

InsertMenu doesn’t allow you to specify an ID if you insert an item as a popup menu, as a result of this the draw/measure item messages were never being sent because there was no item ID to relate to. InsertMenuItem, however, allows you to specify every property a menu can have. Yes, I chose the wrong function, sadly. I kicked myself so hard when I finally realised this, so I recoded all of the menu stuff around InsertMenuItem exclusively.

Visual Studio tips

Finally, just a few minor details that are probably obvious but still worth a mention. I used Visual Studio 2010 (gotta love Dreamspark) for this project so this section probably isn’t terribly useful if you use something else.

COM libraries must have the appropriate registry keys set for the COM service to be able to pick them up, this is usually done in the DllRegisterServer function that you should implement. I discovered (after my extension seemingly randomly re-registered itself, which left me thinking the registration code was wrong) that there’s a project setting under the Linker page called Register Output. Usually this setting is off, but in ATL/COM projects (perhaps even any project that invokes the MIDL compiler) it defaults to on. Be wary of this when debugging as it lead to a lot of confusing results for me.
I wanted to ensure that CopyExt was easy to distribute so I spent a while looking at how to embed the various Microsoft redistributable packages in an NSIS installer until suddenly it dawned on me: static linkage. This is one of those really obvious things that I wish was a bit more obvious to me last week. Rather than messing around with additional DLLs and awkward dependencies, static linkage shoves all the stuff your code requires in the output binary meaning you have no dependencies at all. This is really useful for small projects like this where I didn’t want to force the user to install a wealth of other stuff just for a few menu options.

Source code management

I decided to use Git (for Windows) as the version control for this project because I had an existing account with Bitbucket that I hadn’t used yet. They give free unlimited accounts to those in academia, so how could I refuse!? We use SVN repositories internally but I’d heard good things about Git. From a user-interface perspective they are very similar, the differences are primarily in the back-end: SVN is centralised whereas Git is distributed.

I found that Git feels more natural in this respect because you can commit your changes as you see fit and then push a feature to the server when it’s completed and working. I was the only person working on this project so I didn’t reap the full benefits of the system unfortunately. Here is a very brief summary of the commands I found useful:

git add is the command that you use to stage files for a commit. Every file must be explicitly added for it to be included in a commit. Therefore Git breaks modified files into two categories: staged and unstaged. git add will add a file to a repo and/or mark it as staged depending on what’s required
git rm and git mv delete and move files around the repo respectively.
git status works similarly to the SVN equivalent but it gives you a bit more information. All modified files will be listed and will be broken into the groups staged and unstaged. It will also tell you the number of local commits that have not yet been pushed to the server.
git reset allows you to unstage a change but it will not affect the contents of the file.
git checkout will unstage a change and restore the file to the HEAD revision (similar to SVN revert).
git commit actually writes the staged changes to your local copy of the repository. The command line switch -a is useful if you want to commit every change (including unstaged ones). You should specify a commit message with -m "message" or simply run the command without this option and Git will invoke whichever editor you configure it to use (msysGit defaults to Vim).
git push (specifically git push -u origin master) is the equivalent of the commit of the SVN world — it pushes the changes in the local branch/tag (master) upstream (-u) to the central repo (origin).

This list is by no means exhaustive but it’s enough to get started with a Git repository. I quite liked using Git despite being on the only person on the project if for nothing else than the commit/push separation gave a conceptual division between tasks.

Aside: Source code packaging

Anyone who knows me is well aware that I’m a large fan of Python. I like its simplicity, its power, but most of all, its enormous standard library. I wrote a script to build the source archive of CopyExt for me by doing the following:

Using the git ls-files command to find out what files are committed to the local repository.
Optionally loading a text file (specified as a command-line argument) that contained a list of files that should be ignored.
Finally actually writing all of these files to a GZipped tarball.

Git was invoked using the subprocess module as in the following code example:

import re
import subprocess

def get_files_from_git():
    git = subprocess.Popen(['git', 'ls-files'],
                            stdout=subprocess.PIPE,
                            stderr=subprocess.PIPE)

    if git.wait() == 0:
        files = git.stdout.read()
        files = re.split(r'\r?\n', files.strip())
    else:
        files = []
        print "Error: %s" % git.stderr.read()

    git.stdout.close()
    git.stderr.close()

    return files

In general I would advise against using the Popen.wait function if you’re trying to read from a pipe because the whole thing can deadlock if the pipe’s buffer is too small, but in this case I knew the output was small enough for this not to be an issue. Specific file names can then be removed from files as required in preparation to build the archive:

import tarfile

def make_archive(filename, files):
    with tarfile.open(filename, 'w:gz') as tar:
        for fn in files:
            tar.add(fn)

So the end result is a script that can build my source archives for me without being told about any new changes to the repository! After modifying the code to use Popen.poll it’s a grand total of 50 functional lines! Yep, still an enormous fan of Python.

WordPress hacking

The final hurdle (we’re nearly there now folks) was setting up some web pages to actually host the binaries and provide information describing what CopyExt actually is. I use WordPress to run my website and I wanted to have a specific sub-menu for each page in the CopyExt section containing only relevant links, but it turns out this isn’t as straight-forward as I thought it would be. I searched for some plugins that would allow me to do it but every one I tried only added menus to the right of the page, not as a secondary menu which is what I wanted.

Then I stumbled across custom fields which are basically ways of including meta-data in your posts and pages in a way that themes can access (you may need to enable it in the Screen Options menu at the top of the various editing pages if you can’t see them). Themes can look up these fields using the get_post_meta function which will either return an array of possible values, or a single string value. I modified the header of my theme to include the following:

<?php
if (has_nav_menu('secondary', 'catchbox') ||
    get_post_meta(get_the_ID(), 'subnavname', true)) {
?>

<nav id="access-secondary" role="navigation">

<?php
if (get_post_meta(get_the_ID(), 'subnavname', true)) {
    wp_nav_menu(array(
        'theme_location' => 'secondary',
        'menu'           => get_post_meta(get_the_ID(), 'subnavname', true)));
} else {
    wp_nav_menu(array(
        'theme_location' => 'secondary'));
}
?>

</nav>

<?php } ?>

I can now specify any menu I want to appear on a page in a custom field called subnavname! If that field isn’t present then either the secondary menu won’t appear or the site-wide one will. So this custom field actually can override a site-wide setting if I want it to! Then I made a menu in the normal WordPress menu editor, updated my CopyExt pages to include the custom field and bingo: the menu appeared. It’s not the most elegant method in the world but it is very simple!

Closing

So after all that, CopyExt is finished until someone tells me it’s broken… or I break it myself! As you can see from this post (congratulations if you read the whole thing, it’s a lot longer than I planned) I actually learned quite a lot of stuff from picking up this project again. And despite it eating into what should’ve been a break from work, I actually really enjoyed the whole process. It’s nice to finish projects.

Kier J. Dugan

CopyExt 2 Project Post-mortem

Nullsoft Scriptable Install System

Places to get information

Some Gotcha’s

Embarrassing “bug”

Visual Studio tips

Source code management

Aside: Source code packaging

WordPress hacking

Closing

Nullsoft Scriptable Install System

Places to get information

Some Gotcha’s

Programming related stuff

Embarrassing “bug”

Visual Studio tips

Source code management

Aside: Source code packaging

WordPress hacking

Closing