I looked briefly at frameworks specifically for testing databases, but didn’t see a ton of value beyond just invoking the database from JUnit via JDBC, so that’s what I did. Like the use of dbdeploy, stuff gets started from Ant, with a test target.

<target name="test" depends="create-test-database"
        description="Run tests against database procedures">
        <path id="test.lib.path">
          <fileset dir="lib">
            <include name="**/*.jar"/>
          </fileset>
          <fileset dir="test/lib">
            <include name="**/*.jar"/>
          </fileset>
        </path>
        <!-- Compile tests -->
        <mkdir dir="${target.dir}/classes"/>
        <javac srcdir="test/src" destdir="${target.dir}/classes" includeantruntime="false">
            <classpath>
                <path refid="test.lib.path"/>
            </classpath>
        </javac>
        <!-- Run them -->
        <junit printsummary="on" haltonfailure="on">
            <sysproperty key="dbhost" value="${db.host}"/>
            <sysproperty key="dbuser" value="${db.user}"/>
            <sysproperty key="dbpass" value="${db.password}"/>
            <sysproperty key="dbname" value="${test.db.name}"/>
            <classpath>
                <path refid="test.lib.path"/>
                <pathelement location="${target.dir}/classes"/>
            </classpath>
            <batchtest>
               <fileset dir="test/src">
                    <include name="**/*Test*" />
               </fileset>
            </batchtest>
        </junit>
    </target>

Since it is trivial to create a fresh database, I create a separate one just for testing in the dependent create-test-database target. This lets passing tests be a precondition to updating the real database. In other words, on the build server, the test target has to succeed before the update-database target will be run that will apply the current set of database deltas to the real database.

The sysproperty Ant task is handy here because it lets you pass information to the JUnit tests. This means you can set all the database connection information once in the Ant script, but have access to it in the JUnit tests, keeping things DRY.

In JUnit, the properties set with the sysproperty task are available via System.getProperty, as you can see from this setup function:

@BeforeClass    
public static void classSetUp() throws Exception {
    // Get database connection info from system properties set by Ant.
    String host = System.getProperty("dbhost");
    String database = System.getProperty("dbname");
    String user = System.getProperty("dbuser");
    String password = System.getProperty("dbpass");

    Class.forName("com.microsoft.sqlserver.jdbc.SQLServerDriver");
    String connectionUrl = String.format("jdbc:sqlserver://%s;database=%s;user=%s;password=%s",
                host, database, user, password);
    conn = DriverManager.getConnection(connectionUrl);

    String sql = "DECLARE @RC int; EXEC @RC = [dbo].[UpdateCamera2] ?,?,?,?,?,?,?,?,?,?,?,?,?,?,?; SELECT 'Return Value' = @RC";
    spStatement = conn.prepareStatement(sql);
}

This gets everything set up to call the stored procedure (UpdateCamera2 in this case) as well as save the return value, with cleanup in a tear down method:

@AfterClass
public static void classTearDown() throws Exception {
    spStatement.close();
    conn.close();
}

There’s a helper method to call the stored procedure too. This stored procedure is pretty unwieldy because it was specifically introduced to reduce trips to the database for a performance critical part of the system, and it needs a lot of parameters to do its big pile o’ work.

/**
 * Call the stored procedure with the unwieldy list of parameters. Sorry.
 * @return A bit field whose value depends on whether the site and camera were new or already existing.
 * @throws SQLException
 */
private int execUpdateCamera2(String cameraMac,
                              String cameraName,
                              int cameraProductId,
                              String cameraFirmware,
                              String cameraInternalIPAddress,
                              String cameraIPAddress,
                              String cameraIPCountry,
                              boolean cameraAlertsEnabled,
                              String cameraJidResource,
                              String siteId,
                              String siteName,
                              int siteAlertFilter,
                              int siteAlertFrequency) throws SQLException {
    spStatement.setString(1, "testuser@dvsopstest.com");
    spStatement.setString(2, "unit test");
    spStatement.setString(3, cameraMac);
    spStatement.setString(4, cameraName);
    spStatement.setInt(5, cameraProductId);
    spStatement.setString(6, cameraFirmware);
    spStatement.setString(7, cameraInternalIPAddress);
    spStatement.setString(8, cameraIPAddress);
    spStatement.setString(9, cameraIPCountry);
    spStatement.setBoolean(10, cameraAlertsEnabled);
    spStatement.setString(11, cameraJidResource);
    spStatement.setString(12, siteId);
    spStatement.setString(13, siteName);
    spStatement.setInt(14, siteAlertFilter);
    spStatement.setInt(15, siteAlertFrequency);

    spStatement.execute();

    ResultSet results = spStatement.getResultSet();
    results.next();
    int returnValue = results.getInt(1);
    results.close();

    return returnValue;
}

With all that set up, an individual unit test (that references a couple of other helper methods) looks like this:

@Test
/**
 * A new site is created if the provided site info doesn't match an existing site.
 */
public void newSiteCreated() throws Exception {
    String siteId = UUID.randomUUID().toString();
    String siteName = "Test Site #1";
    int siteAlertFilter = 111;
    int siteAlertFrequency = 1111;

    int result = execUpdateCamera2("11-11-11-11-11", "Unit Test Camera", 16, "1.0", "", "", "-", true, "TestJID",
            siteId, siteName, siteAlertFilter, siteAlertFrequency);

    assertFalse(siteExisted(result));
    HashMap<String, Object> site = queryForSite(siteId);
    assertEquals(siteName, site.get("Name"));
    assertEquals(siteAlertFilter, site.get("AlertFilter"));
    assertEquals(siteAlertFrequency, site.get("AlertFrequency"));
}

I recently had to update this stored procedure, and let me tell you, I didn’t get it right the first time. Or even the third or fourth time. So it was fantastic to have a procedure that took just a few seconds to spin up a clean test database with my latest attempt and run though all the tests. When I finally worked through my SQL mistakes such that the tests passed, I was confident to check in my new delta script.

Previously we had a single development database shared by all. This is an inflexible working arrangement because messing up, even in the very short-term, can impact other people. Also, throwing the whole thing out and starting over isn’t an option. So change number one was private databases for database development.  Not doing this is akin to checking your source code into the main repository in order to see if it works.

To facilitate this, we’re using dbdeploy, which isn’t perfect, but good enough. You start with a baseline schema (just a SQL script that creates all the needed tables and other objects), then add deltas (also SQL scripts) when changes are needed. The simple job of dbdeploy is, given a bunch of deltas, to make sure they’re applied once in the appropriate order. It keeps track of the applied deltas with a table in the database being maintained. You can also try to have a roll-back script for any change, but that tends to get dicey.

In addition to the baseline script and the deltas, I also have a “development” script that does things to make life easier for developers that aren’t necessarily appropriate for a non-private database, like pre-populating the User table with some users for testing.

Everything is controlled by an Ant script, using both the built-in “sql” task as well as the task provided as part of dbdeploy. Here’s what the “create-database” target looks like:

<target name="create-database"        
  description="Create a database from scratch, including the baseline, changelog and deltas">
    <antcall target="echo-settings"/>
    <sql 
        driver="${db.driver}" 
        url="${db.create.url}"
        userid="${db.user}" 
        password="${db.password}" 
        classpathref="dbdriver.classpath"
        encoding="UTF-8" 
        delimiter="GO"
        >
        <!-- Create the database from scratch with the baseline schema. -->
        <fileset file="${db.baseline.dir}/baseline.sql"/>
        
        <!-- Additional setup for development purposes. -->
        <fileset file="${db.baseline.dir}/baseline-development.sql"/>
    </sql>
    <antcall target="create-changelog" />
    <antcall target="update-database" />
</target>

The “create-changelog” target adds the dbdeploy tracking table to the database using a script included with dbdeploy:

<target name="create-changelog"
    description="Create the changelog table in an existing database">
    <!-- Note that we have to specify the database on the url for this script or the database 
    will be whatever is default for ${db.user}, which might be "master", for example. --> 
    <antcall target="echo-settings"/>
    <sql 
        driver="${db.driver}" 
        url="${db.url}"
        userid="${db.user}" 
        password="${db.password}" 
        classpathref="dbdriver.classpath" 
        delimiter="GO"
        >
        <fileset file="dbdeploy/scripts/createSchemaVersionTable.mssql.sql"/>
    </sql>
</target>

The “update-database” target called from “create-database” uses the custom dbdeploy task to apply any deltas:

<target name="update-database"
    description="Apply all available, but not yet applied deltas to a database">
    <antcall target="echo-settings"/>
    <dbdeploy
        driver="${db.driver}"
        url="${db.url}"
        userid="${db.user}"
        password="${db.password}"
        dir="${db.deltas.dir}"
        delimiter="GO"
        />    
</target>

The result is the ability to create a complete database from scratch in just a couple of seconds. It is very liberating because I can mess up in various and sundry ways while trying to get a stored procedure just right, and the consequences are minimal.

Once I’m satisfied that a delta script is good, I check it in and TeamCity (our build server) automatically applies the deltas to the development database using the “update-database” Ant target.

An important part of this is publishing all of the deltas in existence at that moment in the artifacts for the build. That means that our operations guys can take any build, run dbdeploy against the attached deltas, and know that the database will be in the correct state for the application generated by that build. This is true even if many intermediate builds (potentially containing database changes) are never installed in staging or production.

Our workflow for database schema changes is then:

1. Create a .sql file in the “deltas” directory that makes the needed change. You can use SQL Management Studio to figure out the script, but don’t actually make the changes there. Create a script instead.
2. The .sql file should have a name following the pattern:
   <sequence number><description>.sql
   For example:
   001_create_firmware_table.sql
3. Optionally back up your database.
4. Run “ant update-database” which will apply your script to your local database.
5. Test your code that depends on the new database structure.
6. If your script wasn’t right, restore from backup or drop your whole local database and rebuild it.
7. Once you’re sure everything works, check in your .sql script and code changes at the same time.
8. The continuous build server will update the dev database and test your code.
9. If you discover something that needs changing after this, you need to create a new delta .sql script.

Of course, manual schema changes in any environment are against the rules.

I didn’t have any idea what this was saying. My wife suggested it meant that I ought to watch out for diamonds… and maybe nets.

I found this sticker in the other shoe. I think it means the same thing:

Hmm… scary (but potentially useful) hexidecimal-laden error messages or cute emoticon?

Thanks for that.

This task has been lingering undone on my to-do list for months. I imagined that I’d figure out some way to use Dropbox, Google Drive, Sky Drive, or something to keep my data safe in case something catastrophic happened at the house. All those solutions seem so expensive, though. When Amazon Web Services announced Amazon Glacier not long ago, with storage rates at 1 cent per gigabyte per month, I figured I’d found my storage location, but it is just an API. Finding CloudBerry Backup completed the solution.

CloudBerry is a client for a plethora of online storage locations, including S3, Azure, Google Storage, and about a dozen providers I’d never heard of in addition to Glacier. You’re responsible for establishing an account and paying the bill for the storage, but the CloudBerry client takes care of getting your stuff there and back.

While Glacier support was enough to catch my interest, I was also happy to see that I could use it to back up to my NAS. I was dismayed when switching from Vista Ultimate to Windows 7 Home Premium that I’d lost the ability to use Windows Backup to backup to a network location, so I’ve just been using an external USB drive. It was pretty easy to set up the network backup using CloudBerry, with plenty of good options to control what got backed up.

Aside from the missing network support in Windows Backup (for the Home variety of Windows), there a couple of things that bug me about it:

1. It’s not always clear what your’re backing up. You can pick certain paths, but does it include hidden and temporary files? Is there anything that I actually want that is being excluded? What if I don’t want to include some fat file in the backup?
2. Compounding the first problem is that the backed-up data itself is somewhat opaque without the backup client.

CloudBerry gives you a lot of fine-grain control over what gets backed up (I think I’ll survive if my Linux VM isn’t backed up, thanks), and the backed-up data is perfectly navigable when I ssh into my NAS. You can compress and/or encrypt the data as well if you want.

I used the NAS backup to fine-tune my settings. The first run took all night for about 122 GB of data. Presumably backups will be incremental going forward, so much speedier. That will be especially important for Glacier. Having gotten some settings I liked for the NAS, I created a configuration for Glacier, and kicked that off. This takes a long time, as in many days, owing to the generally slow upload speeds from ISPs. CloudBerry provides some bandwidth throttling that can come in handy during the days-long upload.

For the Glacier configuration, I had to manually copy my settings from the NAS configuration. It would have been nice to copy a configuration, but I didn’t see a way to do that. Also, when you edit a configuration, you have to go through the entire wizard. All the values remain as they were, but you have to fully navigate through the dozen wizard screens before you can get to the end and commit the change.

I also ran into a bug when trying to enable the storage cost estimation tool. I got an index out of range exception displayed in a message box, without any further indication of what was wrong or how to fix it. I’ve also seen a couple of NullReferenceExceptions reported from the application, so there are a few rough edges.

Overall, though, I’m happy to call it good.

 Offsite backup for critical files

The adage says, “code is read more than it is written.” If this isn’t completely obvious, try this experiment:

1. Turn off your monitor
2. Write some code

Orthography (literally “correct writing”) are the rules of writing a language, with special emphasis on standardized spelling. There was an argument and rebuttal a little while back in Wired Magazine for doing away with standardized spelling, which was also covered on NPR.

In the NPR interview, Anne Trubek says:

So my basic argument is that if you look at the history of the English language, a lot of people think there are sort of immutable laws that are, you know, God-given or laws of nature. But actually, it’s a bunch of manmade prescriptions and guidelines that change over time.

In other words, Trubek is saying that correctness is somewhat arbitrary, but as something becomes standardized, people start to treat right and wrong in an almost moral sense that isn’t justifiable.

Lee Simmons, a copy editor for Wired, who has the job of making sure words are spelled correctly, responds with:

I would say spelling rules, for what they are, they’re all about making communication easier. If I could use an analogy, the Internet itself is essentially a set of standards – hardware and software standards – that make it possible for people with different devices to communicate. It creates a universal platform. And I would argue that our English spelling system, for all its flaws, provides just such a universal platform…. We can argue about whether we ought to reform the standards, make the system more logical, but I would argue that the standards themselves are something that we need to preserve.

Simply put, it’s easier to read words that are spelled how we expect.

English also developed standard punctuation and good style, again with the objective of writing to be understood, but we sometimes forget things like that when writing code. We laugh at “The Department of Redundancy Department”, but many programmers don’t see anything wrong with code like:

// find the customer with the id
var customer = Customers.Find(customerId);

or

return (a == b) ? true : false;

Does it matter where braces and parentheses go? Certainly not in any moral sense, but there are standards that are more logical than others. If you’re reading an English sentence with parentheses (like this one) and suddenly things get a little weird( like this )then it interrupts the communication. Or what (if I) just start) throwing unnecessary( parentheses in?

return (theResult);

Communication is actually kind of hard. Consider the fact that professional writers, who presumably have some talent for communication, almost never publish the first thing that shows up in the word processor. They revise, rework, rewrite until their ideas are expressed well. Then someone else comes along to fix the places where the writer still didn’t get it quite right.

Do people even notice if you make a similar effort to make your code easy to read? Or do they just say, “Oh, that’s easy,” compared to looking at some complicated mess and say, “Wow, that guy’s smart”?

Coding standards are a tool, one of many available to programmers to facilitate communication. Why not take advantage of it? Your future self will thank you.

Since most of the people on our team work remotely, a group chat became the primary channel for the problem. There were operations people pointing out potentially significant log entries, QA and other interested people reporting every instance of the problem happening — or that they could not reproduce the problem all, ideas and theories about what could be going wrong, and my boss asking what kind of help I needed.

From this, and other similar experiences, I’ve learned a couple of things to do in situations like this:

1. Get a bug report

In times of panic, it can seem like a bureaucratic waste of time to file a bug report. But it is counter-productive to go chasing after a problem without getting the best understanding available of what the problem actually is. This is especially true in cases like this one where some people were seeing the problem frequently and others not at all.

2. Go dark

Of course you can’t just drop out of sight when people are counting on you to solve something quickly, but let them know that you just need some time to think, and barring any breakthrough discoveries, that you don’t want to be interrupted for a bit.

With all of the chat messages coming through, I spent five hours chasing after this and that without making much progress. Finally I came to my senses and just ignored everything and took time to think. After some time deeply analyzing one instance of the problem, it occurred to me that I had written a program a couple of years ago to test this exact scenario. Once I dusted that off and ran it a couple of times, the problem became painfully obvious.

Honestly, I felt kind of dumb that it took me so long to figure it out, but it reinforces how debilitating distraction can be.

• setup
• install
• dist
• distribute
• tools
• utils
• easy
• py

The packaging situation is kind of confusing with all the alternate tools, libraries and methods, so I’m going to take a stab at clarifying things — for how it seems right now, anyway.

 What is a package?

Let’s step back for a second to review some Python vocabulary. A single .py file can be a script or a module — the basic difference being that a script is primarily meant to be run, while a module is primarily meant to export useful stuff for scripts or other modules. Of course a module can be executed too, typically with the if __name__ == '__main__' pattern. A package is a group of modules, providing some additional structure. In short, it’s a directory with an __init__.py and some other .py files in it.

 I don’t think that’s what I mean

This is where the vocabulary gets muddled. If you’re familiar with PyPI (pie-pee-eye), the Python Package Index, you’ll realize there’s something more than directories with __init__.py files. PyPI has what might better be termed distributable packages. Sure, they’re packages, but include metadata, documentation, and other stuff to make it easier to manage third-party Python software on your machine.

Getting started

The distribute page cuts through much of the disutils, setuptools, easy_install, soup with this pithy graphic:

OK, since the New Hotness sounds good, we just want distribute and pip. The graphic even shows you how to get them if you’re on *nix. Observant readers might notice that pip is being installed with easy_install and say, “Hey, wait a minute… easy_install is on the Old and Busted list, but we’re supposed to use that first?”

It turns out that pip (a recursive acronym for “Pip Installs Python”) currently requires distribute, and distribute happens to include easy_install. With Python 3.3 (which, by chance, has gone to release candidate status today), the features of the distribute package become distutils2 in the Python standard library, and distribute is headed for the bit bucket. Sorry, this was supposed to be clarifying things, but it’s getting back into the realm of chaos. So for now, we’ll press forward with distribute as the Not-Totally-New Hotness, but workable strategy for today.

If you’re on Windows, you can just download distribute_setup.py from here:

http://python-distribute.org/distribute_setup.py

Then, as indicated above, you can run this from a command prompt:

python distribute_setup.py

The install creates a Scripts subdirectory in your Python install directory, and puts easy_install.exe in there. It would be a good idea to add the Scripts subdirectory to your Windows PATH. The New Hotness pip.exe goes in there too, once that’s installed. If you go the easy_install route, you’ll get a UAC prompt on:

easy_install pip

Then pip will be installed. Alternatively, (and without the need for UAC elevation) you can download the pip package directly from:

http://pypi.python.org/pypi/pip/

The download is way down at the bottom of the page as a .tar.gz file, which isn’t the most convenient format for Windows, but 7Zip can remedy that difficulty. Once the archive is unpacked, you can install pip with this command from within in the directory where the pip archive was unpacked:

python setup.py install

Once you’ve done this, you may delete the contents of the unpacked archive — everything you need has been copied into your Python install.

Now you can use pip to search for packages on PyPI and install them. For example:

pip search tornado

This will find packages with “tornado” in their name or description, which includes not only the package specifically named “tornado”, but related packages as well. Installing a package is, not surprisingly:

pip install tornado

This downloads the Tornado web server and copies it into the site-packages subdirectory of your Python installation (C:\Python27\Lib\site-packages for example). Since site-packages is in the Python sys.path, you can now import Tornado types from a Python script located anywhere on your system. Uninstalling a package is equally obvious:

pip uninstall tornado

To see what packages you have installed, run:

pip freeze

 Advanced pip

“freeze” seems an odd verb for listing installed packages. This is because there is more to it: You can “freeze” a set of packages with version numbers that are all known to work peacefully together, creating a requirements file that can be used to recreate that same environment somewhere else. More details about this are in the pip documentation.

Let’s say you’re developing a package and want to make that package available to scripts on your system. I guess you could check your source out right into site-packages such that it can be edited in-place, but that seems awkward and unpleasant. Instead, you can install your package with pip using “edit mode”:

pip install -e C:\Projects\Alert\dvsdriver

The path specified here is where my Python project resides from my source control checkout. With the -e option, instead of copying the package to site-packages, a path configuration file (a file with a .pth extension) is created in site-packages instead. The .pth file adds the C:\Projects\Alert\dvsdriver directory to the Python path, allowing you to develop your package in its natural location while still making it available for importing from other scripts.

You can also use pip to install packages from places other than PyPI, including local files, URLs of alternate package indexes, and directly from version control repositories.

Virtual environments

If you get into the situation where you need one version of a package for one script, and different version of the same package for another script, you’ll probably want to look into virtualenv. This is a package that lets you create multiple separate Python environments that are completely isolated, so they can have different sets of packages installed into them. The environments set up with virtualenv already come with distribute and pip in them, so they’re ready to be populated with other packages.

 Creating Distributable Packages

Now that we’ve got a sense of what distributable packages are and how to use them, how do you create one?

The main thing is to create a setup.py module, which looks something like this:

from distutils.core import setup

setup(name='MyProject',
      version='1.0',
      author='Eric Smith',
      packages=['myproject'],
      )

Notice that we’re importing from distutils, which means Old Stuff, but it gives us a working package that can be managed with pip:

pip install -e \Users\Eric\Projects\MyProject

There’s good information about project structure, useful files (like README.txt), additional metadata to include in the setup call, and general package building in The Hitchhiker’s Guide to Packaging.

Looking forward

As mentioned previously, distutils2 is nearly upon us. With that, the setup.py script will be replaced by a declarative setup.cfg and all previous libraries (distutils, distribute, setuptools) will be deprecated. Pip will stick around, though.

So there’s my attempt to clarify Python packaging. If I’ve made egregious errors, please let me know and I’ll fix them.

I want a reminder at the specified reminder time, and another one at the appointment start time. This is how it typically goes for me:

PHONE: Ding!
ME: Oh yeah, I’ve got that meeting in 10 minutes. Let me just finish this little bit of code in the meantime…

(15 minutes later)

CO-WORKER VIA IM: Hey, you coming to this meeting?
ME: Shoot! Sorry! Be right there.

The lack of this feature has created a market for tons of reminder “nag” apps that try to make up for it. I don’t really want a whole other app that piles on inane features in an attempt to justify its existence, though.

2. Current month is white

OK, this is kind of picky, but shouldn’t the current month be white while the previous and next months are gray? The current scheme is opposite that and my brain struggled with it.

3. Combined day and agenda view

The webOS calendar has a brilliant combination of an agenda and day view. It lets you see all of your appointments without huge swaths of blank screen pushing them out of view, but still makes it easy to add a new appointment at any time of day by tapping the accordion section.

This was on the Discount Tire mobile site. It’s awesome that you can schedule a service appointment on the web using your mobile phone. It’s not so awesome that the form validation doesn’t compensate for the trailing space that my phone’s keyboard automatically added. “Normal” people would give up when the site refused to accept such an obviously correct entry.