So it turns out the complete project I talked about in my previous post wasn’t so complete: the datastore didn’t work. Getting it to work (with maven) was one of the most frustrating things I’ve done in a long time. No small part of the frustration was that I’ve spent so long getting the infrastructure set up for this project without having gotten to any of the “interesting” bits. Luckily, I think with the datastore working most of the infrastructure work should be mostly done.

That said, there are still some things I haven’t done. I haven’t yet retrieved the data from the datastore (though I uploaded the app (via maven) and verified that the data got set properly). I also haven’t figured out how to do datastore migrations. The first problem I don’t expect to be too hard (storing should be harder than retrieving, knock on wood) and I find the second problem interesting (unlike wrestling with poorly documented third-party tools for almost a week).

I’ve updated the link to my zip to include my new discoveries. There are many things about my solution that I don’t like, but I’m at the point where I really just want to start working on the fun stuff. The main problem is that I had to implement the datanucleus enhancement via an ant task rather than the datanucleus maven plugin. I have a separate pom.xml in the buildResources folder that installs the (very simple) ant plugin, though I haven’t figured out how to get that to automatically install alongside the main pom.xml’s build, nor automatically get installed when I change it. I’m sure there’s a way (and if anyone knows, I’d love to know too), but I’d rather put my energy toward the game I want to make.

Here’s the link to the zip: http://euclid.poundcs.org/~kevin/cleanGaeWithLift.zip There’s a readme there explaining what to do to get it to work (it should be simple). I’ve sprinkled the pom.xml file with comments describing why I’ve done certain things the way I did. Any comments on it would be greatly appreciated. The project includes a single, simple datastore object (User.java - yes, I made the datastore objects in Java since I had some trouble injecting Scala classes. I’ve read it’s doable, but I wanted to tackle one thing at a time) which can be used as an example.

Time for bed. I can’t wait to actually get started on the fun parts of this project :)

http://euclid.poundcs.org/~kevin/cleanGaeWithLift.zip

A zip containing a clean GAE + Lift + Maven project. To use it all you need to do is change two files (describe in README.txt). The three jars maven will want can be found in buildResources/gaeJars (there are more jars than that since during the build step they’re copied to the war/WEB-INF/lib folder, along with any other dependencies). This project is Eclipse friendly. More details can be found on my wiki.

I managed to get Hudson to automatically upload my GAE project on check-in! It was a real bear to get set up, but I’m glad it’s working. I’ve updated my wiki with the new, working pom. You can find that here. You’ll need to change a few things, but most notably the gae.account and gae.passwordFile properties set at the top of the file. It should work on Windows and OS X. Linux too, though I haven’t tested it.

Here are some of the pitfalls I ran into while setting this up, and how I got around them:

• The GAE uploader doesn’t let you pass in your password via an argument. Presumably for security (while this is probably a good thing) it was very frustrating to work around and was the root cause of most my problems. Luckily they let you pass the password in via STDIN, if you can convince ant to do so.
• If you enter the password manually, Google sends back a cookie that helpfully authenticates you for 24 hours, saving you from having to enter your password anymore. Unfortunately this feature made it near impossible for me to test my mvn script locally since it would always authenticate properly. I had to go on a time-consuming hunt for the cookie (which, by the way, on Windows is in the registry: HKEY_CURRENT_USER -> Software -> JavaSoft -> Prefs -> com -> google -> appengine (via this page)) to delete it. Once that was finished I could continue.
• To run the upload task, originally I had used the <java> ant task to run the GAE uploader. This worked, and there’s even an “input” (and “inputstring”) parameter to specify a file that should populate STDIN for the task, but for some reason ant refused to actually pass the data. I don’t know if this was ant’s fault or java’s, but whatever the reason the password wouldn’t get entered.
• I switched to the <exec> task to calling GAE’s given script files. This also has an “input” parameter and, wonderfully, it works as advertised. The password was entered and the project was successfully deployed to GAE on my local (Windows) machine!
• Trying to get this to work on Hudson was a bit tricky. Linux and OS X need to exec a different file. Luckily some digging revealed the <condition> task that sets a property based off a condition (like OS), letting me set the file to execute based off the OS.
• Checking in this pom.xml file almost worked, except Hudson got a permission error. Apparently the GAE upload script doesn’t have the execute permission bit set, so I had to call sh with the file as a parameter (alternatively I could have written a task to chmod the file). Because this changed the number of parameters on Windows vs. OS X/Linux, I no longer could use individual <arg> tags on my <exec> task and I switched to a single <arg line=”” /> tag instead to get it to work.
• This didn’t quite work either though, as my Hudson project’s name had a space in it (“Kevin Game”) which made the GAE uploader break apart the path to the WAR folder. Putting quotes around the path didn’t make a difference and I didn’t know how to do a replace on a property in ant, so I simply renamed my Hudson project to KevinGame and everything worked!

Oh, and I also updated GAE from 1.2.2 to 1.2.5. Apparently newer versions have been out for a while, I’m not sure how I missed them. I’ve updated my wiki with the new instructions. However, anyone who used my old pom.xml file will have to manually add the new jars to maven when it prompts.

I also set up my SPF headers for kevinmook.com to prevent spammers from spoofing my email address. I found a fantastic SPF validater here: http://www.kitterman.com/spf/validate.html. Not only does it check your current SPF header, it lets you type in one to test (as well as enter IP addresses to test against). It definitely saved me a bunch of time, especially given the minimum 1/2 hour TTL on DNS headers GoDaddy provides.

Time to go relax after a long day! Hopefully this post will help someone else get their project up and running too!

So it took a few days longer than expected but I finished setting up most my infrastructure (again) on Snow Leopard. The biggest hangup (apart from spending all my free time on Wednesday/Thursday/Friday doing DoInk things and was out of town for the weekend) was that Snow Leopard broke macport’s TomCat install so I had to manually install it. However, I managed a lot in the past two days in addition to getting everything working again. I got my wiki put up on a public server (http://wiki.kevinmook.com) with hourly backups to a timemachined copy running on my mini (so if something happens to the public server I can still access all the information). I still haven’t had time to figure out how to securely give Hudson my password to upload to GAE; hopefully that won’t be too hard.

Anyway, I need to get ready for bed soon. Before I leave you, here are some interesting links on my wiki:

• http://wiki.kevinmook.com/InstallingAndConfiguring - A list of the things I’ve installed/configured so far for this project. I’ve tried to break things apart into individual parts where possible so they’ll be mix & matchable, but a few of them I configured together.
• http://wiki.kevinmook.com/UsefulTools - Some useful tools I’ve used along the way. Right now it’s pretty empty (just rsync and launchd), but they’re powerful tools and as I use them more I’ll likely add more content.
• http://wiki.kevinmook.com/ConfiguringGoogleAppEngineWithMavenScalaLiftAndEclipse - Yes, I know the name is long and specific, but at least it’s descriptive, right? I mention this separate from the InstallingAndConfiguring page since I think it may be particularly interesting. Included on that page is my current pom.xml. Oh, and this is important for anyone who wants to use the Lift framework on GAE: make sure you use Lift 1.1 (I used 1.1-M4)! Version 1.0-SNAPSHOT doesn’t work with the Google App Engine!

That’s it for now, goodnight everyone!

I managed to get a lot done in the past few days. I took Thursday/Friday (mostly) off for some much needed R&R, but managed to get Scala/Lift working on the Google App Engine, all built from maven! I also got Hudson set up and working (which ended up being a pain due to maven having a fixed .m2 repo location relative to the user’s $HOME). Uploading to GAE mostly works, though I need to find a way to give it my password other than through stdin.

All this was documented on my wiki, which is currently out of commision due to me upgrading my server to Snow Leopard (as is my Git repo, Hudson, etc.). I figured I’d upgrade now, while all this stuff is fresh in my mind and before I get even further down the rabbit hole. Once I get my server back up and running, I’ll move my wiki to a public server and directly link to it. This way as I find new tricks/update my steps, the information will stay centralized rather than scattered around many blog posts.

Anyway, I’m going to get back to setting up the server (macports is busily at work). Hopefully Thursday (I work at DoInk on Wednesdays) I’ll have some wiki links to point at with interesting tidbits.

I’ve been strugling to get Maven to make an Eclipse GAE project. I’m getting a lot closer, but several problems remain. Here are my goals:

1. Have “mvn install” successfuly run (and pass) unit tests, then upload the bits to GAE.
2. Have “mvn eclipse:eclipse” make a usable Eclipse project that not only makes the dependency jars accessable to Eclipse, but puts them in the WEB-INF/lib directory so GAE can use them.

So far I’ve gotten “mvn install” to successfully run, though I haven’t tried unit tests yet and I don’t have it uploading to GAE. I also managed to get eclipse:eclipse copying dependencies to WEB-INF/lib (excluding the GAE jars that already sit there), though I still need to have it get rid of old, unused dependencies if they’re removed from the pom.

To prevent duplicate references, I configured the pom to exclude the GAE jars from the Eclipse project when running eclipse:eclipse. Instead, I just have them copied to the WEB-INF/lib directory and Eclipse accesses them the same way GAE would (other than JUnit). Unfortunately, I still have to add any new dependencies to both the dependencies list and the exclude list. Hopefully I can find a way to whitelist dependencies for the maven-eclipse-plugin configuration, rather than blacklisting. Another option would be to avoid using eclipse:eclipse and instead run my custom build command to copy the dependencies, but that would make other updates awkward (junit), and if I accidentally did eclipse:eclipse I’d have a mess.

Here’s a link to my current pom.xml. My pom.xml file is templated from an example helloworld package someone posted at the bottom of this page as a comment (I had to modify it a bit to use a newer version of GAE). If you look, you’ll see there’s a section on uploading to GAE during the “install” phase. I haven’t tested it yet, but I’m hopeful it’ll work.

Anyway, it’s almost 4am. I’ll continue this tomorrow. Hopefully soon everything will be working and I’ll get to actually start playing around with Scala and Lift!

I’m switching my wiki to MoinMoin. MediaWiki works fine, but after using the markup a bit I realize I prefer MoinMoin.

To install MoinMoin, I followed the instructions here, which document how to install MoinMoin on an Apache server.

Here are the things I had to do differently:

I decided to install to /usr/local, so I had to replace /moinmoin with that.

My apache httpd.conf file was in a different location, having installed it with ports.

When setting up the Directory block in apache’s httpd.conf file, I had to give it an absolute path rather than the aliased one. I also had to explicitly set up a Directory block for access to the moin/htdocs folder.

Later, I’ll follow the setup instructions to finish installing the wiki.

Edit: Apparently it’s not possible to only allow admins to create accounts on MoinMoin. To disable user creation globally, I had to add this to my wiki’s config (per this suggestion):

actions_excluded = DefaultConfig.actions_excluded + ['newaccount']

While this blog’s useful for documenting pitfalls and successes, a wiki can be updated a lot easier than this blog and can keep information much more organized. So, logically, I’ve decided to set up a wiki! My plans for this blog haven’t changed, instead I’ll likely link directly to the wiki (if I decide to make it open to the world).

I’m going with MediaWiki. Mostly because I’m somewhat familiar with it, having set it up in the past. I’ll be hosting it on my Mac Mini out of my apartment (at least for now) to keep expenses low, since I don’t expect a large amount of traffic. I haven’t decided if I’ll make it open to the world. I probably will, but that involves spending time hardening it.

To set up MediaWiki, I had to install php5 and mysql. I already had ports and Apache2 installed, so I used ports:

sudo port install php5 +mysql

I also had to modify httpd.conf, adding index.php as a DirectoryIndex:

DirectoryIndex index.php index.html

Telling it to toss .php files to the php processor:

<FilesMatch .php$>
SetHandler application/x-httpd-php
</FilesMatch>

And add the php5 module to the module list:

LoadModule php5_module        modules/libphp5.so

Next, I set up MySQL:

First I initialized the MySQL databases:

sudo /opt/local/lib/mysql5/bin/mysql_install_db --user=mysql

Next, I told mysql to start at boot by changing the RunAtLoad variable to true in the mysql5.plist:

sudo vi /Library/LaunchDaemons/org.macports.mysql5.plist

And added the plist to launchctl:

sudo launchctl load -w /Library/LaunchDaemons/org.macports.mysql5.plist

And rebooted OSX to start up the mysql server (since it wasn’t installed before). I’m sure this can be done without rebooting, but I also wanted to make sure it came back up with the machine. I also may have been able to do this without a few of those steps, but I had to do a bit of trial-and-error and google searches to get it to work

Next, I downloaded MediaWiki into /opt/local/apache2/htdocs/wiki, navigated to http://server/wiki, and ran the configuration script. The wiki was then up and running.

I then did some basic hardening with the config files. Nothing too fancy (since I won’t put anything sensitive on the wiki), but I made it so only admins can create users, and only users can create/edit pages. This is my personal wiki, so I don’t want unknown others messing around. :) To do this, I added the following to LocalSettings.php in my wiki folder:

$wgGroupPermissions[‘*’][‘createaccount’]    = false;
$wgGroupPermissions[‘*’][‘edit’]             = false;
$wgGroupPermissions[‘*’][‘createpage’]       = false;
$wgGroupPermissions[‘*’][‘createtalk’]       = false;
$wgGroupPermissions[‘*’][‘writeapi’]         = false;
$wgGroupPermissions[‘autoconfirmed’][‘autoconfirmed’] = false;

Installing the Git client on Windows went relatively smoothly. I installed msysgit with all the default options (using the built in ssh client, for example) other than allowing it to add itself (but not all the unix tools) to my %PATH%.

To add a git repository I had to right click a random folder in explorer, go to “Git GUI Here,” then went to Repository -> Clone. Target directory was the local folder to put the repository in (it can’t exist), and source directory looked like so: “git+ssh://git@server:port/project”. I also had to generate a ssh key before cloning (Help -> Show SSH Key).

Adding the SSH key to gitosis was easy. Going into the administrative directory (as described in the blog post linked to in the last post), I created a new username.pub file in the keydir, added the username to the gitosis.conf file, git committed, and git pushed.

Following the guide here, I installed gitosis to manage my remote Git repository.

A few things I did differently:

To create a user I used OSX’s GUI, rather than the command line. I set their home directory to be on my Drobo (a raided external drive) - to do this I right clicked the user in the GUI and went to advanced options. I then had to copy everything in the old /Users/git folder into the new folder

My pub ssh key file didn’t have a username in it, so the init script raised an InsecureSSHKeyUsername(repr(user)) exception. I appended kevin@kevinmook.com to the end of it and it initialized fine. Apparently you can use whatever username you want, but this will be the username you use to set gitosis permissions later.

I had to add the following to ~/.bashrc

export PATH=/usr/local/bin:/usr/local/git/bin:$PATH

A few things I did wrong the first time:

I didn’t start by using gitosis. I tried to follow the instructions found here. I was worried by the gitosis page’s warning about the inability to set ports (I thought gitosis may be doing funky protocol stuff), but it turns out that’s a git issue, not a gitosis one. If you follow those instructions, every user will have to have ssh console access to the machine.

I tried to change my username after initializing gitosis. You can do this, but you need to change the filename in the gitosis/keydir folder, as well as edit the username within said pub file.

You have to export PATH in ~/.bashrc, it can’t be ~/.profile. I sym linked them to be the same file so the settings would be shared across my terminal and git ssh sessions.

All and all, setting up Git wasn’t too difficult. The only thing that got me stuck for a few hours was the ~/bashrc file, as well as waisting about an hour setting up git without gitosis (mostly because I didn’t know what I was doing).