Selenium::WebDriver::Error::ElementNotDisplayedError:
Element is not currently visible and so may not be interacted with

After spending way too long to try and solve this problem (“THE ELEMENT IS VISIBLE DAMN YOU!!!!”) I found a few little tweaks to my setup that help remove and/or mitigate the error.

The key to this error is that Capybara is trying to find and/or interact with an element and it’s not quite ready on the page yet because some JavaScript hasn’t executed yet, or there’s a delay with the test web server.

jQuery Effects

If you’re running tests, there’s a pretty good chance you don’t care about any jQuery effects. You’re just interested in the result. If that’s the case, then add this line to your page layout:

= javascript_tag '$.fx.off = true;' if Rails.env.test?

That will turn jQuery effects off in the Rails test environment.

Capybara “wait_for”

This is a particularly helpful method that, like its name implies, causes Capybara to take a chill pill, and wait for a particular condition to be met. Call it like this:

wait_for (15) {page.should have_content "Some text."}
wait_for(30) {page.should have_button "Button Text"}


This just gives the page a little time to catch up before Capybara blows up on you.

capybara-webkit

A new Selenium driver from the fine folks at Thoughtbot, this is a headless (this just means it doesn’t open a visible window) browser that Capybara can interact with. Ideally it should be faster than opening up a browser window for each test. I managed to get it working, mostly. It’s still kind of new, so there are a few things missing, but it might be a good idea to watch the github project.

For me, one of the best things about using Ruby/Rails/Sinatra for web development is HAML/SASS, which allow me to quickly write and maintain HTML/CSS without having to worry about all the pointless stuff, like brackets etc. A by-product of using HAML/SASS is that when you have to write in something else, it causes me grief. JavaScript is a case in point.

Coffeescript (and to a lesser extent) bistro_car solve this. It brings the beauty of a ruby-like language to writing JavaScript, so that once again you don’t have to worry about all the crap that goes along with it.

However, I ran into a slight problem when following Dr Nic’s setup tutorial today. CoffeeScript worked, bistro_car worked within the console, but it didn’t work when accessing the page through the browser. But some path re-working, and a kind word or two from @sutto, and we’re good to go.

For future reference, here’s what I did. I’m using the homebrew package manager. Mainly because it’s awesome, and I use MacOS X, which is also awesome.

So, I installed node.js, coffeescript and bistro_car like so:

brew install node
brew install coffee-script
gem install bistro_car

Then, add the following to your environment.rb file:

config.gem 'bistro_car'

Create an ‘app/scripts’ directory in your Rails app, which is where you can store all your .coffee files (application.coffee etc).

You then get bistro_car to include those script files by putting this in your layout/template/page/whatever:

coffee_script_bundle

Then, you load the page, and bistro_car automagically uses coffee-script to convert your .coffee files to javascript. Except, of course, when it doesn’t work because something is screwed somewhere.

As with almost every problem on a *nix computer, the problem is a path one, and thankfully solved with some simple symlinks. It seems passenger operates in it’s own little world when it comes to paths, and it just didn’t want to play nice with my environment. So, I tricked it as follows:

sudo ln -s /usr/local/bin/coffee /usr/bin/coffee
sudo ln -s /usr/local/bin/node /usr/bin/node

And that, my friend, is how you skin the unix path cat.

Then you need to create a “.autotest” file with the following:

Then it’s just a case of typing “autotest” in RAILS_ROOT.

Simple.

DropBox

DropBox is simply file synchronisation done right. You install it, drag the files you want to have synced between two computers into the Dropbox folder, and BOOM, it copies everything up to their server, and when you set it up on the second computer, copies everything back down again. After the initial sync, which can take quite a while if you’ve got lots of stuff, it’s pretty much seamless. Create, or make a change to, a file on one computer, and before you know it, it’s available on the other computer.

Need access to those files from another computer, or even from your phone? No problem, they have a web interface that give you full access to all of your “synced” files. Simple.

To add a lovely cherry to the ice cream sundae that is DropBox, they also have versioning. That means if you make a series of dunderhead changes to a document, you can just rollback to the last good copy. Awesome.

Git and GitHub

For the longest time (mostly while I was still working on Windows) “Source Control” was some mystical mumbo jumbo that only people in large companies needed to worry about. Of course, that was mainly because at the time the options for Visual Studio only included Microsoft SourceSafe, which is perhaps some of the crappiest, most unusable software ever foisted on the unsuspecting public by a company with a long history of foisting crap on the unsuspecting public.

Once I switched to MacOSX and started working with Rails, it became clear that all source control systems weren’t created equally. Subversion seemed to be the choice of the community, so I loaded everything up onto my own SVN server. A world was opened up to me, and I could code with wild abandon, safe in the knowledge that I could rollback with no danger. Of course, that was the ideal world, but the basic approach worked. I created branches, tagged releases and deployed sites from SVN.

As with most things, people started grumbling about SVN and some of it’s shortcomings: difficulties in merging different code branches, no distributed repositories and some general discontent. Git was the solution, and the rails community just jumped ships in the middle of the night. Before you knew it, everyone was using Git, and also GitHub. I won’t go into the specifics of why this combo is so good, except to say that if you’re not using git, well, good day to you sir.

git and config_files

In addition to using git to store your development work, you can also use it to manage system configuration files. I found the wonderful config_files. It’s a collection of shell, git, vim and irb configuration files. So? How does that help?

A great feature of git, and github in particular, is that you can “fork” a repository. You effectively copy someone else’s repository, and you can then make all the changes to it you want, without affecting the original. If you want to let other people know about your changes, you can send the original developer a “pull request”, and they can grab your changes, and incorporate them, or not.

So, fork the config_files repository, and then clone it to your own system. Run the “install.sh” script, which basically just sets up a series of symlinks in your user directory. These symlinks point to the files in the git repository. You make any changes you want to the repository and commit them up to github. Then do the same on your other computer, and bingo, you’re synchronising your configuration between machines.

And why would you want config_files? Well, you get some great all round system setup tidbits, but my favourite bit is how it prints the current git branch as part of your prompt, saving you the need to try and work out what you’re in the middle of doing.

Tasty.

Backup Overview

Using a combination of Super Duper, a couple of backup external HDD’s, Dropbox, Jungle Disk and Amazon S3 I have built up what I think is a relatively comprehensive and reliable backup system.

Local Backups

SuperDuper is a wonderful program that creates bootable mirrors of a hard drive. Effectively what this means is that if a hard drive fails, you can just replace it with the backup copy. You can schedule the backups to run as often as you want, which ensures that your backup copy is fresh and useable in case of a disaster.

And what am I backing up? I have two main drives that I use: The system disk (the built in drive in my iMac which contains all of my work files, personal documents, applications and the Operating System), and an external FireWire800 drive that stores photographs, videos, movies and TV shows.

Amazon S3

Amazon S3 is basically an online storage system that can be accessed via an API to upload, manage and retrieve data. Amazon charges for both the uploading of data (US$0.10 per Gb), and then for the actual storage of that data. The uploading charge is a one off cost, and the storage costs are charged monthly (US$0.15 per Gb per month).

Jungle Disk

One downfall of S3 is that it’s not actually setup to be used without additional software to manage the mechanics of the backups. Step up Jungle Disk. It’s pretty simple software: you give it your S3 account details, tell it what to backup, when to back it up and it will go ahead and take care of it for you. Simple. And then you get your monthly bill from S3. Nothing else to do.

One big downside of the whole online backup setup is the time it takes to actually backup any large amount of data, and that’s a limitation of my internet connection more than anything else. When you’re uploading 100+Gb, be prepared for a bit of a wait

The cost of Amazon S3

While the monthly cost of the Amazon S3 service makes it a perfect online backup solution for data that you would class as priceless (photos, videos of the kids, work files, personal documents), when it comes to backing up stuff that can be easily replaced at a relatively low cost, such as iTunes TV shows and Movies, there soon comes a point where it’s not actually economically feasible to use online storage.

iTunes TV Shows

The average 1 hour (42 minutes of network TV) HD episode of a TV show on iTunes is about 1.4Gb. In addition to this, you also get an iPod/iPhone compatible SD version which is generally 600Mb. So, a single TV show is effectively 2Gb of data that needs to be backed up.

What is the cost of backing this file(s) up? Well, there’s a US$0.20 charge for uploading it to Amazon S3 initially, and then a US$0.30 charge per month. At that rate, it only takes 10 months of storing the data online until you’ve actually paid for the file twice. This means that if you lost the original file 12 months after you first bought it, you’d actually be better off buying the file again.

This effectively renders online backups for iTunes TV Shows pointless, considering how often you’re actually likely to watch a TV show, and also how cheap the cost of having a local HDD mirror is.

iTunes Movies

Currently iTunes Movies are only available in SD (boo to the movie studios and Apple for this one) so the files aren’t as large as they could be, but they’re still pretty sizable, weighing in at 1.67Gb for the recent “The Dark Knight”, which cost US$14.99. And how long until it’s not feasible to store this on Amazon S3?

At the purchase price, it would take 4.99 years until you’ve paid twice. However, Apple drops the price of new release films to $9.99 after about 4-6 months, so the replacement cost is greatly reduced. At this price the cut-off becomes 3.33 years. Obviously this timeframe requires a judgement call as to whether or not it’s worth it. Personally, I’d rather just trust my local HDD backups.

iTunes/Amazon Music

Music files are obviously considerably smaller than video files, and as such are going to incur a greatly reduced monthly fee for storage. Your average iTunes album costs approximately US$9.99 and is generally around 110Mb. This small size means that it will actually take about 49.95 years until it’s been paid for twice, buy which point you’ll either be A) dead, or B) listening to music on your personal bone implant that plays whatever music you want that’s being broadcast by SkyNet.

So, music is one area where it’s probably economical to maintain an online backup of your files, particularly considering how annoying it would be to go and re-purchase the 800+ albums you’ve got in your library in the first place.

Other possible solutions and problems.

Whenever you’re talking about local HDD backups, it’s always worth considering a Drobo, which is a redundant array of HDD’s that theoretically keeps your data happy and safe. I don’t have a Drobo, but I know that users who have one swear by them.

The one problem with local backups is that, of course, they are susceptible to local threats, e.g. fire and theft. There’s no point in having duplicate hard drives that slavishly mirror each other if some reprobate comes along and pilfers them both. Which means you need a third mirror, that you store off-site. Which in turn means you need a fourth drive that you store off-site in rotation with the third one. An endless cycle.

I’ve been working on an updated version of the Soil Quality website for a little while now, and recently needed to deploy the site to a staging server for testing. I ordered up a 1/4Gb Accelerator from Joyent, configured it, and deployed the app, just like I’ve done with 20+ other sites, and BOOM, straight into a brick wall.

I opened up the log files and saw this error:

** Daemonized, any open files are closed.  Look at /tmp/soilquality-mongrel.8200.pid and log/mongrel.8200.log for info.
** Starting Mongrel listening at 127.0.0.1:8200
** Define INLINEDIR or HOME in your environment and try again

Wonderfully descriptive I know, but something I’d never run into before. So, fire the the google and it turns out it’s a common error, with a common fix: just put

ENV['INLINEDIR'] = RAILS_ROOT + "/tmp"

into your config/environment.rb file, make sure the directory exists and that it’s writeable by the mongrel, restart, and away you go.

But of course, that didn’t fix it, did it.

Much hair pulling ensued. I finally enlisted the help of Darcy Laycock and together we managed to track the problem down to the attachment_fu plugin, which was causing the problem as the mongrel process booted. OK, so now we knew where the problem lay, but what was causing it.

It turned out to be the ImageScience image processor, or more specifically the way the attachment_fu plugin and ImageScience work together ON A 2ND GENERATION JOYENT ACCELERATOR – eg the ones that use pkgsrc. Didn’t seem to cause the problem on an ubuntu machine, nor on one of the older BlastWave based Accelerators. I’m not sure why as of yet, as I was more worried about getting some sleep last night when we managed to fix the problem.

And the fix? Basically, strip ImageScience out of attachment_fu. Remove

vendor/plugins/attachment_fu/lib/technoweenie/attachment_fu/processors/image_science_processor.rb

and remove “ImageScience” from this line

@@default_processors = %w(ImageScience Rmagick MiniMagick Gd2 CoreImage)

in this file

vendor/plugins/attachment_fu/lib/technoweenie/attachment_fu/attachment_fu.rb

Like I said, I have no real idea why this is happening at the moment, but I’ll try and work it out and update this post if I get anywhere.

Once again, thanks to Darcy for A) being a sounding board, B) helping fix the problem and C) being up and available when I needed him

There is a huge list of international and national speakers on the program, one of whom is Ben Buchanan. I recently conducted a short email interview with Ben, talking about some of the subjects that he’ll touch upon in his talk Hacking Humans: Advocating for a better web

Ben is just one of a range of awesome speakers who are going to be presenting at Edge of the Web on the 6th November. On the following day, there are 4 half-day workshops on a diverse range of topics.

And then, to cap the two days off in style, Friday night sees the WA Web Awards, where, in a burst of glorious self congratulation, the best of the WA Web industry are recognised for their efforts. Get your tickets here.

Thursday, 25th September 2008

Opening Keynote – Lynne Johnson

Boo. So mismatched to the audience it wasn’t amusing in the slightest. If we were a bunch of magazine publishers from 2003, then it might have been interesting, but we weren’t, and so it wasn’t.

Derek Featherstone

Nubbins. Yay!!

I’ll just pretend that the opening keynote didn’t happen, and that this was the start. It’s always good to see someone who A) has command of their field and B) is a great presenter. Totally comfortable on stage, and thoroughly knowledgeable about his topic, which he managed to make seem almost exciting, which is pretty hard for what is essentially a dry topic.

iPhone Development

Tim Lucas and Pete Ottery worked on the iPhone Site and did a good presentation on some of their experiences. Perhaps because I had already started on the iPhone site for 5 Senses and I had already encountered most of those problems, I didn’t really come away with much. But that’s cool, because it was an intro level talk, not for an iPhone Dev Conference or anything.

JavaScript Libraries/Frameworks

Yeah, good times, much laughter, fun, and frivolity, and then when Cameron rolled out the Drum Machine – FUCKING BRILLIANT.

I’m not sure if I actually learned anything, except that just about everyone is using jQuery, and those of us still on Prototype are mocked mercilessly.

August de la Rejos

Hmmm, ok, but hold the ads.

An interesting talk about the progress of user interaction with computers, as well as the psychology behind it, that had some wonderful videos outlining potential future developments, but was undermined by the fact that they were MicroSoft corporate videos. Oh well.

WebJam8

Bucketloads of awesome, mixed in with tasty beverages, fine companies that sponsored the tasty beverages, fine talks, and conversation. Mad times. Cannot wait until WebJam 9 as part of Edge of the Web

Friday, 26th September 2008

Jeffrey Veen

Hot. Wired.

Awesome talk about visualising data, and what good design can achieve. Memorable, and featuring the entirely wonderful Gapminder World

Gina Bolton

No. Not good. I don’t know if Gina has a ninja like mastery of CSS (I’m guessing she does since she consults/designs for companies like Apple and has written a book), but statements like “from what I’ve heard when talking to developers, computers start counting at 0” doesn’t make code-monkey happy.

A thorough understanding of your topic, and avoiding statements like “I haven’t checked the blogs for a month or so, so I don’t know if the CSS3 Working Recommendation has been ratified or not.” Seriously!

Michael™ Smith

Not too bad. Bit of a laundry list. Of course, any recitation of the trials and tribulations of HTML 5 isn’t really going to be anything but dry. Still, good to hear for the most part. However, the take away from this, Gina’s and Douglas Crockford’s talk is that it’s been 10 years since any successful revision to HTML, CSS or ECMAScript, which just instills wonderful amounts of trust and faith in committees.

And really? Michael™ What is this – Burning Man?

Myles Eftos.

Huzzah! The madpilot flew like a crazy bastard over the top of the dangerous territories of OpenID and OAuth. And if that isn’t enough, he launched incursions into HTTP 1.1 as well as REST and SOAP. Lucky for all he came back unharmed.

Douglas Crockford

Good. Real code. Real ideas.

And scary. (We are so totally screwed if even half of what he was saying comes to fruition.)

Mark Pesce

Oversold. A fishing basket full of “ok, but meh”. One of the apparent pitfalls of giving what, from all accounts, was an amazing presentation at WDS07 that inspired everyone who went, is that the next year expectations are particularly high. When those expectations aren’t met (probably through no fault of Marks) it leaves people underwhelmed. I didn’t necessarily agree with the central theme of the presentation (that twitter and the like are going to transform society and its’ institutions), so that didn’t help.

One thing I will say is that if you need to make a point that you aren’t going to do next year’s closing keynote, you probably shouldn’t be doing this years.

A slight down to end on.

DRINKS!!!!!

Wheee! All that needs to be said can be summed up in these two tweets that I, um, twitted.

Salad is what food eats

09:19 PM September 26, 2008 from twitterrific

Has teh drunk on. Hi meat lady!

12:18 AM September 27, 2008 from twitterrific

Would I go again?

Totally. As we are no doubt about to find out, running a conference is a tricky thing. Getting a mix of the right speakers, hoping they are all in top form, and cultivating the vibe is no doubt a difficult thing. I had a great time in Sydney: I met some great people, learned lots, felt inadequate (I mean, you’re talking to Douglas Crockford over a beer … what do you say?), and saw a Europe video on a large screen.

Every now and then a story will appear in my feed reader about the use of one browser being higher than another, and more often than not, that site has a skewed audience, which means that it’s users are more likely to have the latest and greatest than other, less browser obsessed users.

So, bearing in mind that this probably has no relationship to the wider world than any other site, here is the browser share of visitors to the 5 Senses Website over the past 21 months.

And the breakdowns for the various versions of Internet Explorer, where “IE x” is all versions other than IE 6 and IE 7.

and FireFox.

What is evident from these numbers is that while IE is still the dominant browser, with almost 2/3rds of market share, Firefox has been slowly and steadily increasing. But not necessarily at the same rate that IE is losing share. Firefox seems to be losing a bit of ground to other browsers, namely Safari.

The interesting number is that Safari has doubled over the sample period. True, it’s still third by more than a few lengths, but it seems to be taking some of those gains from IE. I’m guessing this is mainly due to the increased market share of Apple in general, and not some massive use of Safari on MS Windows.

Other, possibly unrelated points that can be extracted is that FireFox users are more likely to upgrade their browser than IE users. This can be seen by the almost complete drop off in FF 1.5x users at the beginning of the timeframe, and the rapid rise of FF 3.x in the last 3 months, to the point that it almost equals FF 2.x. That said, there certainly appears to be some die-hard holdouts for FF 1.x, who are disappearing at a slower rate than those still using FF 1.5.x

So, not a scientific sample and/or analysis by any stretch, but still interesting. The surprising item here is the doubling in share of Safari (which, in the authors less than humble opinion is the best browser there is.) True, it didn’t have far to go to double that share, but it still had to be done.

The upshot of all of this is that, unfortunately, there is still a large portion of the internet population still using IE 6 (approximately 20% of all visitors it seems), so it still needs to be supported, or those people need to be encouraged to migrate upwards. How do we do that? Well, your guess is as good as mine.

rake rails:freeze:edge TAG=rel_2-0-1

and the appropriate version would be copied into your vendor/rails directory.

Now if you do that, rake downloads a zip of the edge release. Which is fine and all, but sometimes you don’t want to be on edge … like in any production site.

So, I found a screencast that goes through the process, but I thought I’d actually put the text into a post, mainly for my own reference more than anything else.

$ rails path_to_app

$ cd path_to_app
$ git init
$ git submodule add git://github.com/rails/rails.git vendor/rails

At this point, git will effectively clone the repository, so that you can then choose one of the branches to “freeze” to. Type “git tag” to get a list of all the available tagged branches. Choose the one you want and type

$ git checkout v2.1.0

And that’s it. Slightly more involved than the old way, but still none too shabby.