Treat with Jermolene

The organisers have posted the video from the talk that I gave at the Next08 conference in Hamburg earlier this month. It doesn’t seem to be possible to link directly to the video; you’ll need to scroll down until you find “Jeremy Ruston”:

http://www.next-conference.com/next08/program.html

I was a bit more nervous than usual because I was standing in at short-ish notice for my boss. I added some laconicity, but the material is essentially his, as you can see in this video of JP giving the talk last December at LeWeb 3 in Paris. Osmosoft helped out with the slides, using RippleRap and TiddlyWiki to maintain my strict PowerPoint quarantine.

Happily at least one person liked it.

Along with several of my Osmonaught colleagues, I will be attending OpenTech 2008 in London on Saturday July 5th 2008. It’s a very low cost event (£5 to get in!), and is focussed nicely on the hacker community. I went to the last one in 2005 and had a great time.

This time, I’m giving a talk entitled “TiddlyWiki Tales” which will be a round-up the latest TW developments, both from Osmosoft and the wider community. Later on in the day, my colleague Paul Downey is talking about “The Web as Agreement“.

We’re also holding a competition with a TiddlyWiki flavour and some interesting prizes:

The TiddlyWiki OpenTech competition gives you the chance to win a BUG (plus four modules)! The BUG base station and modules use open source software and snap together to create whatever device you want. There’s also a great second prize of a GP2X Linux powered handheld games console. A stack of O’Reilly books will be the third prize.

For your chance to win one of these great prizes, all you have to do is show us your TiddlyWiki skills. TiddlyWiki is written entirely in javascript, and has a great plugin architecture, so just like the BUG you can twist it to do whatever you want. It can pull in RSS feeds, and because it can be a local file (and you don’t have to worry about same domain policies), it’s great for mashups. There are already 400+ plugins you can use, but don’t let that stop you from writing your own new ones!

(See the Osmosoft site for full details).

So, I’ll be attending BlogTalk 2008 next week in Cork, along with most of the Osmosoft contingent. I’ll be running a brief demo of some of the latest TiddlyWiki wizardry, and we’ll have a little exhibition stand too. Being a webby sort of conference, we’re running a RippleRap server so that people can create and share notes about the conference sessions (regardless of the state of the wifi!).

I went to BlogTalk 2006 in Vienna as an independent hacker, and thoroughly enjoyed my first exposure to a crowd of people who are generally at the forefront of thinking about social software. Thomas and John do a tremendous job of bringing people together, and so I’m delighted that this time around BT are one of the sponsors of the event.

So, here at Osmosoft been lucky enough to have access to an iPhone to play with for the last few weeks. Our first discovery was that TiddlyWiki works just fine on the iPhone - apart from being excruciatingly slow. TiddlyWiki is so meaty you really feel the 20x speed difference between a MacBook Pro and an iPhone. Anyhow, I’m confident that we can slim TiddlyWiki down to the extent that it runs at a reasonable speed on these new portable slow AJAX devices. (If you try the leaner and meaner first version of TiddlyWiki on an iPhone, it’s a good deal snappier).

The other problem is that saving doesn’t work properly because the TiddlyWiki is being accessed via http. We can expect that TiddlyWiki adaptations that support serverside saving (like TiddlySpot) will work, but that doesn’t really speak to the TiddlyWiki vision of untethered access to the information that you need.

I quickly discovered a potential solution: the iPhone supports bookmarks whose URL is a data URI. The procedure is to use something like Hixie’s data: URI kitchen to encode a file (HTML, images or PDFs all seem to work fine; possibly Word and Excel Documents too) into a data: link. Then right-click on the link and select “add to bookmarks”. You’ll end up with a bookmark that starts something like this:

 data:text/html;base64,PCFET0NUWVBFIGh0bWwgUFVCTElDICItLy9XM0MvL0RURCBY....

When that bookmark is invoked, the base64 data is converted back into a binary file, which is then displayed by the browser - all without recourse to the Internet.

The really interesting thing is that Safari/iPhone support bookmarks of up to 32 megabytes or so, which suggested the possibility of embedding an entire TiddlyWiki in a bookmark. And, it turns out that it works — giving you offline read-only access to TiddlyWiki files on an iPhone.

Here are the steps to convert any TiddlyWiki to use offline on the iPhone

1. Locate the TiddlyWiki file you want to convert (it can be stored locally or on the internet)

2. Go to Hixie’s data: URI kitchen in Safari

3. Make sure the ‘base64′ checkbox is enabled

4. Either enter the URL of the TiddlyWiki file in the text box, or use the ‘Choose file’ button to select a local file

5. Click ‘Generate’

6. Wait; it can be quite slow to do the conversion with a big TiddlyWiki file

7. Right click on the resulting link and select ‘Add Link to Bookmarks…’

8. Go to iTunes and make sure that you’ve selected to sync your Safari bookmarks with the iPhone

9. Sync the iPhone

10. On the iPhone, select the bookmark

This is all still quite rough and ready; there’s still a bunch of work to do to get a first-class TiddlyWiki experience on the iPhone. In my next post I’ll be looking at optimising the TiddlyWiki user interface for the iPhone, to make it look and feel more like a native application.

I’ve started regularly hanging out in the TiddlyWiki IRC channel. It’s an open chatroom that anyone can join; there’s half a dozen or so regulars, including FND and Saq, and several of my colleagues at Osmosoft.

The channel is called #tiddlywiki and it lives on the server irc.freenode.net.

IRC is quite old-school; there’s a bunch of different clients available for various platforms. I’m using something called Conversation for the Mac.

So, I started TiddlyWiki back in September 2004. It is now, by many measures, a moderately active and successful open source project. The forums are busy, the ecosystem is vibrant, and many people get a great deal of satisfaction and enjoyment from using TiddlyWiki, improving it, and helping others to do so.

Sometimes, people ask me how they can start a similar project, particularly how they can build an open source project around an idea that they’ve got. For a long time, the question used to make me stop short and give the glib answer that I didn’t know, because it had happened to me by accident. I’m acutely aware that I knew next to nothing about open source until long after I first published TiddlyWiki, and I felt that my experience was so specific to TiddlyWiki that it wouldn’t help anyone else.

Now, though, with the benefit of hindsight, there might by now be a few things I can generalise from my experience. Perhaps. Anyway, I’ll try for three things fairly simple observations for the moment.

1. You have to be fearless, if not reckless. In my case, for the two years or so that I was precariously working full-time on TiddlyWiki, I didn’t earn anything like the money I’d earned in my previous jobs. The only way I could survive was not to put away any money to pay the tax bills that I was building up but to spend everything I was earning on day-to-day living and bills. You can juggle things in the UK so that you don’t pay tax on your income for two years if you’re careful, but my cunning plan was clearly not something that a fully trained accountant would advise anyone to do. Especially given that I had no indication of any future payout that a happy-go-lucky angel investor would recognise, let alone one to impress the friendly tax man. Living hand-to-mouth is never fun, but I got huge motivational sustenance from the enthusiasm of the other people who were putting so much into the project. I reasoned that dozens of strangers wouldn’t be investing in it if TiddlyWiki were without serious merit, and my natural blind optimism filled in the leap that something of serious merit would eventually allow me to pay my bills.

2. You have to build the thing before they will come. An idea alone is not enough. In the case of TiddlyWiki the first proof-of-concept version that I released in September 2004 has around 850 lines of JavaScript code, much more than a trivial proof of concept or a wireframe sketch. I then worked on TiddlyWiki for around 9 months as a pretty much solo effort, driven by the bug reports and feature suggestions that I was receiving. For all of that period, it would have been hard for me to describe TW as an open source project in anything other than license terms (back then there was no subversion repository, no mailing list and no bug tracking system). It was only when TiddlyWiki itself was mature enough to sustain a user community that the demand built up for it to become a proper open source project.

3. You have to make it easy for your users to find and use your source code. In the case of TiddlyWiki, the website is the product is the code is the documentation, which has neatnesses of its own. But there’s more than that: one of the things I’ve learned from working with the TiddlyWiki community is that there are an awful lot of almost-programmers out there: people who couldn’t reliably tell you much about the syntax of JavaScript but have enough wit and common-sense to splice and modify samples of code. Especially if they have the means to try out their changes easily and safely. Give them these tools, and these people can do wonders. In contrast, many open source projects have much more complicated installation, build and execution requirements. It’s often surprisingly hard just to locate the source code and browse your way to a particular part of the codebase.

A great example of how successful TiddlyWiki has been at lowering the barriers to contribution is the Digital Supplement for “Web Campaigning” by Kirsten Foot and Steven M. Schneider” (MIT Press 2006). The authors have published their source materials as a TiddlyWiki file that fellow researchers can download and use to begin their own work, use as a template or indeed study while on an aeroplane. The really interesting thing is the way that Steven assembled their own unique customisation of TiddlyWiki, consisting of a handful of off-the-shelf plugins and one that he wrote himself to handle their unique requirements for footnotes and endnotes. The macro isn’t particularly complicated in programming terms, but it radically simplifies the annotation of data, and demonstrates a beautiful blurring of boundaries between programming and writing.

So, next Thursday at 10am, I’m presenting a workshop about TiddlyWiki at BT’s offices in Adastral Park near Ipswich - better known to geeks as Martlesham Heath. It’s mostly an internal BT event open but we can accommodate a limited number of external visitors. I know it’s short notice, but if anyone out there is able to come along, you’d be very welcome - leave a comment here, or drop me an email.

So, Google have finally stopped hinting and gone and launched Gears:

Google Gears (BETA) is an open source browser extension that enables web applications to provide offline functionality using following JavaScript APIs:

• Store and serve application resources locally
• Store data locally in a fully-searchable relational database
• Run asynchronous Javascript to improve application responsiveness

Which is marvellous stuff. Several people have pointed out that the requirement for a client side installation will slow down adoption. I’m not so sure; for quite a few people, the ability to work offline is plenty valuable enough to prompt them to install. And don’t underestimate the lengths that Google will go to to try to get their Google Pack on as many PCs as possible.

I’d very much like to experiment with using Gears as backend for TiddlyWiki; one thing I’m particularly curious about is whether offline applications are tied to the browser on which they were last used online.

I’m delighted to announce that the mighty BT has acquired my tiny little company Osmosoft Limited. I’m joining BT as Head of Open Source Innovation, and I’ll be building a crack open source web development team called BT Osmosoft. To say the least, this is big news for me personally, and I hope will have a positive and lasting impact on the future of TiddlyWiki.

BT is becoming a remarkable thing: a truly internet-scale consumer company that doesn’t rely on owning “secret sauce” software for it’s business. At the most senior levels, there’s an appetite to embrace open source that wouldn’t disgrace a web 2.0 startup. I’ll be working with a great many talented and interesting people, and I’m looking forward to it immensely.

Meanwhile, TiddlyWiki has benefited from something rather magical: a global community of eager people who have gathered around it and generously contributed to it, striving to make it better in a spirit of good-natured sharing. I’m regularly astonished by the inventiveness and resourcefulness of this community; I feel a part of something much bigger and more significant than I could ever manage on my own.

I’ve always kept TiddlyWiki fiercely independent — for instance, not carrying advertising (or indeed accepting venture capital investments). I feel that to do anything else would be disrespectful to the grass roots users and enthusiasts who make TiddlyWiki so useful and intriguing. Now that I’m taking up a commercial position it’s necessary to take certain steps to enshrine that independence more formally.

I have therefore legally assigned my copyrights in TiddlyWiki to an open, non-profit foundation called UnaMesa. I think that TiddlyWiki is at once too fragile and too important to be wholly owned by any one player in the ecosystem; common ownership allows everyone to work together on a level-playing field. There’s a lot more to say about UnaMesa, and I’ll return to it in a later post.

I’m looking forward to being able to improve some areas of TiddlyWiki that have not received enough attention in the past - like a better plugin catalogue, automated testing, better accessibility, and easier security. This won’t be by BT taking over the project, but rather by supporting the open source process and helping out with resources when and where it can.

I hope BT’s endorsement of TiddlyWiki will open up new applications that we haven’t thought of yet. To meet the challenges that they bring, I’ll continue to strive to keep the core of TiddlyWiki true to it’s origins as a lean, efficient non-linear personal web notebook.

UPDATE: I’ve now published these bookmarklets to http://www.tiddlywiki.com/#TiddlyBookmarklets

So, I’ve been preparing for the release of TiddlyWiki 2.2 and have just run into a niggling little problem that’s ended up triggering something quite fun.

I’m at the point where I’m preparing the content for the new release, taking the existing tiddlywiki.com content, folding in the new beta documentation and making sure that everything is up to date and cross-references. The problem arises because the new release adds a “changecount” extended field to each tiddler. It tracks how many times that tiddler has been modified since it was created or received from a server. Our current build process will propagate an extended attribute like that all the way from me making some minor edit in my local copy of tiddlywiki.com up to publication on the internet. That’s a problem, because we should be distributing tiddlywiki.com with clean content, with no “changecount” field.

Various more or less terrible solutions occurred to me immediately:

• Doing a search and replace in the final index.html file to remove all the “changecount” attributes just before publication
• Modifying the build process to clear out the “changecount” attribute
• Writing a new macro that presents a button that scrubs “changecount” attributes from every tiddler

The problems with these solutions being “tedium”, “not knowing Ruby” and “inappropriateness” respectively. Inappropriateness being that such a macro would have to form part of the standard distribution/build process, and yet it seems rather a hack that shouldn’t really be needed.

All of which led to the idea of using bookmarklets - the ultimate platform for hacks. A bookmarklet is a browser favourite or bookmark that has a “javascript:” URL containing executable code, instead of a normal “http:” URL. So, when you click on it, rather than navigating to a new page, it executes the JavaScript code - as if it were on the current page. That means that bookmarklets can reach into the internal structure of a page and do almost anything - including things like stripping “changecount” attributes. With most browsers you can pack enough code into the URL to do some quite useful things.

I knocked up a couple of proof of concept TiddlyBookmarklets to explore what can be done:

• ScrubTiddlyFields - Strips out all the extended fields from a TiddlyWiki document, including that pesky “changecount” field
• ScrubTiddlyShadows -Restores any overridden shadow tiddlers in the current TiddlyWiki document. Handy when you’ve gone mad with PageTemplate customisations and your TiddlyWiki document won’t display properly
• TiddlyRescue - Strips out the raw content of a TiddlyWiki document and displays it in a new window. Handy when you’ve inadvertently been editing an online version of TiddlyWiki that isn’t letting you save changes in the usual way.

Here’s how to use them:

1. Drag one of the links above straight into your browser links bar
2. Alternatively, right-click on the links and select “Create favourite” or “Create bookmark”
3. Navigate to a TiddlyWiki document such as http://www.tiddlywiki.com/
4. Select your new bookmarklet to execute it

PS If you’re interested in writing your own bookmarklets, check out this excellent bookmarklet editor that simplifies packing in all that JavaScript, and some useful tips.

UPDATE: Looks like I don’t know enough about WordPress to make the bookmarklet links work properly (thanks Ace_NoOne). I’m fiddling about to try to get them working…