The Exchange Project - Syncing Exchange to iCal and back

As noted elsewhere, I decided to go on with the Exchange2iCal project I started almost one year ago.I’m actually getting along nicely, also due to the fact that Jeffrey Harris released the wonderful VObject package some days ago. (Thanks to Jon for the tip!)

So far, the following bits and pieces are working:

• I can get the whole Exchange Calendar as an ics file, including recurrence, alarms and meeting status.
• I can parse and compare two iCalendar files and get the following information:
  • New Local/Remote Events
  • Changed Local/Remote Events
  • Deleted Local/Remote Events

The first part can be considered “stable”, that is, if you’re OK w/ read-only access to you calendar, you could basically use that part. You can download these here: e2i.

Warning

These scripts are provided strictly as is. It’s alpha status at best, hacked together to work on my environment and not tested elsewhere than my Mac. There isn’t a README available and I assume that you know how to get the different tidbits needed to make it running, in particular: * Python 2.3 * PyObjC (If you want to profit from Keychain-based authentication to Exchange) * Basic knowledge of Python

(If you can’t, then maybe you’re not the exact target audience for these scripts and better wait for the moment where I have something a little more user-friendly or where Apple themselves release an iSync to Exchange. I know there are quite a bunch o’ people in the wild that’d like to see this working, but sorry, it’s just not there yet.).

I further assume that you know how to read a program and change certain parameters. There is some documentation in the code, but remember: This is not for the faint-hearted! And now go and run exchange.py

The second part (parsing & comparing) is still very buggy and unstable. I get pretty decent results but the parsing & comparing is somewhat unpredictable.

---

Now, I am heading for the following: * Increase the stability of the comparison^[1] * Implement a first one-way updater * Wrap the thing up so fetching, comparing and updating will become one monolithic sync process * Write an iSync plugin

(This last point should be the easiest, as PyObjC can now be used to program NSBundles in Python, as seen here. (Hat tip: John Gruber)

Some other tidbits are going along with this as well: * Increase the stability and usefulness of the XSLT that transforms the Exchange data to iCalendar * Bring meeting attendee info into the iCS if possible * Recur to less file-based transactions (i.e., less data-dumps and more advanced uses of stuff like LibXML2’s Python bindings etc.) * Lure our Sysadmins to install Exchange 2003 ASAP. * Become better in Python programming * Set-up some project space with a constant URL so people can find this — if they want

---

Rationale

You might ask yourself the question what the rationale of this project is? I’ll admit, I’m only doing this to get /.ed once in my life. But before you go and call the dot to slash me, bear in mind that this project is not yet finished — some more time is needed to get something that’s worth a /.

(Actually, getting Exchange on my Mac desktop would be great, too … )

---

^[1] That is, better understand the VObject architecture

More on Exchange & OS X

Some of you might remember my calendaring experiment that happened almost one year ago. Well, there’s some news from that front, despite the fact I decided to abandon it.

1. First, the bad news: No, there’s no real two-way sync between iCal and Exchange. Yet.
2. Second, the bad news: I got degraded from an Exchange 2003 Server to an Exchange 2000 Server, which is bad, because I can’t issue WebDAV SEARCH queries anymore.
3. Third, the bad news: Every time I dump the Calendar with a PROPFIND request, I download some 6MB of Calendaring data.^[1]

And the good news? Well, there’s some, too.

• My new employer does work with SSL, which means I don’t have to open the VPN everytime I want to do some testing.
• I found some great code to access the OS X Keychain, so there’s a way to keep my passwords out of the code.
• I found some iCalendar Python class I can use to access & compare (!) iCalendar events. Mhm, yummy!
• I’m pretty confident I can finish a working example of that kid before Apple releases Tiger. (Which, as it seems, does not really do iCal Exchange sync.)

Well, there’s still some more bad news to add. Exchange migration is planned to be finished later in 2005. And I still regularly fall back on my posts when I search for more information on this topic on Google. (However, I also found a great example on ASPN, which is first here :))

But now to some of the tidbits of the above list:

The Exchange Server

I got to know that somehow, Microsoft crippled their Exchange Server before version 2003 XP1 in a way that it won’t work. More specifically: The examples on MSDN that show how a WebDAV SEARCH request is issued won’t work in Python or elsewhere, with the exception of Microsoft’s Windows-only XmlHttp Active X object.

Apparently, Service Pack 1 for Exchange 2003 fixes this, so there’s hope.

The PROPFIND

PROPFIND is a WebDAV extension to HTTP requests. It allows one to dump all or specific properties of all elements in a WebDAV store. The good news here is, I can get all the data I want. The bad news is: I can’t filter it server-side, which means a 6MB dump each time I get the Calendar events!

I might get that down to 3MB or so, by carefully choosing the properties I really need, and do some filtering on the last changed date elsewhere, so there’s hope as well.

The Keychain

One of the greatest things so far is the discovery of the [Keychain Framework][] on SourceForge. Thanks bdash for the link and the code example.

With the help of this Framework and PyObjC, you can really easily access the Keychain, create new entries and–more important–fetch passwords for existing entries! Whew!

Some sample code looks like this:

import objc, new
def getPassword( aService, anAccount = '' ):
    """Will get the password for aService/anAccount from the Keychain"""

    Keychain = new.module('Keychain')
    objc.loadBundle('Keychain', Keychain.__dict__, 
    bundle_path='/Library/Frameworks/Keychain.framework')

    defaultKeychain = Keychain.Keychain.defaultKeychain()

    return defaultKeychain.passwordForInternetServer_forAccount_port_path_inSecurityDomain_protocol_auth_( aService, anAccount, 0, '', '', 0, 0 )

Pay close attention to the trailing “_” at the end of every function name! Yes, I got caught here in the first place!

Anyway, this is a real helper and I’m thinking of some other frameworks I could use …

The iCal.py

Last, but not least, I found iCal.py, a little helper class that loads and interprets iCalendar files. It’s far from complete but a very good start, indeed–and it saves me from learning Ruby.

The Outlook

So, where have I gone since the post in January? Not so far by some means–but a lot if you understand what progress I made with respect to some of the points outlined above. I feel much more confident to actually come up with a solution that works!

---

^[1] I might eventually cut them by some good amount, once I know what properties I really need.

Newsreader vs. Del.icio.us

With respect to news feeds, I’m definitely a collector, which is part of the reason why I prefer PulpFiction over NetNewsWire: I keep a big number of news as persistent data, with the exception of some feeds with a very low SNR.

So, while most items just land in the database, I occasionally flag some items because they have some more significance to me: Either, they point to some important reference or they are particular well written posts.

At the time of this writing, I have about 230 flagged posts and several thousand non-flagged ones. Obviously, there’s not much more one can do (yet), except for adding labels to flagged articles. Once smart folders will be implemented in PulpFiction, this might change, though. But it’s a nice collection of useful stuff, anyway, and far better than bookmarks in Safari, as there’s actually some content associated with.

---

Recently, I signed-up for a del.icio.us^[1] account. Some days ago, I discovered Cocoal.icio.us, a Cocoa client for del.icio.us. Besides the fact that you can easily manage and search your bookmarks, it integrates with Newsreaders through the NetNewsWire common post interface^[2].

All in all, this represent an intriguing way to manage essential or important information. However, I was facing a dilemma with respect to my current strategy:

Where do I manage my bookmarks? Do I keep the flagged system in PulpFiction or do I store all interesting URL’s in del.icio.us?

After some musing, I came to the conclusion that the best thing would be, to combine the two: As Jon Udell pointed out in his post about del.icio.us, you can actually access the whole shebang via a RESTful API: With some Python magic, I can synchronize my flagged posts with del.icio.us.

As Jon already pointed out, this should give the bets of the two worlds: I will be able to keep the associated meta-data of a post and still have the advanced bookmarking of the online repository, including the possibility to find related posts — which in turn I can either blog myself to have some meta-data or — maybe even better — I just subscribe to the del.icio.us feed of that group of URL’s.

This will give me the added benefit of leveraging the group mind. Or as Jon put it nicely:

The group mind knows things that I don’t; I know things that it doesn’t; use of a common tag effects a two-way transfer that benefits both.

---

^[1] del.icio.us, for those who don’t know it, is a social bookmarking service. Members can post their bookmarks, tag individual bookmarks and do many more things with their lists. Part of the deal is, that everyone can see your bookmarks and that you can see who’s relating to the same bookmark and/or tag.

del.icio.us also gives you hints if other people subscribed to the same URL etc. You can search by tag, user etc.

^[2] Beisdes Cocoal.icio.us, there are Ecto and MarsEdit on the client side, as well as NetNewsWire and PulpFiction on the reader side. There might be some more, but these are the ones I’m currently aware of.