Stefan Tilkov's Random Stuff

On Monoliths

| Comments

A while ago, I gave a talk at QCon about breaking up monoliths (there’s a video up on InfoQ), repeated it in a slightly improved version at JavaZone (see slides and video), and the topic continues to come up in almost every consulting engagement and client workshop I’ve been involved in since then. Like many of the topics I talk about, it’s somewhat unfair that I get the positive feedback and people assume I came up with the ideas all on my own: Most of stuff like this is the result of collaboration, with my colleagues at innoQ (see for example an article I wrote with Phillip Ghadir for ObjektSpektrum if you read German), as well as customer staff. But wherever it originated, I found that it strikes a nerve with many developers and architects, not only in big companies that conduct million-Euro development projects, but also in smaller e-commerce companies and even startups that have started to become successful.

The main idea is this (no surprise for almost everyone, I guess): Nobody wants monoliths, i.e. big systems composed of hundreds of thousands or millions of lines of code (in a language like Java) or tens of thousands (e.g. in Ruby), yet everyone ends up having them. And once you have one, you’re basically stuck with them: They’re incredibly hard to maintain, extend, and modernize; yet they provide value and can’t simply be replaced (something that many organizations attempt but fail at, because it’s awfully hard to create something new that is not only great in terms of architecture, but also can actually function as a full replacement for all of the old system’s features.

So what’s the proposed remedy? To talk about that, we need to take a step back and find out how we actually end up systems that are too big in the first place. My theory is that the number one reason is project scope.

When a project is started, there is an assumption that it’s the goal of a project to create a single system. This typically goes unquestioned, even though the people or person coming up with the project boundaries often don’t decide this consciously. This is most obvious if they’re non-technical people who make decisions on a budget basis.

So the very first thing we should be doing as architects (or lead developers if you don’t like the term) is to find out what it actually is we should be building. Is it really a single system? If our task is to replace an existing system with a new one, it’s very tempting to just accept existing boundaries and go with them. If we’re consolidating two systems, it’s equally tempting to view our scope as the union of the predecessor systems’ scope. In the rare cases where our task is to actually modularize something existing, it’s because of business reasons (such as deregulation). Again, while it might seem like a good idea to just accept the boundaries being suggested to us, it’s not at all clear why this should be a good idea. After all, if whoever came up with those boundaries is not an architect or developer, what makes us think they made a good choice?

In my view, the most important thing to do, then, is to find out how many systems we should be building in the first place. It may be a single one, but it may also be two, five or a dozen (though probably not more) – clearly, the decision should be made very consciously, because whatever system boundaries you pick, you will likely be stuck with them for a very long time.

As “system” is a term that can mean almost anything, I need to define what I mean by it in this context. A system is an independent unit that is developed according to its own rules, and only connected to other systems in an unobstrusive fashion. A system, according to this model, has its own database, business logic, and user interface; it’s deployed separately. It’s likely developed by a different team than other systems. It has its own life cycle, in terms of development as well as deployment. It’s operated autonomously. It has its own test suite. In basically every regard, it’s as different from all the other systems as a piece of commercial off-the-shelf software would be. (In fact, one of the systems may end up being a piece of standard software.) Is that the same as the “Micro Services” idea? If you watch James Lewis’s great talk (here’s a recording, also done at JavaZone; in fact his was scheduled directly after mine), you’ll find a lot of similarities, but the major difference is probably the size of each individual unit. But to me, seeing similar concepts appear in different contexts is a very good sign.

It doesn’t really matter that much whether you get the number and distribution right with the first attempt – in fact, you can reasonably consider that to be highly improbable. But it’s one thing to find out you should have built six or eight systems instead of seven, i.e. get it wrong in one or two places, and a completely different one to notice it should have been seven instead of one.

I’ve rambled on for long enough for a single post, so here’s a preliminary conclusion: How many systems you build should be a very conscious decision. It will affect the life of those tasked with evolving and maintaining it for years to come.

Harassed by Getty Images

| Comments

Ah, the joys of “Intellectual Property”. I’ve been a long-time fan of the wonderful demotivational posters offered by From time to time, I point people to them, e.g. by tweeting about it. In the past, when there was no Twitter (yes, that time existed), I used this blog to do so – on one occasion not only using a plain <a>, but an <img> element as well, embedding one of their images on this site (this is the post minus the image). After all, why not send a little traffic to these fine folks?

But obviously Despair uses stock photos – a perfect use case if there ever was one –, and the rights to the particularly cheesy one used in this poster apparently belong to Getty images (see? I did it again. Sue me.). Now I’ve received a letter, first from their internal legal department, and – after explaining the misunderstanding on their part – now from their lawyer. In both cases, they insist that we need to license the image to use it.

As I didn’t copy the image of the poster, but only link to it, this seems entirely absurd to me – particularly if Despair properly licensed the image, which I’m quite sure of. (If at all, Despair might have more reason, but I can’t believe they’d be that unreasonable, purely out of their own interest.) But my guess is the legal trolls at Getty believe it won’t be worth the hassle to me. They’re wrong – I don’t believe they deserve a single cent of my (or the company’s) money. If you have any advice, or want to share some of your own experience with these people, please leave a comment.

Keyboard Optimization

| Comments

For the last few months, I’ve been continuously tweaking my laptop settings to optimize my typing speed and efficiency. Here are some of the things I’ve learned, and some of the tools and approaches I’m currently using.

  • I finally took up “real” touch typing, i.e. using 10 fingers and a classical system. For years I’d been quite satisfied, even a little proud, of my awesome typing skillz, only to be shot down when I took the first real speed typing test and found myself scoring a ridiculous 30-40 words per minute, the reason being mostly mistakes and the “occasional” peeking if some not-so-common character came along. There are a ton of tools for learning touch typing. The one I spent the most time with is the excellent (and free) Tipp10, which is available in both offline and online versions. There’s also the very nicely done Keys if you’re on a Mac and have a US keyboard. If you like to have some social interaction, both 10 Fast Fingers and Typeracer are also quite nice. Finally, for a programmer, is a fantastic resource.

  • I switched to a US keyboard layout a while ago. I don’t know why I’d put this off for so long; while I originally started out using a German keyboard layout (QWERTZ instead of QWERTY), I easily write half of what I produce in English, so it wouldn’t have hurt to do this earlier. The most important benefits are that almost all of the characters required in programming languages are far easier to type using a US keyboard and all of the keyboard shortcuts, particularly in editors such as my favorite, Emacs, suddenly make a lot more sense. Because I still type a lot of German texts and hated the default input method for umlauts and the German “ß”, I installed the great “USGerman” keyboard layout, which allows me to use Option+a, u, o and s to get the appropriate characters with a single combination. This has turned out to be a very workable solution.

  • It also made me use Cmd for Meta in Emacs, which is both good and bad: It’s good because when you use Emacs, you use Meta a lot; it’s bad, because some of the keyboard shortcuts for moving around now need to be done with Cmd (in Emacs) and Option (everywhere else), which can be a bit annoying. Also, I ordered my new laptop with a German keyboard layout by accident, which ended up being great because it means I now have no chance to actually look at the keyboard for those special characters anymore. (It also allowed me to turn the keyboard lighting all the way down.)

  • Speaking of Emacs, I wanted to make sure I have the Emacs short cuts available everywhere. This is actually the case to a large degree by default on a Mac, but I wanted to be able to rely on combinations such as Ctrl-M, Ctrl-H, Ctrl-I, etc. Enter the slightly ugly, stupidly named, but incredibly powerful KeyRemap4MacBook, which allowed me to ensure Emacs keys work everywhere. (I hear this is possible for vi users, too.)

Now I find myself not taking my fingers off home row very much, and I’m comfortably moving along at 60-80 wpm. There’s a weird satisfaction in this – I noticed that in a recent company-internal discussion, many of my co-workers did not seem to see much of a point in taking up touch-typing because they don’t see typing speed as the limiting factor. One reason I might see this differenly is because I (sadly) don’t program much these days, but produce a lot of prose instead. And there, at least, I’m absolutely confident that not having your typing get in the way is a huge asset.

Why I Like Schemaless Data Structures

| Comments

Martin Fowler has written an infodeck (which is actually a nice format for these kinds of things) on schemaless databases and in-memory structures. I agree with most of it, at least as far as database design is concerned. But I believe there are two important concerns missing, and that’s the reason why I don’t agree with the conclusion. Note that my concern is not really with the database part, but with the extrapolation to in-memory data structures, so you should read the following with this in mind:

First, a major effect of using “schemaless”, simple data structures is loose(r) coupling between pieces of logic (functions) operating on them. If I pass a map to a piece of code, that code will extract what it’s interested in, possibly transforming the data in the process (ideally into a new data structure using efficient copy strategies). It will thus only depend on what it actually uses. Take the following Clojure code as an example (this would be doable similarly in Python, Ruby, Perl, Scala and even Java, although with way more boilerplate in it).

;; "projects" is a Clojure set containing three maps
(def projects #{
                {:id "1",
                 :kind :time-material,
                 :description "Consulting for BigCo",
                 :budget 25000,
                 :team [:joe, :chuck, :james]}
                {:id "2",
                 :kind :fixed-price,
                 :description "Development for Startup",
                 :budget 100000,
                 :team [:john, :chuck, :james, :bill]}
                {:id "3",
                 :kind :fixed-price,
                 :description "Clojure Training",
                 :budget 3000,
                 :team [:joe, :john]}})

;; all-members returns all team members in all projects
(defn all-members
  (reduce conj #{} (flatten (map :team projects))))

;; yields #{:chuck :joe :james :john :bill}

The code and the data structure are coupled just by one map key (:team in the example); I can add other data elements without problems as long as I maintain that contract. In languages such as Clojure, a huge library of useful functions to manipulate data rely on this fact (conj, flatten, map and reduce in this case.)

More importantly, it’s possible (and actually quite common) to use generic data structures at boundaries, such as between modules/namespaces. At their interfaces, these modules accept simple data structures, not complex object graphs. In fact this makes it a lot easier to evolve a single-process program into a distributed one: The kinds of interfaces you use internally resemble what you’d be using remotely (in fact there’s a very nice mapping between a nested Clojure, Ruby or Python data structure and e.g. JSON.)

Some languages, such as Clojure, take this to an extreme: Almost everywhere you’d create a new class in an OO language, you’d just use a simple data structure, such as map, a vector, set or list. In a statically typed limited OO language such as Java, you will almost always end up creating new classes. Of course you can use Map in Java, too, but it would not be idiomatic (and the reverse is true for Clojure as well, which enables you to create “normal” Java classes, too.) Some languages are somewhere in the middle, as shown by the Ruby code in Martin’s example. But I find it not very useful to favor one style over the other by default, unless you consider the language you’re using.

Secondly, the use of custom-built data structures – and that’s what a schema boils down to if viewed from a programming language standpoint – always means additional code. You might consider this irrelevant, but it’s nicely shown in interfaces such as those of WSGI, Rack or Ring when you compare them to their equivalent in Java: A simple map containing a method or response code, headers and a body vs. different concrete classes for requests with tons of specific attributes (and bonus getters and setters). Restricting the use of schemaless design to the cases where you actually need dynamic variability misses this advantage.

In summary, I think of Martin’s points are valid, and he definitely spent more time on articulating his conclusion than I did in writing this comment. But I think those two aspects – coupling and verbosity – are worth considering when you need to decide which approach to use.

New Languages

| Comments

Being a programming language geek, I typically try to use the Christmas vacation to learn (or rather, play with) a programming language I don’t know. This year, I find this very hard, as there are so many interesting languages one could spend time with. Some I have considered are:

  • Go: I was thoroughly unimpressed with this language when it came out, and I still fail to see a lot of interesting stuff in it. But I’ve heard many people I respect say only good things about their experience with it, so maybe I should reconsider.
  • Rust: At first glance, this seems to be a very nicely designed language (and it has a really excellent tutorial). Even though its language features are very advanced, it seems to be intended for low-level use cases (that I mostly don’t have).
  • Fantom: Seems to be interesting, too; I remember I looked at it a long time ago, but never in depth.

What do you think? What else is worth a look?

Some Thoughts on SPDY

| Comments

What follows is a number of claims about SPDY, none of which I can back up in any reasonable way. My hope is that readers who know more will educate me. SPDY, in case you don’t know it, is a replacement for the HTTP wire protocol. It originated with Google and aims to preserve HTTP’s application semantics while improving its efficiency and performance.

  • Supporting true multiplexing on top of a single TCP connection is great. There is no way anybody can prefer the HTTP 1.0 model, which forces a separate TCP connection per request, nor the HTTP 1.1 model, which allows for persistent connections but still requires serialized request/response interactions (never mind HTTP pipelining as it doesn’t work in practice). Browsers having to open separate connections to the same server to achieve parallelism is not a satisfactory solution.
  • Reducing header overhead is also an excellent idea. I’ve heard some criticism about the way this is actually done in SPDY, but it very clearly serves no purpose to have a browser send e.g. the same ridiculously long user agent string with each and every request.
  • I used to not care much for the push support, i.e. the opportunity for the server to actively send stuff to the client, for the same reason I’m not a fan of Websockets: I don’t think you actually need this in practice on the application level. But in a session done by Jetty developer Thomas Becker today, I learned about a quite intriguing usage of this in Jetty’s SPDY implementation: On the first request of a page and the subsequent request for the resources that are referenced in that page, Jetty will build a map based on the Referer header – it essentially remembers which secondary resources a page references. When the next request comes along, it can actively send the referenced resources before the client actually asks for them.
  • I think the fact that SPDY requires TLS is a mistake. While I totally concede that most traffic on the Net is (and should be) encrypted, there are many use cases e.g. within an organization or for public information where this does not make sense. Besides, it prevents the usage of intermediaries, even though I admit these will be much harder to build for SPDY than for plain HTTP anyway.
  • While SPDY proponents point to impressive performance improvements, they are the more impressive the worse the website is implemented. For sites that are already optimized in terms of front end performance, e.g. minimize and compress content, minimize the number of requests, usage proper caching, the effect is going to be much less. That said, some of the things we do in terms of optimization, e.g. combining multiple CSS or JS files into a single one, are not exactly milestones of architectural purity.
  • For machine-to-machine communication – i.e. typical RESTful web services – I don’t think SPDY will have the same kind of effect as for Web sites, but I’m willing to let myself be convinced otherwise.
  • One of the sad but inevitable things when introducing a binary protocol as opposed to a text-based one is reduced transparency for humans. If SPDY becomes successful – and I have small doubts it will – being able to telnet to a servers port 80 is going to be what I miss most.

SPDY has a very good chance of essentially becoming the new HTTP 2.0, and I’m happy about it: I’m pretty confident the HTTP WG with the formidable Mark Nottingham taking care of things will produce something that will be usable for a long time to come.

innoQ Company Events

| Comments

Since the beginning of innoQ’s existence, 13 years ago, we’ve maintained a regular schedule of so-called “company events”. In my opinion, this is one of the really, really great things about innoQ, and it’s also quite different from what others do. Which is a sufficient excuse for me to write this …

So what’s an innoQ event? All of innoQ’s consultants meet at some more or less remote venue and spend two or three days there, discussing projects, technology, methods, in general: anything of interest to our work. Most of the time we use the classical conference format (an innoQ guy presents something, followed by sometimes controversial discussion), but we use other approaches, such as open spaces, pecha kuchas, and lightning talks, too. We occasionally invite guests (we were lucky to have e.g. Steve Vinoski, Michael Hunger, Markus Voelter, Michael Nygard pay us a visit). While the location is mostly somewhere in Germany, we go to Switzerland sometimes, and one event per year is reserved for a trip “far far away” (in the past years we went to e.g. Prague, Barcelona, Paris, Rome, Budapest, and Strasbourg; these are the only events where we actually spend a day just sightseeing). Some of the events focus on project reports, others are programming events, one event per year is dedicated to company strategy.

What is amazing to most people I talk to about this is the frequency we do this with, and the resulting amount of time, effort and money we invest. We do 6-8 events per year, 2 of them three days long, the rest two days. Events are always scheduled during regular workdays, typically Thursdays and Fridays; attendance is mandatory. This adds up to 15-18 days per person per year, with the most significant cost factor being the “lost” revenue. Of course there’s also a lot of effort involved in organizing the whole thing: The colleague who does this essentially organizes 6-8 small conferences (we’re regularly about 50 people these days) per year (no small feat; thanks, Thomas).

It’s worth every single cent.

Company events are among the very best things about innoQ. They serve to help us to spend some quality time discussing various topics, whether it’s company strategy, a new programming language, library, framework or product, a new approach to project management, or some very project-specific problem. We’re also able to invite candidate employees to a setting where they have a great chance to get to know how innoQ works.

Most importantly of all, they’re fun. We spend a lot of time doing geek things, but there’s always time for great food, occasional drinks, and socializing and talking about other important things in life.

So if you see me tweet from Barcelona during the next three days (where I plan to spend some time with the works of one of my favorite artists tomorrow), you know why I’m there.

Hypermedia Benefits for M2M Communication

| Comments

Hypermedia is the topic that generates the most intensive discussions in any REST training, workshop, or project. While the other aspects of RESTful HTTP usage are pretty straightforward to grasp and hard to argue with, the use of hypermedia in machine-to-machine communication is a bit tougher to explain.

If you are already convinced that REST is the right architectural style for you, you can probably stop reading and switch over to Mike Amundsen’s excellent “H Factor” discussions. That may be a bit tough to start with, though, so I thought it might make sense to explain some of the ways I use to “pitch” hypermedia. I’ve arrived at a number of explanations and examples of meaningful hypermedia usage, explicitly targeted at people who are not deep into REST yet:

  • “Service Document” and “Registries”: Especially for people with an SOA/SOAP/WSDL/WS-* background, the idea of putting at least one level of indirection between a client (consumer) and server (provider) is well established. A link in a server response is a way for a client to not couple itself to a particular server address, and a server response including multiple links to “entry point” resources is somewhat similar to a small registry. If providing links to actual entry point resources is the only purpose of a resource, I’ve become used to calling it a “service document”. Of course these documents themselves can be linked to each other, allowing for hierarchies or other relationships; they can support queries that return a subset of services; they can be provided by different servers themselves; and they are way more dynamic than a typical registry setup. In other words, the very simple approach included in HTTP is far more powerful than what most crazily expensive registries provide.
  • Extensible contracts: The merits of being able to link to resources can be exploited to add functionality to existing systems without breaking their contracts. This is most visible in server responses that have one or more places where you can put additional links. As your server or service evolves, you can add links that will be ignored by existing clients, but can be used by those that rely on them. The concept of “a link to a resource” is both sufficiently generic and specific to be meaningful enough, especially if you include a way to specify what they actually mean via a link rel=… approach (but more on that in a separate post).
  • Co-Location Independence: What I mean by this slightly weird term is the fact that while resources that are exposed as part of a service interface are sometimes (or maybe even often) designed in a way that requires them to be part of the same implementation, they very often are not, i. e. they could at least in theory be part of a different system. (In fact you can reasonably argue that there should be no assumption about this at all, neither on the server nor the client side for something to be rightfully called “RESTful”, but I simply haven’t found that to be doable in practice.) In those cases where resource don’t need to be hosted by the same implementation, you can and should couple them via links and have clients navigate them instead of relying on common URI paths.

There are quite a few more examples to talk about, but I won’t do that now as I promised to publish something today and don’t want to get into the habit of keeping drafts lying around for too long again. (I know, lenghty this is probably not. Sue me.) So please let me know in the comments what you think of these three if you’re just starting to pick up REST, and what additional ways of explanations you use if you already have done so.

Waterfall Sucks, Always. Duh.

| Comments

Today, I had an interesting discussion over Twitter related to project organization in restricted environments. (Update: I’ve removed all references to the actual discussion because the person I interacted with felt mis-quoted, and I don’t think that it was actually that important with regards to what I actually wanted to get across.) This prompted me to take a break from my usual topics and elaborate a bit on my thoughts with more than 140 characters. All this assumes you’re in the role of not only having to actually deliver a piece of software, but also to get the necessary funding – regardless of whether you’re part of an internal organization that requires approval from top management or you’re an external partner that needs to sell to its customer. That said, I’ll focus on the second scenario as that is my primary focus at innoQ.

First of all, in an ideal world, the customer understands all of the reasons that led to the Agile movement, e.g. accepts that an upfront specification that’s 100% correct is an unattainable pipe dream, agrees to participate in the actual development process, and most importantly, understands that what needs to be built will only become clear during the actual development project. We do have some customers who understand this very clearly, and they agree to our favorite pricing model: We offer a sprint/iteration for a fixed price or on a T&M basis, and after each sprint the customer can decide to continue or to stop (which will require one final short wrap-up phase). This reduces the customer’s risk, which is often seen as a benefit big enough to outweigh the perceived disadvantage of not knowing what the overall cost will be. It’s great to be able to work in an environment where everybody’s goals are perfectly aligned, and this is the case in this model.

Unfortunately, this ideal model is not always an option. Of course one way for a development organization to ensure that all projects are done this way is to simply refuse doing it in any other fashion. That’s a good option, but whether it’s actually doable strongly depends on internal power distribution or external market forces.

But what do you do when you have to accept a different set of restrictions? For example, the customer/stakeholder might require a fixed-scope, fixed-time, fixed-price offer. My guess is we can all agree that this is bad idea for everyone involved. But how do you approach things if you just have to do things this way? What do you do if, as an additional downside, the developers assigned to the project are not as skilled as you’d like the to be?

As possible answer might be to use a classical waterfall approach, but I think this is never a good choice. At the very least, go with an iterative approach, even if that means you have to game the system to do that.

Of course you have to put up some effort into an initial up-front analysis. You’ll be aware that much of what you find out may actually turn out to be wrong, but it’s still better to make a slightly more informed estimate up front as opposed to a totally bogus one, especially if you’re an external partner that’s supposed to provide a fixed-price quote. Then, make sure that you grow the system in increments – i.e., build a first working system, using a number of interesting use cases; then add functionality in the next iteration, and continue until done.

Typically, this will resemble something like an agile process – but with slightly larger iterations (e.g. maybe 6 weeks instead of two), and with the added amount of documentation required to fulfill the typical waterfall requirements. (If this reminds you of a Unified Process kind of thing, that’s no coincidence.)

In the end, you’ll have created all of the documents and other artefacts required, but simply not in the order they were supposed to be generated (first analysis, then design, then implementation, then test), but with the trimmed-down focus of each iteration.

Is this perfect? Not even remotely. But in my experience, you have a far greater chance to meet your goals than with actually following the waterfall approach, and even more importantly, management is likely to accept it (partially because it’s obvious, partially because you don’t tell them about it).

If you can’t get away with that, you’re really out of luck, and it’s as they say: You need to change the company, and if you can’t, change companies.

Announcing “ROCA”

| Comments

In the past few days, we finally managed to write down a few ideas on Web frontend design, basically a set of rules to apply if the goal is to come up with a Web app that is actually on the Web as opposed to be tunnelled through the Web. We tried to come up with a catchy name, and finally arrived at ”ROCA”, a conveniently pronouncable acronym for “Resource-oriented client architecture”.

I am aware that for many folks, specifically those who are interested in REST and thus likely to read this, a common reaction might be “Duh”. And one of the main things I’d like to stress is that we have not invented a single thing, but rather collected a certain set of rules that we found many people liked, but couldn’t name.

Since we started discussing this, we’ve found strong support, as well as violent opposition. Which is exactly what we were looking for, because in only very very few cases, people didn’t understand what we described, and that’s the whole point of the effort: Give a name to a certain cohesive set of practices so that they may used as a reference both when you agree with them, want to build a system that adheres to them or criticize them because you disagree.

I’m looking forward to comments over at the discussion page. If you believe we should make changes, please fork the GitHub repo and create a pull request.