–

Next, Michael Tamm presents “Fighting Layout Bugs”, a library to detect layout bugs.  This is something that will save a lot of time and prevent many layout bugs by bringing an unprecedented level of automation to testing page rendering.  This is something to keep an eye on.

–

Let’s talk about model-based testing.  Mark Utting gives an introduction.

Then, Atif Memon discusses applying model-based testing to web applications.

Finally, Darko Marinov talks about test selection when conducting model-based testing.

Before watching this talk, I suggest brushing up on the following terms if you’re not already familiar with them: domain model, aggregate root and ubiquitous language. (In one system I work on, we use the words “session” and “ticket” to describe the same concept.  Precise, consistent and accurate language leads to better communication and understanding.)

Click here to watch the talk.

Here are some valuable lessons I see in Greg’s talk:

• Software runs.  It’s about action.  Most models I’ve seen focus on nouns, such as customer and order, and make actions, such as place order and cancel order, secondary concepts.  A functional design focuses the model on actions, not aggregate roots.  Software with a focus on action makes testing and auditing, which are fundamentally about verifying action, natural.

• Think of the state of an aggregate root as the result of a stream of events.  As Greg mentioned, Martin Fowler dubbed this design Event Sourcing.  Distributed source control systems focus on changes to a file rather than versions of files.  This subtle difference is a behind much of the value distributed source control systems offer.  Greg talks about how building your state from a stream of events can result in the ability to rewind and playback history.  To see how this might be visualized, check out the playback feature of Google Wave, demonstrated at 49 minutes into this video.As a bonus, since a stream is your data, you can easily start writing code in a functional style, even in Java.  Sequences are the core interface to data for Clojure and LINQ.Also, Greg mentioned the the ability to use a document database.  These are the kinds of databases that power Google and Amazon.

• Look for places to implement eventual consistency.  Here’s an example of an opportunity for eventual consistency:  You want to provide the ability to mark a blog post as a favorite.  Here’s a series of steps to implement eventual consistency: You have an application responsible for rendering the blog post and if it has already been marked as a favorite.  User clicks “mark as favorite.”  This triggers a call to a different application which accepts the command, validates it, inserts the command into a queue to update the read-only data stores and response to the client that the command has been accepted.  The UI then displays the item has been marked as a favorite.  If the user refreshes the page before all read-only data stores have been updated, then they may see the blog post as not marked as a favorite.  As Greg would say, so what?  This is unlikely to happen and the consequences are small compared to scalability gained by making this operation asynchronous.

• Greg talks about designing business processes as if they were to be done with paper: “Paper was awesome because it never gave the impression of global consistency and it is the thinking that went into the optimization of paper processes that can help us optimize our transactional systems.“  I thought his restaurant example in the talk might not click with some viewers, so I’m expanding on it here.  When you place an order at a restaurant, you’ve inserted a command in a queue.  The waitress transfers the message from the queue she’s holding to a queue for the kitchen.  The kitchen then prepares your order and notifies the waitress of the event that the order is ready to be served.  The kitchen may then decrement the inventory records.  The waitress then picks up the order and serves it to you.  The reason why this not a fully consistent state is that the cook knows about your order after you’ve placed it.  The inventory is decremented after the food is cooked.  It’s eventually consistent.  And it works.As a slight tangent, human processes are often automated because automation is thought to be the solution to poorly executed processes.  Strive to first standardize processes before automation as the process is often ill-defined.  Sometimes, simply defining a process cures problems.  From The Toyota Way Fieldbook: “Standardization is a key element of the Toyota system. A process that is not standardized is fraught with chaos, variation, and the associated problems of continually “riding the wave.” According to Toyota, standardization is the baseline for continuous improvement, the time when real improvement begins and is measurable.”

As a final note, one could benefit from applying one or some of the ideas in this talk.

Related links:

• Domain-Driven Design: Tackling Complexity in the Heart of Software by Eric Evans

• Bertrand Meyer talks about command and query separation in his book Object Oriented Software Construction.

• Bounded context
• Behavior Driven Development or BDD
• Given, When, Then

“Man acts as though he were the shaper and master of language, while in fact language remains the master of man.” -  Martin Heidegger

Update Sept 2, 2009: I received feedback that such an architecture is only for mission critical systems. In his talk, Buy a Feature: An Adventure in Immutability and Actor, David Pollack, the creator of the Lift web framework for Scala, talks about an application, Buy a Feature, which implements this architecture.

In his talk, Unit Testing, Miško Hevery explains unit testing and makes a case for unit tests.

…

In his talk, Don’t look for things, Miško Hevery provides a practical guide to the Principle of Least Knowledge (aka Law of Demeter, aka Don’t Ask, Tell)

…

In his talk, Global State and Singletons, Miško Hevery explains why global state leads to bad APIs and difficult to test code.

…

In hist talk, Inheritance, Polymorphism, & Testing, Miško Hevery presents his premise that most if statements can be replaced with polymorphism and explains why one would want to do so.

…

In his talk, GuiceBerry, Luiz-Otavio Zorzella makes a case for using dependency injection in your test code

XMPP addresses many complexities of inter-node communication in a distributed application, such as presence, heartbeat, queuing messages for offline nodes (offline messages), group broadcast (chat rooms), and status.

Here some other resources promoting instant messaging as a communication bus for distributed applications:

• Mickaël Rémond’s blog talks about using XMPP (Jabber) inside distributed Erlang applications. Here are slides from Mickaël Rémond’s presentation entitled Messaging with Erlang and Jabber.
• A proposed standard for game sessions using XMPP (Jabber) by by Michal Vaner
• A proposed universal Go server using XMPP (Jabber) by Mike Albon
• Peer-to-Peer: Building Secure, Scalable, and Manageable Networks by Dana Moore and John Hebeler – book on Amazon
• Jabber Developer’s Handbook by William Wright and Dana Moore – book on Amazon

Related links:

• Lightweight Languages 1
• Lightweight Languages 2
• Lightweight Languages 4
• Event-Based Programming: Taking Events to the Limit by Ted Faison – book on Amazon
• IM Instant Messaging Security by John Rittinghouse – book on Amazon
• Distributed Computing: Principles, Algorithms, and Systems by Ajay D. Kshemkalyani and Mukesh Singhal – book on Amazon

Related links:

• Nexus: Small Worlds and the Groundbreaking Theory of Networks by Mark Buchanan – book on Amazon
• Linked: How Everything Is Connected to Everything Else and What It Means by Albert-Laszlo Barabasi – book on Amazon

Related links:

• Why Programs Fail: A Guide to Systematic Debugging by Andreas Zeller – book on Amazon

I enjoyed the part about defaults.  Simple tips for improving concentration:

• Configure your email client to not notify you when an email arrives.
• Configure your instant messenger to not notify you when anyone changes status.
• Change your browser’s home page to be something simple, something that does not display recent news, quizzes, weather, traffic, recent emails in a personal email account, recent blog posts. I prefer the fastest-loading option: about:blank, and I use Firefox’s built-in search.

I recently saw Merlin Mann speak at a OmniFocus meeting hosted by The Omni Group.

Related links:

• Getting Things Done: The Art Of Stress-Free Productivity by David Allen – audiobook on Amazon

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

void process(final List<Foo> foos) {
    final ArrayList<bar> batch = new ArrayList<bar>(batchSize);
 
    for (final Foo foo : foos) {
        if (isValid(foo)) {
            final Bar transformed = transform(foo);
            batch.add(transformed);
            if (batch.size() % batchSize == 0) {
                processBatch(batch);
                batch.clear();
            }
        }
    }
    processBatch(batch);
}
 
// returns true if we want to process this Foo, otherwise returns false
private boolean isValid(final Foo foo) {
    return true;
}
 
// makes a Bar out of a Foo
private Bar transform(final Foo foo) {
    return null;
}
 
private void processBatch(final ArrayList<Bar> bars) {
    openTransaction();
    for(final Bar bar : bars) {
        executeQuery(bar);
    }
    closeTransaction();
}

We have some code duplication: processBatch appears twice in process. Also, we’re iterating over two collections in order to process each Bar: we’re iterating over foos in process and bars in processBatch. With the help of Google Collections, we can do better:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

void process(final List<Foo> foos) {
    final Iterator<Foo> validFoos = Iterators.filter(foos.iterator(), isValid);
    final Iterator<Bar> bars = Iterators.transform(validFoos, transform);
    final Iterator<Iterator<Bar>> batches = Iterators.partition(bars, batchSize, false);
 
    while(batches.hasNext()) {
        final Iterator batch = batches.next();
        processBatch(batch);
    }
}
 
private final Predicate isValid = new Predicate() {
    // returns true if we want to process this Foo, otherwise returns false
    public boolean apply(final Foo foo) {
        return true;
    }
};
 
private final Function transform = new Function() {
    // makes a Bar out of a Foo
    public Bar apply(final Foo foo) {
        return null;
    }
};
 
private void processBatch(Iterator<Bar> bars) {
    openTransaction();
    while(bars.hasNext()) {
        final Bar bar = batch.next();
        executeQuery(bar);
    }
    closeTransactioin();
}

You may want to look at the Javadoc for the Iterators class in Google Collections to better understand what’s happening here.

We removed the duplicate code. And while we still have two loops, one in process and one in processBatch, these loops are iterating over the same collection: foos. In addition, process has fewer lines of code. Granted, we have more lines of code elsewhere, but a real process method is likely to be more complex. By reducing the number of code blocks (braces) in process, we’re making it easier to read and maintain.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

/**
 * Removes duplicates from the elements returned by provided iterator.
 * The provided iterator must return the elements in the order elements in the order defined by the comparator.
 * The first element of a series of duplicates is returned by this iterator.
 * @param  Type of element returned by the iterator.
 */
public final class DedupingOrderedCollectionIterator<E> implements Iterator<E> {
    private final PeekingIterator<E> iterator;
    private final Comparator<? super E> comparator;
 
    /**
     * @param iterator iterates over an ordered collection
     * @param comparator implements the same comparison used to order the collection
     */
    public DedupingOrderedCollectionIterator(final Iterator<? extends E> iterator, final Comparator<? super E> comparator) {
        this.iterator = Iterators.peekingIterator(iterator);
        this.comparator = comparator;
    }
 
    public boolean hasNext() {
        return iterator.hasNext();
    }
 
    public E next() {
        final E next = iterator.next();
        while (iterator.hasNext() && comparator.compare(next, iterator.peek()) == 0) {
            iterator.next();
        }
        return next;
    }
 
    public void remove() {
        throw new UnsupportedOperationException();
    }
}

First, if you’re confused about the generic wildcards, watch Josh Bloch’s presentation entitled Effective Java Reloaded and watch for the reference of “producer extends, consumer super” (aka PECS) or reference Josh Bloch’s second edition of Effective Java.

Second, using Google’s PeekingIterator often reduces the code you need to deal with iterators.

Third, when implementing an iterator, you usually have to worry about what happens when next() is called when hasNext() would return false. What exception are you supposed to throw? And I keep coding the same implementation for remove() and always forget what exception it’s supposed to throw. Google’s AbstractIterator to the rescue!

Here’s the same iterator written with com.google.common.collect.AbstractIterator<T>:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

/**
 * Removes duplicates from the elements returned by provided iterator.
 * The provided iterator must return the elements in the order elements in the order defined by the comparator.
 * The first element of a series of duplicates is returned by this iterator.
 * @param  Type of element returned by the iterator.
 */
public final class DedupingOrderedCollectionIterator<E> extends AbstractIterator<E> {
    private final PeekingIterator<E> iterator;
    private final Comparator<? super E> comparator;
 
    /**
     * @param iterator iterates over an ordered collection
     * @param comparator implements the same comparison used to order the collection
     */
    public DedupingOrderedCollectionIterator(final Iterator<? extends E> iterator, final Comparator<? super E> comparator) {
        this.iterator = Iterators.peekingIterator(iterator);
        this.comparator = comparator;
    }
 
    protected E computeNext() {
        if (iterator.hasNext()) {
            final E next = iterator.next();
            while (iterator.hasNext() && comparator.compare(next, iterator.peek()) == 0) {
                iterator.next();
            }
            return next;
        }
        return endOfData();
    }
}

AbstractIterator allows you to write concise iterators with consistent behavior.

• Flow of Change which discusses managing change in source control systems
  Speaker: Tony Smith, European Technical Services Manager for Perforce Software
• The Truck Driving Problem which discusses developing innovation
  Speaker: Mary Poppendieck, author of Lean Software Development: An Agile Toolkit

1
2
3
4
5
6
7
8
9

[Test]
public void DivideByZeroThrowsException()
{
  Assert.Throws<System.DivideByZeroException>(
    delegate
    {
      DivideNumbers(5, 0);
    });
}

This code snippet was taken from xUnit.net’s documentation.

Previously, each test method could have an expected exception. Now, one can have multiple calls which are expected to throw an exception in a single test method. Good thinking!

In his talk, Searching for Evil, Professor Ross Anderson discusses research done by himself, Dr. Richard Clayton, Tyler Moore, Steven Murdoch, and Shishir Nagaraja.

Click here to view the embedded video.

Related links:

• Security Engineering: A Guide to Building Dependable Distributed Systems by Ross J. Anderson – book on Amazon
  

I would like to see compile-time and full IDE support for design by contact.

Related links:

• Design by Contract, by Example by Richard Mitchell and Jim McKim – book on Amazon
• The Practical Guide to Defect Prevention by Marc McDonald, Robert Musson, and Ross Smith – book on Amazon

• Languag-Oriented Programming and Language Workbenches
• Scala is a functional and object-ortiented language that runs on the JVM, it’s similar to F# (F sharp)
  
• Closures for Java
  
• Fast x86 Emulation in Java This reminds me of the CLI back-end for GCC
  

Architecture:

• To Know the Dependencies is to Know the Architecture – Dependency Structure Matrix
  
• OSGi Best Practices
  
• Advanced Spring Framework – Spring OSGi
  
• Extreme Web Caching

Concurrency:

• High-Performance Java in a Multicore World – memory latency matters
  
• Scaling Up a Real Application on Azul
  

Related links:

• Practical Model-Based Testing: A Tools Approach by Mark Utting and Bruno Legeard – book on Amazon

Related links:

• The Future of Reputation: Gossip, Rumor, and Privacy on the Internet by Daniel J. Solove – book on Amazon

There’s a Java applet version of Dasher, but I encourage you to try a native version.

Contributions to Dasher have been made by David MacKay, David Ward, Alan Blackwell, Phil Cowans, Tim Hospedales, Iain Murray, Hanna Wallach, Matthew Garrett, Chris Ball, Keith Vertanen, Tadashi Kaburagi, Brian Williams, Chris Hack, Ingrid Jendrzejewski, Frederik Eaton, and Piotr Zielinski.

Related links:

• Information Theory, Inference & Learning Algorithms by David J. C. MacKay – book on Amazon

Also, Amazon found every 100ms of latency cost them 1% in sales and Wikia found that faster pages reduces exit rate.

Related links:

• Getting Things Done: The Art of Stress-Free Productivity by David Allen – book on Amazon
• Lifehacker: 88 Tech Tricks to Turbocharge Your Day by Gina Trapani – book on Amazon

Related links:

• The same talk given at Stanford Law School
• QuestionCopyright.org

instead of:
“C:\\directory\\file.xml”
or
“SELECT field FROM table where field=var”

you can do:
@”C:\directory\file.xml”
and
@”SELECT
field
FROM
table
WHERE
field = var”

Stream Upload(string filename)

closing the stream would just end the file upload, not the network connection.  And I want multiple connections and async methods.

Ideally, you would access the data via an API or a feed. Google, Amazon, Flickr and many other sites provide free programmatic access to data. Sites like Ebates offer feeds of data for free. If your data is not available for free, you may be able to purchase access to an API or feed.

Screen scraping can be used well for other purposes, e.g. testing the UI layer of a web application. In this application of screen scraping, the same organization generates and tests the UI.

What you really want is to be inside these poker web sites. You want to be able to be a sort of plug-in for your customers. I suggest taking a look at the Book Burro extension for Firefox, then visit Amazon. Book Burro adds an interface widget on top of the Amazon page with prices from other bookstores on the web. This is a cross-platform solution.

I’ve read that the poker sites aren’t much bothered by ‘poker bots’.  As long as people gamble, they’re happy. Maybe you could find a poker site that would let you plug in your software… or make your own site for all the users of your software to play on and they think they’re playing other people. A game server, now that’s a completely different kind of software.

That’s bad.

This way, you can send the headers and navigation to the client so they’ll see something while you’re waiting for something to finish. Perhaps the client won’t even finish downloading the headers when the render completes, you’ve gotten the first byte out faster.

Microsoft Word has autosave.  By default, every ten minutes the document you’re editing is persisted to disk in case of an unexpected termination of the application.  I always change this to one minute, because why would you ever want to wait ten?

And why doesn’t New Egg’s shopping cart remember what I had in it?  I’ve had business people tell me customers get angry when they see items in their cart many months later.  I argue, at least have it be an option to review.  Like an automated (private) wish list.

Suppose you have a web server produce views of product information that is expensive to acquire. Perhaps it’s over a web service or a remote database.  Often, you’ll cache this information in memory, trading aging data for faster request processing.  But what if there’s a power outage, and both the web server and database suddenly shut down.  If your application starts running before the web service or database, then your application is missing an opportunity to process requests. If you had stale data, you could still provide information to anxious customers.  Even if you didn’t want to show old quantity levels or prices, you could still provide a description, and maybe a notice that you’ll be fully operational in a moment.

Consider dumping in-memory state, such as a cache, to disk if the connection to the data source is lost or during low load.