David Leston's
Software Development Blog

The next big language

My opinion on the next big language:

First, let’s talk about loosely-typed vs strongly-typed languages. Also, people argue that a string type system gives you a false sense of correctness, and you have to write tests anyway, so the types are just getting in your way. The problem, I believe, lies with the majority of software developers.  The majority of developers bang on code until they think it works. This absolutely isn’t the best way to go about creating software, but it’s the most common way. For this majority, a type system helps development speed and correctness. And for the majority who aren’t going to write comprehensive tests, static analysis is a valuable tool.  And while, in theory, static analysis can be just as effective on loosely-typed code, the most effective static analysis tools have historically been available only for strongly-typed languages. I’ve heard people describe Java’s type system as clunky. It’s important to keep in mind that Java’s type system is not the definition of a strongly-typed system.

Scala attempts to improve on Java’s type system. For people who are into languages, Scala is fun. There’s a lot of language to play with – which is its biggest barrier to becoming the next mainstream language. There’s more concepts to think about, more syntax and symbols. Version 2.8 of Scala cleans up many inconsistencies. I’m hoping that version 3 will shrink the language.

Speaking of consistency and shrinking a language, this brings me to Clojure. In contrast to Scala, Clojure has little syntax. In contrast to Common Lisp, Clojure provides consistent interfaces for data structures. I very much enjoy Clojure’s model for dealing with concurrency. Clojure’s challenge to becoming a mainstream language is that of every Lisp: for the masses, it’s too different of way of programming.

Java was explicitly designed to be easy for C programmers to adopt. I think the next big language will be one that is explicitly designed for Java developers to adopt, even if that language may not technically be the best general-purpose language at the time. I’d argue that the easiest of the alternative language to transition to is Groovy. Groovy is essentially Java but with a few wish-list items implemented, such as closures.  But I’d argue that Groovy doesn’t differentiate itself enough from Java to incite large-scale adoption – most would look at Groovy and say Java is good enough.

I quite like C#, especially it’s support for declarative, composable, lazy queries a la LINQ. I also like type-safe function pointers, first-class properties and events, and the tooling is the best I’ve seen. The fact that the primary implementation of C# is restricted to running on the Microsoft stack is its major challenge for adoption. Yes, the Mono and Silverlight implementations may become solid enough for wide-spread adoption, but most people view any Microsoft technology as too risky to be used as a foundation.

I recently spoke to someone working in the ESB space who observed a cyclic patterns of the industry moving between a dominant general-purpose language to many general-purpose languages. Today, a most developers use many languages: the various XML configuration languages (Ant, Spring, Hibernate), HTML, CSS, shell scripts. So the thought that using multiple general-purpose languages is certainly a possibility, especially with interfaces between systems now commonly being defined in terms of language-independent REST, HTTP and XML. There’s much being said for polyglot organizations, but my guess that most businesses will stick with Java until at least a few years after it’s generally accepted that some other single language is the best to use. And with projects like Project Coin, Java probably has a decade longer as the dominant language.

–

Here are some videos that played a role in the formation of my opinion:

Comparing Clojure to previous Lisps and Haskall

How Clojure makes immutability fast and an introduction to how it manages concurrency

Practical Scala examples

An argument for polyglot porogramming

Programming functionally with a non-functional language

Posted in Other | No Comments »

Prototyping, fighting layout bugs, and model-based testing

Recently, I’ve ran into a few interesting talks related to testing.  In this first talk, Steven Dow talks about some early studies on how prototyping has an effect on the performance of the end result.  As you’ll see in the Q&A, people are hungry to know more about how we can structure tests for our ideas.

–

Next, Michael Tamm presents “Fighting Layout Bugs”, a library to detect layout bugs.  This is something that will save a lot of time and prevent many layout bugs by bringing an unprecedented level of automation to testing page rendering.  This is something to keep an eye on.

–

Let’s talk about model-based testing.  Mark Utting gives an introduction.

Then, Atif Memon discusses applying model-based testing to web applications.

Finally, Darko Marinov talks about test selection when conducting model-based testing.

Posted in Other | No Comments »

Architecting for scalability and audit logs by explicitly modeling state transitions

In his talk, Unsleash Your Domain, Greg Young presents a dense discussion of topics about which I am passionate.  At its core, the talk is about how to guarantee a correct audit log and architect for scalability.

Before watching this talk, I suggest brushing up on the following terms if you’re not already familiar with them: domain model, aggregate root and ubiquitous language. (In one system I work on, we use the words “session” and “ticket” to describe the same concept.  Precise, consistent and accurate language leads to better communication and understanding.)

Click here to watch the talk.

Here are some valuable lessons I see in Greg’s talk:

• Software runs.  It’s about action.  Most models I’ve seen focus on nouns, such as customer and order, and make actions, such as place order and cancel order, secondary concepts.  A functional design focuses the model on actions, not aggregate roots.  Software with a focus on action makes testing and auditing, which are fundamentally about verifying action, natural.

• Think of the state of an aggregate root as the result of a stream of events.  As Greg mentioned, Martin Fowler dubbed this design Event Sourcing.  Distributed source control systems focus on changes to a file rather than versions of files.  This subtle difference is a behind much of the value distributed source control systems offer.  Greg talks about how building your state from a stream of events can result in the ability to rewind and playback history.  To see how this might be visualized, check out the playback feature of Google Wave, demonstrated at 49 minutes into this video.As a bonus, since a stream is your data, you can easily start writing code in a functional style, even in Java.  Sequences are the core interface to data for Clojure and LINQ.Also, Greg mentioned the the ability to use a document database.  These are the kinds of databases that power Google and Amazon.

• Look for places to implement eventual consistency.  Here’s an example of an opportunity for eventual consistency:  You want to provide the ability to mark a blog post as a favorite.  Here’s a series of steps to implement eventual consistency: You have an application responsible for rendering the blog post and if it has already been marked as a favorite.  User clicks “mark as favorite.”  This triggers a call to a different application which accepts the command, validates it, inserts the command into a queue to update the read-only data stores and response to the client that the command has been accepted.  The UI then displays the item has been marked as a favorite.  If the user refreshes the page before all read-only data stores have been updated, then they may see the blog post as not marked as a favorite.  As Greg would say, so what?  This is unlikely to happen and the consequences are small compared to scalability gained by making this operation asynchronous.

• Greg talks about designing business processes as if they were to be done with paper: “Paper was awesome because it never gave the impression of global consistency and it is the thinking that went into the optimization of paper processes that can help us optimize our transactional systems.“  I thought his restaurant example in the talk might not click with some viewers, so I’m expanding on it here.  When you place an order at a restaurant, you’ve inserted a command in a queue.  The waitress transfers the message from the queue she’s holding to a queue for the kitchen.  The kitchen then prepares your order and notifies the waitress of the event that the order is ready to be served.  The kitchen may then decrement the inventory records.  The waitress then picks up the order and serves it to you.  The reason why this not a fully consistent state is that the cook knows about your order after you’ve placed it.  The inventory is decremented after the food is cooked.  It’s eventually consistent.  And it works.As a slight tangent, human processes are often automated because automation is thought to be the solution to poorly executed processes.  Strive to first standardize processes before automation as the process is often ill-defined.  Sometimes, simply defining a process cures problems.  From The Toyota Way Fieldbook: “Standardization is a key element of the Toyota system. A process that is not standardized is fraught with chaos, variation, and the associated problems of continually “riding the wave.” According to Toyota, standardization is the baseline for continuous improvement, the time when real improvement begins and is measurable.”

As a final note, one could benefit from applying one or some of the ideas in this talk.

Related links:

• Domain-Driven Design: Tackling Complexity in the Heart of Software by Eric Evans

• Bertrand Meyer talks about command and query separation in his book Object Oriented Software Construction.

• Bounded context
• Behavior Driven Development or BDD
• Given, When, Then

“Man acts as though he were the shaper and master of language, while in fact language remains the master of man.” -  Martin Heidegger

Update Sept 2, 2009: I received feedback that such an architecture is only for mission critical systems. In his talk, Buy a Feature: An Adventure in Immutability and Actor, David Pollack, the creator of the Lift web framework for Scala, talks about an application, Buy a Feature, which implements this architecture.

Posted in Programming | No Comments »

Clean Code

The Clean Code Talks concentrating on writing testable code.

In his talk, Unit Testing, Miško Hevery explains unit testing and makes a case for unit tests.

…

In his talk, Don’t look for things, Miško Hevery provides a practical guide to the Principle of Least Knowledge (aka Law of Demeter, aka Don’t Ask, Tell)

…

In his talk, Global State and Singletons, Miško Hevery explains why global state leads to bad APIs and difficult to test code.

…

In hist talk, Inheritance, Polymorphism, & Testing, Miško Hevery presents his premise that most if statements can be replaced with polymorphism and explains why one would want to do so.

…

In his talk, GuiceBerry, Luiz-Otavio Zorzella makes a case for using dependency injection in your test code

Posted in Programming | No Comments »

Using instant messaging for distributed application communication

During the first session of the Lightweight Languages 3 workshop (includes video of the talks), Dana Moore and Bill Wright presented ACME: Toward a LL Testing Framework for Large Distributed Systems (abstract), in which they described a distributed application which used the XMPP (aka Jabber) instant messaging protocol to communicate between nodes.

XMPP addresses many complexities of inter-node communication in a distributed application, such as presence, heartbeat, queuing messages for offline nodes (offline messages), group broadcast (chat rooms), and status.

Here some other resources promoting instant messaging as a communication bus for distributed applications:

• Mickaël Rémond’s blog talks about using XMPP (Jabber) inside distributed Erlang applications. Here are slides from Mickaël Rémond’s presentation entitled Messaging with Erlang and Jabber.
• A proposed standard for game sessions using XMPP (Jabber) by by Michal Vaner
• A proposed universal Go server using XMPP (Jabber) by Mike Albon
• Peer-to-Peer: Building Secure, Scalable, and Manageable Networks by Dana Moore and John Hebeler – book on Amazon
• Jabber Developer’s Handbook by William Wright and Dana Moore – book on Amazon

Related links:

• Lightweight Languages 1
• Lightweight Languages 2
• Lightweight Languages 4
• Event-Based Programming: Taking Events to the Limit by Ted Faison – book on Amazon
• IM Instant Messaging Security by John Rittinghouse – book on Amazon
• Distributed Computing: Principles, Algorithms, and Systems by Ajay D. Kshemkalyani and Mukesh Singhal – book on Amazon

Posted in Other | No Comments »

Network theory – detecting cascades

In his talk, Dynamics of real-world networks, Jure Leskovec discusses where to place sensors in a network to detect cascades such virus outbreaks or rising memes.  I particularly enjoyed Jure’s explanation of how the cost-effective lazy forward-selection algorithm, which his team developed, helps to balance the cost of the sensors versus the reward of early detection.

Related links:

• Nexus: Small Worlds and the Groundbreaking Theory of Networks by Mark Buchanan – book on Amazon
• Linked: How Everything Is Connected to Everything Else and What It Means by Albert-Laszlo Barabasi – book on Amazon

Posted in Other | No Comments »

Debugging on Software Engineering Radio

Another podcast I listen to is Software Engineering Radio.  The Debugging episode talks about how testing does not prevent the need to debug, that debugging is a search problem, and tools for estimating where source for bugs are located.

Related links:

• Why Programs Fail: A Guide to Systematic Debugging by Andreas Zeller – book on Amazon

Posted in Other | No Comments »

Time and attention: THE soft skill of every knowledge worker

In his talk, Time and Attention, Merlin Mann talks about being conscious of what grabs your attention, negotiating requests for attention, and communicating organization culture around communication.

I enjoyed the part about defaults.  Simple tips for improving concentration:

• Configure your email client to not notify you when an email arrives.
• Configure your instant messenger to not notify you when anyone changes status.
• Change your browser’s home page to be something simple, something that does not display recent news, quizzes, weather, traffic, recent emails in a personal email account, recent blog posts. I prefer the fastest-loading option: about:blank, and I use Firefox’s built-in search.

I recently saw Merlin Mann speak at a OmniFocus meeting hosted by The Omni Group.

Related links:

• Getting Things Done: The Art Of Stress-Free Productivity by David Allen – audiobook on Amazon

Posted in Other | No Comments »

Processing in batches and improving maintainability

Here’s a common pattern:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

void process(final List<Foo> foos) {
    final ArrayList<bar> batch = new ArrayList<bar>(batchSize);
 
    for (final Foo foo : foos) {
        if (isValid(foo)) {
            final Bar transformed = transform(foo);
            batch.add(transformed);
            if (batch.size() % batchSize == 0) {
                processBatch(batch);
                batch.clear();
            }
        }
    }
    processBatch(batch);
}
 
// returns true if we want to process this Foo, otherwise returns false
private boolean isValid(final Foo foo) {
    return true;
}
 
// makes a Bar out of a Foo
private Bar transform(final Foo foo) {
    return null;
}
 
private void processBatch(final ArrayList<Bar> bars) {
    openTransaction();
    for(final Bar bar : bars) {
        executeQuery(bar);
    }
    closeTransaction();
}

We have some code duplication: processBatch appears twice in process. Also, we’re iterating over two collections in order to process each Bar: we’re iterating over foos in process and bars in processBatch. With the help of Google Collections, we can do better:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

void process(final List<Foo> foos) {
    final Iterator<Foo> validFoos = Iterators.filter(foos.iterator(), isValid);
    final Iterator<Bar> bars = Iterators.transform(validFoos, transform);
    final Iterator<Iterator<Bar>> batches = Iterators.partition(bars, batchSize, false);
 
    while(batches.hasNext()) {
        final Iterator batch = batches.next();
        processBatch(batch);
    }
}
 
private final Predicate isValid = new Predicate() {
    // returns true if we want to process this Foo, otherwise returns false
    public boolean apply(final Foo foo) {
        return true;
    }
};
 
private final Function transform = new Function() {
    // makes a Bar out of a Foo
    public Bar apply(final Foo foo) {
        return null;
    }
};
 
private void processBatch(Iterator<Bar> bars) {
    openTransaction();
    while(bars.hasNext()) {
        final Bar bar = batch.next();
        executeQuery(bar);
    }
    closeTransactioin();
}

You may want to look at the Javadoc for the Iterators class in Google Collections to better understand what’s happening here.

We removed the duplicate code. And while we still have two loops, one in process and one in processBatch, these loops are iterating over the same collection: foos. In addition, process has fewer lines of code. Granted, we have more lines of code elsewhere, but a real process method is likely to be more complex. By reducing the number of code blocks (braces) in process, we’re making it easier to read and maintain.

Posted in Programming | No Comments »

Authoring Iterators Using Google’s Collections Library

Let’s implement an iterator.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

/**
 * Removes duplicates from the elements returned by provided iterator.
 * The provided iterator must return the elements in the order elements in the order defined by the comparator.
 * The first element of a series of duplicates is returned by this iterator.
 * @param  Type of element returned by the iterator.
 */
public final class DedupingOrderedCollectionIterator<E> implements Iterator<E> {
    private final PeekingIterator<E> iterator;
    private final Comparator<? super E> comparator;
 
    /**
     * @param iterator iterates over an ordered collection
     * @param comparator implements the same comparison used to order the collection
     */
    public DedupingOrderedCollectionIterator(final Iterator<? extends E> iterator, final Comparator<? super E> comparator) {
        this.iterator = Iterators.peekingIterator(iterator);
        this.comparator = comparator;
    }
 
    public boolean hasNext() {
        return iterator.hasNext();
    }
 
    public E next() {
        final E next = iterator.next();
        while (iterator.hasNext() && comparator.compare(next, iterator.peek()) == 0) {
            iterator.next();
        }
        return next;
    }
 
    public void remove() {
        throw new UnsupportedOperationException();
    }
}

First, if you’re confused about the generic wildcards, watch Josh Bloch’s presentation entitled Effective Java Reloaded and watch for the reference of “producer extends, consumer super” (aka PECS) or reference Josh Bloch’s second edition of Effective Java.

Second, using Google’s PeekingIterator often reduces the code you need to deal with iterators.

Third, when implementing an iterator, you usually have to worry about what happens when next() is called when hasNext() would return false. What exception are you supposed to throw? And I keep coding the same implementation for remove() and always forget what exception it’s supposed to throw. Google’s AbstractIterator to the rescue!

Here’s the same iterator written with com.google.common.collect.AbstractIterator<T>:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

/**
 * Removes duplicates from the elements returned by provided iterator.
 * The provided iterator must return the elements in the order elements in the order defined by the comparator.
 * The first element of a series of duplicates is returned by this iterator.
 * @param  Type of element returned by the iterator.
 */
public final class DedupingOrderedCollectionIterator<E> extends AbstractIterator<E> {
    private final PeekingIterator<E> iterator;
    private final Comparator<? super E> comparator;
 
    /**
     * @param iterator iterates over an ordered collection
     * @param comparator implements the same comparison used to order the collection
     */
    public DedupingOrderedCollectionIterator(final Iterator<? extends E> iterator, final Comparator<? super E> comparator) {
        this.iterator = Iterators.peekingIterator(iterator);
        this.comparator = comparator;
    }
 
    protected E computeNext() {
        if (iterator.hasNext()) {
            final E next = iterator.next();
            while (iterator.hasNext() && comparator.compare(next, iterator.peek()) == 0) {
                iterator.next();
            }
            return next;
        }
        return endOfData();
    }
}

AbstractIterator allows you to write concise iterators with consistent behavior.

Posted in Programming | No Comments »

Parleys.com

Parleys.com publishes talks from software conferences.  I like seeing high-quality slides alongside video of the speaker.  Paryleys’ podcast is an audio only version of the talks you can watch on their site. Two of my favorite talks are:

• Flow of Change which discusses managing change in source control systems
  Speaker: Tony Smith, European Technical Services Manager for Perforce Software
• The Truck Driving Problem which discusses developing innovation
  Speaker: Mary Poppendieck, author of Lean Software Development: An Agile Toolkit

Posted in Programming | No Comments »

TV Ontario’s Big Ideas

I’ve received some requests to share some of my favorite podcasts. The first I’d like to share is a non-technical podcast called Big Ideas.  It’s the only regularly scheduled program dedicated to the art of the lecture and the importance of ideas in public life.  One of my favorite Big Ideas lectures is by Oxford University’s Harvey R. Brown.  This lecture is a fun ride on the logic train, discussing motion and time. Enjoy.

Posted in Other | No Comments »

Creative use of closures in testing.

Having moved to Java, I do miss closures. xUnit.net has a creative use of closures in their unit testing framework:

1
2
3
4
5
6
7
8
9

[Test]
public void DivideByZeroThrowsException()
{
  Assert.Throws<System.DivideByZeroException>(
    delegate
    {
      DivideNumbers(5, 0);
    });
}

This code snippet was taken from xUnit.net’s documentation.

Previously, each test method could have an expected exception. Now, one can have multiple calls which are expected to throw an exception in a single test method. Good thinking!

Posted in Programming | No Comments »

Phishing and other scams.

I always find these security talks entertaining.

In his talk, Searching for Evil, Professor Ross Anderson discusses research done by himself, Dr. Richard Clayton, Tyler Moore, Steven Murdoch, and Shishir Nagaraja.
http://video.google.com/videoplay?docid=-1380463341028815296

Related links:

• Security Engineering: A Guide to Building Dependable Distributed Systems by Ross J. Anderson – book on Amazon
  

Posted in Programming, Usability | No Comments »

Design by Contract

I would like to see design by contract become mainstream. JSR-305, Annotations for Software Defect Detection, is a step in the right direction. The applicability of this standard is broader than the name suggests. Here’s a talk about the JSR by Bill Pugh:

I would like to see compile-time and full IDE support for design by contact.

Related links:

• Design by Contract, by Example by Richard Mitchell and Jim McKim – book on Amazon
• The Practical Guide to Defect Prevention by Marc McDonald, Robert Musson, and Ross Smith – book on Amazon

Posted in Programming | No Comments »

Favorite JavaOne Videos

I’ve moved over to Java. Here are my favorite sessions from JavaOne.
Videos require free registration. You only need to register once for all the videos.
Language:

• Languag-Oriented Programming and Language Workbenches
• Scala is a functional and object-ortiented language that runs on the JVM, it’s similar to F# (F sharp)
  
• Closures for Java
  
• Fast x86 Emulation in Java This reminds me of the CLI back-end for GCC
  

Architecture:

• To Know the Dependencies is to Know the Architecture – Dependency Structure Matrix
  
• OSGi Best Practices
  
• Advanced Spring Framework – Spring OSGi
  
• Extreme Web Caching

Concurrency:

• High-Performance Java in a Multicore World – memory latency matters
  
• Scaling Up a Real Application on Azul
  

Posted in Programming | No Comments »

Automated Testing – Model-Based Testing

In his talk, Model-Based Testing: Black or White?, Mark Utting discusses the difference between black-box and white-box models and their affect on the ability to automate testing.
http://video.google.com/videoplay?docid=5521890509476590796

Related links:

• Practical Model-Based Testing: A Tools Approach by Mark Utting and Bruno Legeard – book on Amazon

Posted in Programming | No Comments »

Signals, Truth, & Design

In her talk, Signals, Truth, & Design, Judith Donath discusses intentional and unintentional signals as well as truth in signals.

Related links:

• The Future of Reputation: Gossip, Rumor, and Privacy on the Internet by Daniel J. Solove – book on Amazon

Posted in Other | No Comments »

Closures for Java

In his talk, Closures for Java, Neal Gafter provides a description of and an argument for closures in Java.

Posted in Programming | No Comments »

A new human interface

In his talk, Dasher: information-efficient text entry, David MacKay introduces a unique human input mechanism, Dasher. I’ve used Dasher and quite like it.

There’s a Java applet version of Dasher, but I encourage you to try a native version.

Contributions to Dasher have been made by David MacKay, David Ward, Alan Blackwell, Phil Cowans, Tim Hospedales, Iain Murray, Hanna Wallach, Matthew Garrett, Chris Ball, Keith Vertanen, Tadashi Kaburagi, Brian Williams, Chris Hack, Ingrid Jendrzejewski, Frederik Eaton, and Piotr Zielinski.

Related links:

• Information Theory, Inference & Learning Algorithms by David J. C. MacKay – book on Amazon

Posted in Usability | No Comments »

Faster sites mean more money.

Google found that when pages took only half a second longer to appear, usage of their site dropped 25%.  In her talk, Scaling Google for Every User at the Seattle Conference on Scalability, Marissa Mayer communicated these results. The explanation of the statistic starts around 9 minutes into the video and takes about 3 minutes.

Also, Amazon found every 100ms of latency cost them 1% in sales and Wikia found that faster pages reduces exit rate.

Posted in Usability | No Comments »

Pharming

In his talk, Drive-By Pharming and Other WebApp Bummers, Sid Stamm discusses creative exploits.

Posted in Other | No Comments »

Managing Email

In his talk, Inbox Zero, Merlin Mann discusses one of the most important soft skills of a knowledge worker.

Related links:

• Getting Things Done: The Art of Stress-Free Productivity by David Allen – book on Amazon
• Lifehacker: 88 Tech Tricks to Turbocharge Your Day by Gina Trapani – book on Amazon

Posted in Other | No Comments »

Copyright regime vs. civil liberties

In his talk, Copyright Regime vs. Civil Liberties, Rick Falkvinge, founder of the Swedish Pirate Party, makes an interesting argument.

Related links:

• The same talk given at Stanford Law School
• QuestionCopyright.org

Posted in Other | No Comments »

Functional list processing with anonymous delegates

Derrick Coetzee has an article about functional list processing with anonymous delegates. Keep code close to where it’s used.

Posted in Programming | No Comments »

Using @ infront of literal strings

Using @ infront of literal strings lets you do:

instead of:
“C:\\directory\\file.xml”
or
“SELECT field FROM table where field=var”

you can do:
@”C:\directory\file.xml”
and
@”SELECT
field
FROM
table
WHERE
field = var”

Posted in Programming | No Comments »

FTP with streams in .net c# vb.net

There needs to be a good opensource ftp library for .net.  I want to be able to write to the FTP stream, not pass a byte array or even a pointer to a file.  Why can’t I get a

Stream Upload(string filename)

closing the stream would just end the file upload, not the network connection.  And I want multiple connections and async methods.

Posted in Programming | No Comments »

Screen scraping poker web sites

Screen scraping is very brittle. It will require continual maintenance and will never be complete. A good screen scraper is like writing a parser. Extracting semantic meaning from text with a poor signal to noise ratio is non-trivial.

Ideally, you would access the data via an API or a feed. Google, Amazon, Flickr and many other sites provide free programmatic access to data. Sites like Ebates offer feeds of data for free. If your data is not available for free, you may be able to purchase access to an API or feed.

Screen scraping can be used well for other purposes, e.g. testing the UI layer of a web application. In this application of screen scraping, the same organization generates and tests the UI.

What you really want is to be inside these poker web sites. You want to be able to be a sort of plug-in for your customers. I suggest taking a look at the Book Burro extension for Firefox, then visit Amazon. Book Burro adds an interface widget on top of the Amazon page with prices from other bookstores on the web. This is a cross-platform solution.

I’ve read that the poker sites aren’t much bothered by ‘poker bots’.  As long as people gamble, they’re happy. Maybe you could find a poker site that would let you plug in your software… or make your own site for all the users of your software to play on and they think they’re playing other people. A game server, now that’s a completely different kind of software.

Posted in Programming | No Comments »

Apple Store sessions time out.

You have to re-enter the store if you’ve been idle.

That’s bad.

Posted in Usability | No Comments »

Faster first byte.

override render(HtmlWriter)
HtmlWriter.Flush
DoSomeExpensiveOperation
// or join with a thread that’s loading state
base.Render(HtmlWriter)

This way, you can send the headers and navigation to the client so they’ll see something while you’re waiting for something to finish. Perhaps the client won’t even finish downloading the headers when the render completes, you’ve gotten the first byte out faster.

Posted in Programming | No Comments »

Consider restoring view state from the previous sessions.

Many Integrated Development Environments do this.  Stickies on Mac does this.  Email clients retain sort preferences.  Why not save sort preferences in a web application? Why doesn’t my Instant Messaging client bring up the conversation windows I didn’t explicitly close?  My DVD player asks me if I want to continue from the last time I was watching that specific video.

Posted in Usability | No Comments »

Should you be persisting more state?

Lost state is state that has been destroyed, deleted, garbage collected or otherwise removed from storage not because of some domain specified reason.  In the best case because of limited storage capabilities, but because the value of the state is underestimated.

Microsoft Word has autosave.  By default, every ten minutes the document you’re editing is persisted to disk in case of an unexpected termination of the application.  I always change this to one minute, because why would you ever want to wait ten?

And why doesn’t New Egg’s shopping cart remember what I had in it?  I’ve had business people tell me customers get angry when they see items in their cart many months later.  I argue, at least have it be an option to review.  Like an automated (private) wish list.

Suppose you have a web server produce views of product information that is expensive to acquire. Perhaps it’s over a web service or a remote database.  Often, you’ll cache this information in memory, trading aging data for faster request processing.  But what if there’s a power outage, and both the web server and database suddenly shut down.  If your application starts running before the web service or database, then your application is missing an opportunity to process requests. If you had stale data, you could still provide information to anxious customers.  Even if you didn’t want to show old quantity levels or prices, you could still provide a description, and maybe a notice that you’ll be fully operational in a moment.

Consider dumping in-memory state, such as a cache, to disk if the connection to the data source is lost or during low load.

Posted in Programming | No Comments »