Next week in Mountain View, CA, we'll be hosting the world's first Selenium Users Meetup. If you're in the area and are even remotely interested in development methodologies, QA, or automation, come on by. The event is on Monday, February 25 from 6:30PM to 9:00PM. You can find out more here
Does anyone know of a simple, standalone cross-domain XHR interface that allows for the ability to swap between a real XHR implementation and a cross-domain, iframes-based implementation?
I know Dojo and DWR, among others, have them, but they are all part of a larger framework. Because this is going to be used with Selenium, which needs to avoid polluting common popular JS namespaces, I'd prefer to find something that is standalone.
I'm very surprised one doesn't pop up when I search google.
All software development organizations, take notice: Atlassian builds software right and the result is continually great software. I can't speak for Crowd (haven't used it), but I certainly know that the process defined in this article certainly will put your project on the path for success (in terms of technology... of course if the idea sucks, a nice dev process won't save you!):
Basically we have taken the following steps:
- Move from CVS to SVN
- Move from Ant to Maven 2
- Add some level of test coverage, and build this into our development process
- Add integration tests for the Crowd Console
- Hook all this up with Cargo and Maven 2
- Drop it into a continuous integration server
(Via Atlassian Developer Blog.)
The only thing I think they could improve on is the use of JWebUnit/HtmlUnit. They should really step up to Selenium, especially considering how often bugs crop up due to inconsistencies across web browsers. But still, that is merely an incremental improvement to their very solid foundationl. On top of that, JWebUnit has an abstraction layer on top of Selenium, so it should be an easy switch.
About a month ago I wrote that I thought the two most important things tomorrow's testing tools need to help with are infrastructure and maintenance. I'm about a month late on my promised follow up, but I suppose it's better late than never.
What I wrote was:
By infrastructure, I mean challenges like setting up the test environment, keeping it in a consistent state, ensuring you have all the tools/browsers/databases/whatever always available for automation, running your tests in a timely manner, etc. All of these are much more challenging that JUnit, simply because JUnit only concerned itself with in-memory state and fixtures that could be recreated in milliseconds, not minutes or hours.
As applications become more complex, the work required to automate functional testing also becomes more complex - often more complex than the application itself. For example: it's trivial these days to do Web 2.0 "mash ups" in your app and embed a Google Map in to your page (it's a simple <script> block).
But what about testing that functionality? What about checking the performance of it? Not only does it require a compatible browser (no more HttpUnit or HtmlUnit, now you need to use something like Watir or Selenium). And what if you want to test what your app behaves like when the Google Map service is down or running slow. How do you do that?
The fact is that as we continue to build layers and layers of abstraction and extensible services, our speed at which we can develop rich, complex applications grows and the ability to test those applications becomes a much harder challenge.
Next gen tools must help
This is why I believe that the most fundamental thing that any testing tool (or process) will do in the coming years is to help manage these complexities. It's not an easy problem to solve, but fortunately some ancillary technologies such as virtualization help immensely.
The testing tools of tomorrow can no longer simply assume that automation of the test case (click there, type this, click here, type that, ...) is enough. We are in an age where that type of technology is a commodity (Selenium, Watir, etc). They must take developers to the next level of automation and help automate the process of preparing to run these scripts.
Many QA teams spend a majority of their time not writing automated tests or performing required manual testing, but in fact setting up the application in a QA/testing environment. This is a huge waste of time and is something that is begging to be automated.
Continuous integration is key
Let's pretend tomorrow's tools do this. They now automatically set up your database to a know state. They help you manage snapshots of data that can be used for various automated tests. They reset the state as needed when running automated test cases. They deploy your application properly. They set up a web browser with the cache cleared out. They do it all.
The logical next step is continuous integration. The vision outlined in Martin Fowlers essay can be realized for not just code that can be easily unit tested, but for all the code in your application, even HTML, JavaScript, Flash, etc.
Now when you check in your latest JavaScript changes, you don't have to worry about nagging questions like, "I forget, does IE support hasAttribute()?" (it doesn't). Now all the benefits you got for writing unit tests (instant feedback on failures, etc) can be realized for writing functional tests.
Suddenly there is a huge incentive for getting developers to take part in the process of writing functional tests in addition to the unit tests they write. That's because they are continually reminded of those tests, every time they check in code they are notified if one of those tests fails.
Contrast this to today, where most teams barely even write functional tests, and those that do usually relegate that job to the QA team to run near the end of the release cycle. The difference is like night and day.
Some are doing it today...
The good news is that some people are already doing this. Projects like HSQL (an in-memory database) and Cargo (automatic J2EE deployment tools) help considerably. I recently "blogged":... about how Atlassian is using these tools to get close to tomorrow's development and testing process today.
HSQL and other in-memory databases help here because they allow for your functional tests to have clean fixtures. That is: they provide a simple and quick way to reset the data in your application so you can run the next functional test. And because they are in-memory, you can reset the state for every test, ensuring you get zero conflicts between test X and test Y.
Cargo is another great tool because it does all the heavy lifting required to deploy a J2EE application to a specific app server (Tomcat, Resin, JBoss, etc) and even allows you to programatically configure it, such as define database connection pools. In fact, Cargo recently began to add support for deployment and configuration of databases as well, so you could use it to set up your entire application deployment just before you run a test.
... but soon they will need more
While HSQL and Cargo are nice, they still are merely emulations of the real thing. That is, you likely won't use HSQL in production and you likely won't deploy on the same configuration of Tomcat that Cargo provides. Just as jWebUnit served as an acceptable browser emulator until AJAX arrived, these projects too do their job well for the time being.
However, as deployments become more complicated, so too will we again need to test on the real thing. So just as Selenium is to jWebUnit, tomorrow's testing tools will need to provide an ability for technologies like HSQL and Cargo.
For example, just last week I had first hand experience with this challenge. Our functional tests use HSQL, but our production environment runs on Postgres. I had checked in some new code that executed a query that was passing just fine in our continuous integration process, but when it came time to push to staging, it broke.
The truth is that in a perfect world, we should be testing on the same environment as our production one, and things like HSQL just don't cut the mustard. The reason we use them, however, is pure pragmatism: it costs too much and takes too long to develop a testing environment/functional testing fixture that works with a clone of your production environment.
Rising infrastructure costs
In fact, even those that are doing this today run in to challenges: Atlassian's full integration test suite takes hours to run. Some teams have reported to me that their suites take over a day to complete. And this is all while using shortcuts in their fixtures such as HSQL.
This is where parallelization comes in. The reason these test suites take so long is that they almost always run in sequence. That is: test X runs, then test Y runs, etc. To save time, some teams cut out the heavy lifting in their fixtures (such as deploying the application, setting up the DB, etc) to only happen at the start of the suite rather than each test.
This almost always leads to tests that end up conflicting with other tests (ie: test X and test Y both try to create the same user with a unique username of "bob"). But on top of that, it also begins to introduce dependencies between tests and a preference towards test ordering.
That's because, after a while, developers begin to take advance of this sequential order and do even more nefarious shorts. No longer does test Y need to create a new "Widget ABC", because test X did. Of course, if test X ever changes, test Y suffers for it.
So why do even the best teams do this? Because there really isn't another option today. That's because the cost in terms of hardware/infrastructure as well as developer time are orders of magnitude larger.
If you currently have 500 functional tests and it takes, on average, 10 seconds per test, plus a one-time 5 minute fixture setup time, then you can run all your tests in about 90 minutes. But if you reset your fixture for each test, it would take 43 hours. Pragmatism takes hold usually right when people realize this.
Even at the much smaller 90 minutes, it's often too long to place in to a continuous integration process that runs on every commit. The solution is parallelization. Now image those 500 tests running in parallel. The entire suite could run in 5 minutes and 10 seconds - or a short enough time that you could introduce a complete, reliable, real functional test suite that runs as often as you need.
The limitation today is the hardware costs: 500 build machines would cost a team $1,000,000, plus multiple times more maintaining it, powering it, etc. It's just not possible... today.
The next gen testing tool isn't a tool at all
Assuming that a "next gen testing tool" must be something that radically improves the software development process, it is my belief that tomorrow's next gen testing tool will in fact not just be a tool, but an entire infrastructure, most likely offered as a service that you can rent on an as-needed basis.
It would let you run your 500 functional tests in as many as 500 parallel tasks, or as little as 1. It would let you ramp up the parallel tests as your needs and budget evolve. It would finally allow you to run all your tests after every commit, giving you the maximum level of confidence after each commit.
... and it might even be called HostedQA ;)
It's a long read, but well worth it: check out Faster Feedback and Why You Want It: From TDD to CI and Beyond from the IDEA guys. They really are spot on, and lot of what is said there goes back to the same philosophy that drives IDEA excellence.
I'm really looking forward to what JetBrains continues to pump out - they have an amazingly solid platform (IntelliJ IDEA) to build upon.
"Dan North" (tastapod!) has a good write-up on what he is calling "BDD": Behavior Drive-Development.
He does a much better job than I can do discussing all these items, but the thing that struck me is that almost all his conclusions can be reached by following just one piece of advice: make your test methods really expressive.
I started doing this in 2004 and it has been a huge help. My tests are more focussed, and because of that, I write more tests because it doesn't take as much thought/energy. And in writing more tests, I often design my code better, and ultimately produce software faster. As a nice side effect, quality has gone up, but that is not the primary reason for TDD or BDD.
Unfortunately, Dan's examples of test methods were rather weak - often the agiledox crowd uses really simplistic examples. So I thought I'd give some examples of some of my methods from HostedQA:
And my personal favorites (broken up with a space because they are so long):
Kind of annoyingly long methods? Absolutely. But not only, as Dan points out, is it easier to know what went wrong when a failure occurs, it is also almost brainless to implement these tests (and thereby design your application in the most direct manner possible).
If my test requires that I test that "updating an app config without resources that used to be associated with it causes those references to be deleted", it's really easy to know what I need to test - it's literally spelled out for me.
As a small note: Dan recommends that tests usually have "should" in front of them. I prefer to think that all my test methods (I use TestNG, so I don't have "test" in the method names) have "test that" prepended in front of the sentence. Ie: "test that adding X and Y results in Z" becomes addingXAndYResultsInZ.
What Problem Would Next-Generation Functional Testing Tools Solve?:
Various people have tried various structures including table driven things (like FIT/FITnesse) and Domain Specific Languages. But none of these approaches has quite succeeded in transforming the state of the practice the way JUnit did for unit testing.
(Via Ruminations.)
Elisabeth Hendrickson has posted several posts on next gen testing tools recently.
With HostedQA, which, by the way, recently had some major developments I'll be announcing soon, I tried to attack two main problems that crop up when writing very rich tests like functional and acceptance tests: infrastructure and * maintenance*. I'm sure that there is a lot to do on the reporting front as well, but I'm leaving that alone for now. :)
By infrastructure, I mean challenges like setting up the test environment, keeping it in a consistent state, ensuring you have all the tools/browsers/databases/whatever always available for automation, running your tests in a timely manner, etc. All of these are much more challenging that JUnit, simply because JUnit only concerned itself with in-memory state and fixtures that could be recreated in milliseconds, not minutes or hours.
When I talk about maintenance, I mean the issue of describing tests in a way that is usable by the test writer and evolving the test as the product and requirements evolve. Again, with JUnit test description was easy: the test writer had intimate knowledge with the item being tested and was able to write it in the same language. For complex testing, there are literally a handful of languages being tested (SQL, Java/Perl/Ruby/etc, HTML, JavaScript, Flash, PL/SQL, etc), so right off the bat there is a big disconnect between the simplicity of JUnit and this problem.
I'd like to dive in to the solutions I've come up with for both of these, but it's getting late and I'm already in the wrong time zone. But I promise to follow up in the next day or two on how HostedQA has addressed infrastructure and maintenance problems, all with the goal of making functional testing as ubiquitous as JUnit is.
For now, I leave you with this: These are the two biggest challenges that I see with this type of testing. Do you agree? Disagree? What is your biggest barrier to entry for this type of testing?
Why IntelliJ IDEA Customers Keep Coming Back:
Brian Pontarelli has a passion for the keyboard and makes some good points about usability and support in regards to Java IDEs, specifically IntelliJ IDEA and Eclipse. The comments after the main post are especially interesting, such as where some commenters argue over whether a reported issue counts as either a severe bug or a missing feature. I mean, come on. Does it really matter? The point is that it interferes with Brian's experience of usability and productivity. I've been saying it for a while: It's not about feature lists, it's about usability and productivity. You won't find usability or productivity on a feature list, but it can easily be the most important deciding factor on whether or not to choose one piece of software over another. JBuilder was marketed on the length of its feature list, and look where it got them.
(Via idea.log.)
Spot on. Feature-for-feature, Eclipse has most of what IDEA has. Most of the same refactoring, most of the same plugins, etc. But it doesn't have the intangibles that don't fit on a feature list.
I was just demoing IDEA to a coworker (who is an Eclipse user) and he was very impressed at the subtle things I showed him that Eclipse just completely glosses over:
There are a ton more examples I could provide, but the general rule of thumb is that while most IDEAs might match it feature-for-feature on a check list, none go nearly as deep. And it is that depth that makes IDEA users so much more productive.
I ran across an interesting article entitled “Improving Test Coverage of Ajax Applications” in which the author likens the challenges of testing Ajax applications to that of testing traditional GUI apps. While some newer frameworks (like Selenium) are positioned to actually verify Ajax-ian behavior, they can lead to a false sense of security because of the complexity associated with the combination of actions in using a GUI.
(Via Test Early.)
I posted the following comment to the entry above, but I think it important enough that I bring it up here as well. I'd be interested in hearing about ideas that can help make expressing tests for AJAX applications easier:
Absolutely - Selenium and other tools are now available to help, but there are still real challenges in describing these new types of interactions. The old-school tools (SilkTest, QTP, HttpUnit, etc) either didn't understand the concept of multiple user transactions in a single page or didn't have the technology to deal with AJAX at all.
Selenium helps with the AJAX/multiple user transactions issue, but it still requires testers to express the desire to "click on button X, wait for Y to appear" in their tests. I think there is still room to make this easier to understand and test against, possibly by integrating with AJAX and/or web-based frameworks, and thereby understanding the interactions at a deeper level.
Confluence 2.4 would be developed in a strict six week time-box from inception to release. Anything we had done at the end of those six weeks we'd ship.
(Via Atlassian Developer Blog.)
Atlassian has always been a company I admire greatly. They put engineering first and foremost and let their products sell themselves. I'm really glad to see that they are continuing to push the envelope with short release cycles and lots of automated testing. They have a few interesting things just around the corner that will make watching this experiment that Charles blogs about even more interesting.
Of course, Atlassian wouldn't be able to do any of this without their huge library of automated tests. I'm told they use HtmlUnit for most new tests, and HttpUnit in older tests. Now if I could only get them to try out Selenium, or even better, HostedQA. :)
I just recently discovered that HostedQA usage is picking up: now Apache MyFaces is using it too!
I'm personally really excited about this because it is one of the first cases where an opensource project has integrated with our products without any help directly from me. Very cool!
Today I'm pleased to announced that the OpenQA jobs board is now live. As OpenQA, and the great projects that make up the community, have grown over the last year, a rich community of developers and users has sprouted up around it. Part of that growing process involves needs for commercial support, training, and technical expertise around the opensource tools at OpenQA.
That is where jobs.openqa.org comes in. It is a place where people looking for developers with deep knowledge of OpenQA can go to look for work, as well as a place where people can offer services around OpenQA-related technologies.
As OpenQA continues to grow, we hope to keep adding features like this to help strengthen the bond between opensource QA and commercial companies.
And of course, a small personal note about this for full disclosure: my company, Autoriginate, is one of the first companies offering support and training for Selenium, an OpenQA project. The creation of the Jobs Board certainly helps my company, but it can also be used by any other company offering similar services without exception. We believe this is required to take OpenQA as a community to the next level of growth.
As of a couple days ago, I added HTTPS support to Selenium RC. This is a major addition, as it has been pretty much the only large feature missing from Selenium products. The result is that you can now remotely drive web browsers on sites that use HTTP or HTTPS protocols. Just as Selenium Remote Control was, this code was originally created as part of our HostedQA product.
Using this feature
To get started with testing on HTTPS, you have to use the custom browser launcher for now (we'll add support for it for the specific browser launchers as time goes on). The reason for this is that you need to import a Certificate Authority in to the main browser profile. Look in the advanced/security settings in Firefox/IE for this option.
You can find the CA at http://dangerous-certificate-authority.openqa.org/
Once you have the CA imported to the browser and have changed your tests to use the custom browser launcher, you can begin testing against HTTPS. Everything should just work.
How it works
Unlike other similar web testing tools, this solution doesn't require you to prepare the web browser for each domain you want to test. Rather, you only need to prepare the browser once by importing the CA.
What happens behind the schenes is that as each HTTPS connection is opened to a unique server/host combination, a mini-instance of Jetty is spawned off and used to tunnel SSL connections to the server. However, that Jetty instance needs a keystore that is signed by a trusted CA and is associated with the domain in question.
That keystore could be generated through a series of keytool and openssl commands, but we didn't want you to have to install these locally, since that can be a pain, especially for openssl. So instead, we created a web service at the dangerous-certificate-authority.openqa.org domain that generates the keystore for you.
The downside to this approach is that you must be online for HTTPS support to work. In the future we may offer a fallback that issues the keytool/openssl commands if the online service can't be accessed.
Give it a shot
The latest SNAPSHOT builds at maven.openqa.org should have SSL support available. Give it a shot, or wait for Selenium RC 0.8.2 to come out.
Or, alternative, sign up for HostedQA and start using these features (plus a ton more) today.
A really cool feature we recently added to HostedQA was the ability to pre-load virtual machines in order to optimize automated testing. At first, I wasn't sure if this was going to have any real user experience impact. But after having built it, I wish I'd done it sooner.
It turns out, if you have the RAM and you're using VMs for automated testing, you should really think about pre-loading your VMs ahead of time. Then instead of waiting several minutes to kick off your automated tests, they can begin right away.
Now granted that if your test suite takes 10 hours, then a 3 minute gain isn't that important. But that's not the use case that this is designed to optimize. Instead, this feature, and the general concept of pre-loading VMs, is all based around the use case of creating new tests.
Why creating? Often when someone writes a complex test case that depends on the entire system being deployed and working properly, they don't want to go too far without verifying they are on the right track. They do this by running the test. Now your 10 second test actually takes 10 seconds to run, instead of 3 minutes and 10 seconds. Suddenly, the environment is much more hospitable for test creators.
Version 0.8 of the web application functional testing tool Selenium has been released.
Jeff Xiong explaing the new features...
(Via Ajaxian Blog.)
I'm especially excited about this release because of the implications with our HostedQA and AutoQ products. The reason is because I started the initial work on the multiWindow mode (though, admittedly, the rest of the Selenium dev team was absolutely required to make it finally work properly) for the very purpose of providing a frameless testing infrastructure for HostedQA.
We've actually been using this multiWindow/frameless mode in our products for a while in our products. The result is that instead of seeing your application inside of an iframe, your app runs in its own window, more accurately reflecting the real world usage.
You can see some example screenshots of what it looks like by checking out some of our opensource projects reports.