|
(With contributions from Becky Sowada, Steve Peterson,
and the Pet Market QA team.)
In this article, we'll discuss the quality
assurance (QA) processes for the Pet Market blueprint
application. Here's the good news: all of the
QA techniques that you use for HTML and JavaScript-based
web applications can be directly applied as you QA
Rich Internet Applications.
Fundamentally, a web application is a software project.
There are many good resources for best practices about
development and QA that you can apply during QA. However,
there are a few key lessons that the QA team learned
along the way that will help you deliver Rich Internet
Applications that are usable and error-free.
The quality assurance team on Pet Market had a unique
challenge. The application had to run well on:
- A multitude of web browsers
- Two server operating systems (Windows and UNIX)
- Two client operating systems (Mac and PC)
- Two server scripting environments (ColdFusion
MX and the Microsoft .NET framework)
- The JRun 4 J2EE application server
Fortunately, you probably can eliminate a server
operating system and an application server platform
(J2EE or .NET) from the mix, as your application may
not need to work across multiple back-ends. However,
delivering a high quality Rich Internet Application
still rests squarely on the shoulders of your quality
assurance (QA) team and the processes they use to
ensure a great user experience.
The team
Becky Sowada led the QA effort on the Pet Market blueprint
application. She had four QA engineers on her team:
three with five years or more of QA experience, and
one with only 18 months. The team members came from
a QA team for a Macromedia server product. They had
done user experience QA for web applications and were
knowledgeable about Macromedia Flash-based user interfaces.
Mark Soderberg took the responsibility for automated
testing and performance measures, while Ladonna Williams
and Jamie Huggett wrote the test cases and did functional
QA. Rondi Lund did functional QA and helped with general
tasks.
While most of the team was familiar with Macromedia
Flash, the team had never released a client-server
application using Macromedia Flash Player 6 and ColdFusion
MX. As it turned out, most of the testing effort was
on the client side (80% client and 20% server). Most
of the work revolved around creating a great user
experience on the client. But the team was well configured
for these tasks. Becky noted that if there was a skill
gap on her team, it was in administering and configuring
the various server and application server environments.
[Tip #1: Make sure your QA team has time to
get familiar with Macromedia Flash and has a good
balance of server-side QA, client-side QA, usability,
and server administration skills.]
Becky's team divided the work of the team across
three key areas:
• Writing test cases and other documentation
• Creating automated and semi-automated tests
to save time
• Doing the functional testing
Written test cases ensured that a “QA pass”
(a complete review of the functionality of the application,
including edge cases and boundary conditions) was
the same every time. The team created automated testing
routines to quickly focus on what piece of the application
was breaking. They also automated performance testing.
The functional testing required the QA engineers to
step through the application to make sure that no
functionality was lost between builds and to confirm
that bugs were fixed. By and large, most of the team's
effort was focused on functional and compatibility
testing.
The tools
The team made use of several tools to make their work
easier and more standardized. Ideally, test cases
are based on engineering specifications
(“specs') which describe the as-designed
functionality of a given feature. The test cases describe
the steps needed to ensure that the feature works
as intended. In many teams, the quality assurance
team is involved in the design of those specifications
so they are intimately familiar with the design of
the application. [Tip #2: Get your QA team
involved in the specifications of the application
early and write detailed test cases from those specs.]
The testing matrix was the core document for the
QA team. As the team worked through the functional
QA of each build, they would check off the areas of
the application that had passed QA. This list of tasks
let the team view what tests remained to be done,
which tests where checked with each build, and which
bugs were found for which test cases.
The testing matrix provided a checklist of platforms,
browsers, and operating systems and assigned owners
to each combination. While Macromedia Flash Player
is cross-platform and reliable across a wide range
of browsers and platforms, the QA team wanted to be
sure that the application worked especially well for
the target platforms.
The team used of a bug database to enter bugs that
the engineering team could use to make fixes to the
application. Most software development efforts make
use of a bugbase, so that bugs can be tracked, fixed,
regressed, and sometimes deferred (as little as possible).
The team also built a special test file (a Macromedia
Flash MX-native FLA file), which they used to test
the server side of the Pet Market application across
platforms and between builds. This file
helped the QA engineers
sign off on the various server implementations of
Pet Market by ensuring that all of the APIs and calls
used by the client-side front end of Pet Market were
still functioning. By running the file inside of Macromedia
Flash MX with the NetConnect
debugger add-on, the team could check the function
of the server APIs. This allowed the team to rule
out errors on the server when tracking down an error
in the app. [Tip #3: Build test clients that
can ensure the error-free operation of the server
side of your Rich Internet Application.]
For automated testing, the team made use of Homer,
a free loadtesting utility, and SilkTest,
which enabled the team to script some of the user
interactions. (Editor's note:
While today none of the automated testing tools from
Mercury Interactive, Rational, Compuware, or Segue
directly support Macromedia Flash Player 6, Macromedia
is working to address this with those vendors.) However,
since so much of the Pet Market application relied
on new functionality in Flash Player 6, the team relied
primarily on manual testing. [Tip #4: Automate
as much testing as possible.]
The timeline
The QA team was involved in the Pet Market project
very soon after the development team began to code.
Here is a rough schedule of the key phases of the
project:
• Prototyping, architecture and design: 2 weeks
• Server-side development: 2 weeks
• Client-side feature development: 3 weeks
• Usability testing: 1 week
• Functional Testing: 2 weeks
• Bug Fixing and Beta testing: 4 weeks
• Performance enhancements and platform testing:
2 weeks
• Endgame: 3 weeks
Most of the prototyping and user interaction was
designed before the QA team was fully staffed on the
project. The development team used personas for the
users to ensure that the application was user-centered.
In general, getting the QA team involved early makes
the downstream process smoother, because the QA engineers
will be more familiar with the application and enfranchised
in the decision making process. [Tip #5: Get
the QA team into the process early.]
Developing and doing QA on the the server-side functionality
of the application was a key priority for the team.
The QA team was able to mold their test cases around
the server APIs and sign off on major pieces of functionality.
This freed the development and QA teams to focus on
the user experience and client-side development, which
required more cycles of testing, usability testing,
and feature development. While the server-side APIs
did evolve during this period, the stability of this
part of the application made for a smoother QA process.
[Tip #6: Start QA on the server side of your
application early in the cycle.]
Midway through the schedule, the checkout experience
had been implemented as a single screen with multiple
form fields. When usability
testing showed that users found this design overwhelming,
the team redesigned the entire experience to its current
“accordion” of expanding screens for each
step of the process (login, billing address, shipping
address, shipping options, and credit card). The QA
team had to reset around this new design and build
test cases for the newly redesigned features. Once
the team completed the work, built the installers,
and fixed most of the bugs, the team released the
Pet Market application to the beta list for feedback.
By planning ahead for the feedback, the QA and development
teams were able to add and drop features and incorporate
feedback from a wide variety of sources. [Tip
#7: Leave time in the schedule for revisions and user
feedback.]
At this point in the schedule, the QA team's
processes were well-documented. The QA team was well-versed
in the functional QA requirements of Pet Market, and
three QA engineers could complete a full QA pass on
Pet Market in two days.
At this point, the QA team changed its focus from
the functionality of Pet Market to the reliability
of the application on various platforms. First came
the detection scripts that determined that the user's
browser was configured with the correct version of
Macromedia Flash Player. Dan Tull, the engineer who
implemented the detection script, was surprised at
how little time it took (he used the Macromedia
Flash MX Deployment Kit). He walked into Becky's
cube and said, “The detection is in, or at least
I think it is, because it was way too easy.”
Of course, it was that easy. [Tip #8: Use
the Macromedia Flash MX Deployment Kit for browser
and Macromedia Flash Player detection.]
Finally, it came down to performance enhancements,
platform testing, and the endgame. During this phase,
the QA team served as the team's watchdog, ensuring
that fixes and optimizations did not introduce new
issues. Becky and her team wish that they could have
done performance testing earlier in the process. The
testing was time-consuming, but it ended up having
a big impact on the overall user experience. [Tip
#9: Get performance metrics and testing on the schedule
from the beginning—performance is critical for
a great user experience.]
As the team was testing for platform and browser
compatibility, they uncovered an issue with Mozilla-based
browsers and Macromedia Flash Player 6. While Macromedia
Flash Player 6 is the most reliable and ubiquitous
client runtime by leaps and bounds, Macromedia cannot
always control changes that are made in the browser
environment.
During the endgame, the marketing team (including
yours truly) pushed for new features such as a new
loading sequence and “back button” functionality.
While most projects do not enjoy the luxury of a schedule
that allows for additional development, these new
features added to the user experience for Pet Market.
Becky Sowada, as a QA professional, wanted to ensure
quality wherever possible, but she “recognizes
it as a fact of life” for software development
projects. [Tip #10: Be aware of risks involved
with feature creep and ensure that priorities are
clear. This way the team will have the time to make
the most the most valuable enhancements to your application.]
Wrapping it all up and shipping Pet Market
Finally, it came time to ship Pet Market. The development
team was done, the QA team signed off, and the distibution
bits were delivered to the web team for deployment.
At the end of the process, I asked Becky to reflect
on the experience of QAing and releasing a Rich Internet
Application. Becky indicated that it was easy to apply
standard QA processes to this effort and that the
work itself was comparable to the effort required
for any typical web application. Moreover, the effort
that team spent on the user experience really made
a difference in the quality of Pet Market.
But here is the $64,000 question. Was shipping a
Rich Internet Application easier and cheaper than
deploying a web application? “About the same”
said Becky, “but the user experience is much,
much better.”
Let us know your feedback
on this article.
If you would like to configure the semiautomaster.fla
API Bug Exterminator testing movie to your server,
follow
the configuration instructions.
Download or view the files that accompany this article:
1
Macromedia has confirmed that a binary-post bug introduced
in Mozilla 0.9.6 browsers prevents Macromedia Flash
Remoting from functioning correctly. The affected
browsers include: Netscape 6.x, Opera 6.x, and any
other browsers built on the Mozilla 0.9.6 code base.
This issue was introduced only in these specific versions
and does not exist in earlier versions. Macromedia
is in contact with the Netscape team and has been
informed that the issue has been corrected in Mozilla
1.0, therefore resolving the issue in the Netscape
7.x browser soon to be released. Additionally, Macromedia
is working on a service release to the Flash Player
to work around this bug. Mozilla 0.9.6 browsers account
for approximately 3% of the Flash Player installations
around the world.
|
