DIR Return Create A Forum - Home
---------------------------------------------------------
Airbattle Games
HTML https://airbattle.createaforum.com
---------------------------------------------------------
*****************************************************
DIR Return to: Wing Leader General Discussion
*****************************************************
#Post#: 3416--------------------------------------------------
Scenario Testing Process
By: pilotofficerprune Date: October 13, 2019, 4:01 pm
---------------------------------------------------------
So, as we are starting to wind down testing on Origins, and
begin working up to Falcons, I want to have a post-mortem
discussion on the last year or two of testing and discussed what
worked and what didn't. What can we do better and what can we do
smarter?
#Post#: 3417--------------------------------------------------
Re: Scenario Testing Process
By: pilotofficerprune Date: October 13, 2019, 4:11 pm
---------------------------------------------------------
So here's a thing that concerns me.
Our scenario development process tends to go roughly like this.
(1) Draft. We draft the scenario, often without testing, or
maybe just setting up and pushing it around a little.
(2) Adjustment. The first four or five tests are critical, as
this is often where most of the adjustment and re-engineering
takes place.
(3) Polish, Validation and Balance. After the scenario is
adjusted, further testing validates the changes, adds polish and
adjusts for balance.
It occurs to me that the skills we need for each stage are
different, but we might not be managing this well. I'll go into
this more in my next post.
#Post#: 3418--------------------------------------------------
Re: Scenario Testing Process
By: pilotofficerprune Date: October 13, 2019, 4:19 pm
---------------------------------------------------------
So, we tend to have testers with different talents. This is not
an exhaustive list, but we have testers who are:
(a) Wreckers. They like to wreck scenarios and are helpful in
making them foolproof.
(b) Engineers. They understand how to engineer good scenario
narratives and are helpful at making suggestions for and testing
changes that improve the scenario experience.
(c) Grinders. Thery are not necessarily skilled at (a) or (b)
but are hella useful for simply playing and generating test
data, particularly for balancing.
Now it occurs to me that each type of tester is good for
different phases of development. For example, Wreckers and
Engineers are really valuable in the Adjustment phase of
scenario development, but the Grinders are really great at doing
the volume testing that validates the scenario and helps balance
it.
#Post#: 3419--------------------------------------------------
Re: Scenario Testing Process
By: pilotofficerprune Date: October 13, 2019, 4:23 pm
---------------------------------------------------------
So, bearing all this I mind, do we need to restructure our
testing? With Origins we recruited a whole bunch of testers back
near the beginning of the year, all of different kinds, and then
launched them on the scenarios. However, those guys who were
great grinders tried to help nurse some scenarios through those
early tests and I wonder if it burned them out a little. Would
it have been more effective to have our core team of Wreckers
and Engineers do the first round of testing on everything, and
only then introduce the rest of the testers to grind their way
through semi-mature scenarios and focus on generating that
balancing data?
Any thoughts?
#Post#: 3423--------------------------------------------------
Re: Scenario Testing Process
By: ScottKey Date: October 13, 2019, 10:32 pm
---------------------------------------------------------
As a 90% grinder, 10% engineer I’d say you might be on to
something.
As you know I don’t vassal and that seems to be a hugely useful
tool in a variety of ways. I am limited as to the times I can
set-up and play and I have also found that to be of any use I
MUST take copious notes and pics of the game. These notes then
have to then be translated into a playtest report, all of which
takes time ((even a small scenario which might play in an hour
usually requires around 150 minutes at the minimum to play ,
document, and write-up) . I have found it to be very enjoyable
and frankly I’ve learned a lot about this time period. However I
do worry about burnout....
The best I can say after a small amount of testing on Eagles and
quite a lot on Origins is that my subjective feeling is that I
am best at small/medium scenarios and that I do my best work
when I give a scenario of that sort at least 3 run-throughs. I
find that I tend to do a “straight” initial run at the scenario
(no real effort to strategize before set-up.) The next two plays
then often involve trying to improve the performance of the
loser pf the first bout. See how the scenario resonates. Put it
into context with other scenarios I have played previously.
But in short... I think you are right that the big push should
probably come later in the process, after the really huge
problems have been identified and, at least initially, fixed.
Then bring in the grinders, giving them specific tasks and goals
so that if they drop it will usually be after they have helped
move things along....
#Post#: 3424--------------------------------------------------
Re: Scenario Testing Process
By: ScottKey Date: October 13, 2019, 10:33 pm
---------------------------------------------------------
It’s late....
I’m tired...
hope that is of some help...
#Post#: 3425--------------------------------------------------
Re: Scenario Testing Process
By: Elias Nordling Date: October 13, 2019, 11:31 pm
---------------------------------------------------------
As 60 percent wrecker, 30 percent engineer, and 10 percent
grinder, I think you are onnto something too :-)
#Post#: 3428--------------------------------------------------
Re: Scenario Testing Process
By: pilotofficerprune Date: October 14, 2019, 1:31 am
---------------------------------------------------------
So my test spreadsheet is based on a simple rule, which is: a
scenario is not 'Good' until:
(a) We have completed at least five plays.
(b) We have had two separate people look at this.
Now, to be honest, with experienced wrecker/engineer testers, we
might not need as many as five plays. But I think having two
separate test teams look at each scenario is vital. For example,
Gordon and Andrew are great engineers, but they are not as
diligent as Elias at trying to break scenarios. If both teams
have played a scenario, I feel confident we've done due
diligence and can then move onto balance testing.
It's this first phase we need to get through, and whether it's
three or four or more tests does not matter. It's getting
trusted people to do it.
#Post#: 3429--------------------------------------------------
Re: Scenario Testing Process
By: pilotofficerprune Date: October 14, 2019, 3:20 am
---------------------------------------------------------
[quote author=ScottKey link=topic=123.msg3423#msg3423
date=1571023932]As a 90% grinder, 10% engineer I’d say you might
be on to something. [/quote]
I think you undersell yourself, Scott.
Wrecking is a talent, one that's sufficiently rare that you must
keep your hands on any wreckers in your crew.
However, engineering is a learned ability. It's mostly a
knowledge of the game, experience with the rules, and the
confidence to make proposals for change. Everyone with a certain
amount of testing under their belts - and people like yourself,
Chris, Kevin, Dave and Pete all qualify - have graduated, and
can play a major part in the adjustment phase of scenarios.
Grinding requires a lot less experience, though grinders can
graduate to engineers over time.
#Post#: 3458--------------------------------------------------
Re: Scenario Testing Process
By: pilotofficerprune Date: October 18, 2019, 12:49 am
---------------------------------------------------------
So, looking at the test log numbers, I don't think they lie.
Eagles required something like 8.4 tests per scenario to get to
a happy place, and that's with a relatively small test team.
Origins will be somewhere in the region of 11.5 to 12 tests by
the time we are done, and I think some of that is misapplied
effort, mostly from new or inexperienced testers. Bear in mind
we were over 100 tests by week 5, and it's taken another 33
weeks to approach 300 tests.
Those are interesting figures, because my original estimate for
testing was 215 tests, which I later revised to 240. I suspect
if we'd kept the early testing (i.e. tests 1 to 5 for each
scenario) to just the core cadre of guys, we might have brought
this thing in with a total closer to those numbers.
I don't want to say that any tests are wasted, because no test
can be waste. But I do think we've been inefficient, and I
reckon that's my fault, for not thinking this out and planning
and organising it right.
Upthread I tried to break down testers into categories, but
perhaps that was a distraction, or possibly unimportant, because
it gets people thinking about specific skill sets. On thinking
about this further, I think the only categories that really
count are:
(1) Experienced, and
(2) Inexperienced.
Pretty much everyone reading this is in category (1). You all
have tens or in some cases hundreds of tests under your belt.
You know the rules set well enough to play the game as it is
intended to be played.
Just as importantly, you've established a cameraderie here on
this forum and are happy to speak up with comments and ideas.
You know that your AARs on every test are read, that you are
listened to and that it all contributes to each scenario's
development. (I know I don't respond to every AAR, but they do
all get my attention and move the needle to some extent.)
So, how do we go forward?
I guess what would be good for Falcons would be to see how many
of you amongst the experienced crew want to help out and support
those guys getting each scenario from no-test to 'we've shaken
out the worst stuff'. And only then, when we're done on that,
invite a larger crew in to do the polish/balance testing.
So, two questions:
(a) Does that seem sound? Or are there alternative strategies?
(b) Who amongst you is willing to stick around help out with the
early stages of Falcons next year?
*****************************************************
DIR Next Page