Alright, now that I have some time to rededicate to testing again, I want to get some feedback on the testing system so far. Specifically, I'm looking at feedback related to playtest time and format, the method of downloading (SVN), and reporting bugs (mantis), etc. If you don't use one of the above or don't attend playtests regularly, please state why (e.g. the playtest was at a time I couldn't attend). The more feedback (constructive only please) you give, the better we can make playtesting and Empires.
SVN worked ok. It was just a pain for the first time because it was about a 2GB download but after that the updates were pretty fast. Mantis: It worked ok. I like the fact that sends email notifications so you know what is going on. However it is not clear to me what is the release criteria (maximum number of critical bugs?) Playtests and other comments: - Time for testing was fine. It is helpful to have a set schedule. - It is really helpful if one of the developers is in the server at the scheduled test time and give specific instructions on what to test. - Update the MOTD to state what is the testing about and if there are any special settings or scripts running in the server. - Please state specific testing instructions for each session so the testers are not running around like headless chickens. - Give detailed instructions on how to access the testing server, how to set up your directories in your HD, how to install from the SVN, general mantis guidelines. It took some time for me to figure all the details and it would be less painful if there is information on how to set everything up. - Please please please have the default variables and most updated balanced scripts running for the scheduled tests. Otherwise testers might think that something is just not right when actually it could be that some setting was set to an absurd value and not a bug. (This happened a lot during 2.25 testing). If some special script value is to be tested, then please state it in the MOTD.
Why do we test? If the answers is finding bugs then the current system is fine. Everybody just trolls around with all kind of items. But if we want to test balance issues then we need much more organization. Someone has to be the test leader. This person has to prepare and define the test settings and also define the goal of the test session. But I doubt that this will work. Releasing balance changes untested and fix them afterwards with small releases seems to work best. And the SVN download works fine but it is also a little bit complecated to setup. If we want more testers a download link to the complete package would help.
We had quite a few scheduled tests where no one was around to lead them, or just it wasn't at any kind of time where people would join. They were empty tests, so I started to ignore them. It's great having a lot of test events, but if they're just spammed and no one's around...people will ignore them. Go for 2-3 tests per week.
Tests are best in a public environment. Script tests should be done on the current version where possible, and then use script manifest so that you get a better test. No balance testing in a testing version can ever be successful. You need a public server with skills varying from newbie to pro. Bug testing only needs a few people though.
