Wow this thread added like 40 pages since my class an hour ago. Consider bort's dilemma.
Option 1 - do all the testing by himself - he knows the game well, but he has little experience actually doing gameplans for teams and since he is just 1 person he does not think of all the ideas and exploits a mass of 100,000 people can come up with. Thus featues are released with problems and pontential exploits he did not anticipate and people go crazy when people use these exploits and complain that he should have tested them better. Another problem is that before there ever was anyone else on the test server bort only tested existing builds, which meant YOU were always one step ahead of him since he only tested the players you had already made and not the players you could make. Now it is possible to test "ideal" and super builds to find problems and imbalances BEFORE they happen. What if we find that a WR with 100 confidence and 20 catching catches everything (we did not really find this - I made it up )? Wouldn't it be good to know that before someone makes a player like that and exposes the flaw in the game and then we have another dilemma - fix the obvious problem now during the season or "nerf" it inseason? This is essentially the dilemma with the DT slingshot bug where they were getting 10 sacks a game with high enough agility by slingshoting into the backfield. And when a tester finds something like that he does not say to himself YES! Now I have it, then go build a WR on GLB with 100 confidence. He tells bort what is going on then bort comes up with a solution or a new way of programming WRs that makes the attributes more balanced. This also eliminates the worth of the "inside knowledge" the tester just gained since bort just "fixed" something based on what was reported.
Option 2 - Hire a bunch of people who have no players or teams in GLB to test. Well the really obvious problem with this is that getting up to speed on going on about 1 year's worth of GLB knowledge is not easy. You and I take for granted how much of GLB is second nature to us whereas a new person has mountains and mountains of info to learn, especially if he wants to test gameplans, VAs, SAs, and he needs enough command of GLB to come up with exploits before they happen, and enough experience to even understand the type of exploits that could come up. I won't say more about this because the idea of having people totally foreign to GLB test it is so absurd I doubt anyone would seriously propose that.
Option 3 - Choose a bunch of testers from the site but forbid them from playing the game anymore. Well we absolutely get no added compensation for being a tester so there would be no point in testing for a game you can no longer play, not because of the advantage it gives you, but because the game wouldn't pertain to you any more at all. Who would want to test a game they can't play for no compensation of any kind? Also we all love GLB or we wouldn't play it in the first place, so it would be pretty cruel to ask us to give that up for the sake of testing.
Option 4 - The option bort has chosen. He has entrusted to a few people he trusts the job of testing with the understanding that they are working for the betterment of GLB and not for their own personal gain.
Also I want to briefly address the scenario some people have mentioned of testing what a player would look like with this or that build. First of all, I know it sounds like a simple thing to say, what if my player had 15 more confidence, what would that do? But think about what you would really have to do to set this up. You could make that character easy enough, but to test what he would do? Against who? Against what? You would have to set up a whole team to test it out against. For example, let's say you want to test what your QB could do with 15 more confidence. Well what type of team are you playing? We have absolutely no ability to rip teams from the game to test against so you would have to set up a team yourself. What kind of team would they be? Pass rushing? Fast CBs? Run stopping? In other words what is a "typical" team you would test against or is there even any such thing as a "typical" team? Most teams on GLB right now are a blend of agents and builds, some very good, some average, some below average and many teams are quite different from each other. Then for your own team, how do you set them up? Consider the possibility that you could make your own offense too good because the real issue would be how to interpret your test results? What if your QB plays great? What does that mean? Was that simply due to adding 15 confidence to him? Did you make your D too bad, or your O too good? And if you learn that 15 confidence makes your QB better against the team you played, does that translate to GLB? What is the likelihood that any teams are similar to the one you made? Well surely some will be similar, but not all and not all in the same way. You could play a SA based team and still not have enough confidence. Then again, and this is the real point, did you really learn anything about confidence from your test? I think everyone would concede that since confidence is a major attribute for QBs, adding 15 more of it would make him play better. But exactly how much better did he play? And how did he play better? Did he seem to deteriorate less over the game, or did he throw farther, avoid sacks better, etc etc etc. What does your test tell you about confidence for a QB that you didn't really know already that would give you a huge advantage over anyone else?
I hope this case study is helpful. My point is that while it seems like a simple matter to say, Well a tester could just "preview" his next boost to see what is most effective, this would be an extremely complex and time-consumming thing to do, and I would seriously question the quality of any results you would get from that because of huge problems interpreting your results.