A. Some of it is in this page but the most up-to-date information is in Mozilla Releng readthedocs page.
A. In theory people could check in their configuration files here without affecting production configs. This might not be the best place, and we may very well just remove this location. But if this is useful for people to keep their configs in a shareable, easy-to-access-from-mozharness location, we can keep it.
One benefit of keeping development configs here is if someone makes changes (e.g., obsoletes a config variable, changes a config variable, adds a new required config variable, etc.), it's easy to search for config files that need to be updated and either do so or contact the owner.
A. It should go in mozharness/configs.
The problem with putting a production configuration file in mozilla-central is that mozharness needs the config file at the beginning of the script run.
So to run the mozharness script, you need something that knows how to check out mozilla-central (or is it another branch? what branch?) and gets it to the right revision (what revision?) and then calls the mozharness script.
Mozharness scripts currently run under ScriptFactory in buildbot, which essentially clones a repo (hg.m.o/build/mozharness in this case), then runs a command line. If the config file isn't on the slave at this point, we've got problems.
Could we allow for this? Possibly. Unless we have a strong set of reasons to change this, however, production configs should go in mozharness/configs.
(The config file can, however, point at external files for additional configuration. See: buildprops.json, talos.json, etc. But the config file itself should live in mozharness/configs/.)
Non-production configs can live wherever.
A. As a caveat, we see mozharness as an easier way to contribute than anything touching buildbot, for those unfamiliar with our infrastructure. However, we also want to avoid having multiple projects bottleneck on having one or two release engineers staging everything.
We have hooks in buildbotcustom for easier definitions of mozharness tests, at least.
The marionette configs are a good example of how to define additional mozharness tests:
When mozharness unittests and talos merge into mozilla-central ( https://bugzilla.mozilla.org/show_bug.cgi?id=713055 and https://bugzilla.mozilla.org/show_bug.cgi?id=793022 ), the layout of mozilla-tests/config.py may change considerably.
For other non-test automation, you will probably also have to add hooks in buildbotcustom/misc.py. This will probably be a lot more involved, and probably would best involve a discussion with releng first.
A. Currently, Treeherder gets its Buildbot data from http://builddata.pub.build.mozilla.org/buildjson/builds-4hr.js.gz. That json file is generated from buildbot. There is currently no way to inject any additional or different data into that file, however you can use treeherder-client to submit data directly to Treeherder.
In general, when adding a new test, you can add it to Treeherder via something like this:
A new build platform:
A. You can modify testing/mozharness/mozharness.json, point to your own user repository and push it to try. Read more in here.
NOTE: Use hg branch names instead of bookmarks with mozharness.json
A. This depends on what you mean by "in a matter with parity to production".
As we add more mozharness scripts and script families to production, we'll have better answers for these questions, but as we're adding swaths of new automation, we're having some growing pains.
You can test locally, which I encourage. But this isn't necessarily going to test all production platforms, and especially not all production platform configurations.
You can have buildduty loan you a machine, which would definitely make sure the script could run on production platform configurations. If this covered all production platforms, there would be a strong likelihood of things working when rolling out, but it wouldn't necessarily cover the buildbot <-> script interaction.
We also are going to be helping some people set up their own staging environments which would help test the buildbot portion of the automation as well.
A project that may help at some point in the future is mozharness try, which would allow for running scripts in an environment similar to production:
A lot of testing can be done without having these full environments set up, but not all of it. It depends on the scope and level of risk in the change.
(A dirty not-so-secret: our best attempts at staging everything outside production have still missed some testing for large scale projects.
We try to have full staging runs that are "with parity" by doing the same things, but in ways that only affect staging reporting, or staging uploads. This requires having a setup like production but separate. In practice, this tends to miss small testing elements: for instance, whilst we have a staging Treeherder instance, its only for testing Treeherder changes since it still uses production buildbot data. And since our staging pool of machines is so much smaller, we don't always get full end-to-end testing on all platforms and branches on all affected builders/testers.
For full, 100% parity, the job would likely need to be in production itself, with everything that applies. However, this would most likely negatively affect other people. Usually when testing, you want to avoid affecting production in any way.)
A. Absent the above, you can also test locally.
unit.sh runs python nosetests on mozharness. Best practice says we should add tests for whatever new code we write. In practice, this hasn't been the case, but this will become more important as the code base grows and more changes from multiple developers come in.
(It's not always possible to write tests that can be run locally for all tests: if they require a platform or environment you don't have locally, for example. Someone had a suggestion for how to return pre-cooked output from "fake" external scripts or tools for testing; we need to investigate this.)
As you write a script, if you're splitting actions up to be individual chunks as you should, you can test them individually and make sure they do what you expect before moving on to the next one: does this action clobber the directory as expected? Does this one populate the directory tree as expected? Does this one build or package or install or whatever it is, the way it should?
(You can call an action individually by --ACTIONNAME. You can skip an action by --no-ACTIONNAME. You can add an action to the list of default actions by --add-action ACTIONNAME. You can run two or more actions by --ACTIONNAME1 --ACTIONNAME2.)