The JUnit XML format lends itself to be presented in a powerful UI,
where you can drill down to the information you are interested in very
quickly.
For test failures, this usually means that you want to see the detailed
trace of the failing tests.
With Travis CI, we passed the `--verbose-log` option to get those
traces. However, that seems excessive, as we do not need/use the logs in
almost all of those cases: only when a test fails do we have a way to
include the trace.
So let's do something different when using Azure DevOps: let's run all
the tests with `--quiet` first, and only if a failure is encountered,
try to trace the commands as they are executed.
Of course, we cannot turn on `--verbose-log` after the fact. So let's
just re-run the test with all the same options, adding `--verbose-log`.
And then munging the output file into the JUnit XML on the fly.
Note: there is an off chance that re-running the test in verbose mode
"fixes" the failures (and this does happen from time to time!). That is
a possibility we should be able to live with. Ideally, we would label
this as "Passed upon rerun", and Azure Pipelines even know about that
outcome, but it is not available when using the JUnit XML format for
now:
https://github.com/Microsoft/azure-pipelines-agent/blob/master/src/Agent.Worker/TestResults/JunitResultReader.cs
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This should be more reliable than the current method, and prepares the
test suite for a consistent way to clean up before re-running the tests
with different options.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This makes use of the just-introduced consistent way to specify that a
long-running process needs to be terminated at the end of a test script
run.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When running the p4 daemon or `git daemon`, we want to kill it at the
end of the test script.
So far, we do this "manually".
However, in the next few commits we want to teach the test suite to
optionally re-run scripts with different options, therefore we will have
to have a consistent way to stop daemons.
Let's introduce `test_atexit`, which is loosely modeled after
`test_when_finished` (but has a broader scope: rather than running the
commands after the current test case, run them when the test script
finishes, and also run them when the `--immediate` option is in effect).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
As Unix shell scripting comes at a hefty price on Windows, we have to
see where we can save some time to run the test suite.
Let's skip the chain linting and the bin-wrappers/ redirection on
Windows; this seems to shave of anywhere between 10-30% from the overall
runtime.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This commit adds an azure-pipelines.yml file which is Azure DevOps'
equivalent to Travis CI's .travis.yml.
To make things a bit easier to understand, we refrain from using the
`matrix` feature here because (while it is powerful) it can be a bit
confusing to users who are not familiar with CI setups. Therefore, we
use a separate phase even for similar configurations (such as GCC vs
Clang on Linux, GCC vs Clang on macOS).
Also, we make use of the shiny new feature we just introduced where the
test suite can output JUnit-style .xml files. This information is made
available in a nice UI that allows the viewer to filter by phase and/or
test number, and to see trends such as: number of (failing) tests, time
spent running the test suite, etc.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This patch introduces a conditional arm that defines some environment
variables and a function that displays the URL given the job id (to
identify previous runs for known-good trees).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This will come in handy when publishing the results of Git's test suite
during an automated Azure DevOps run.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
In the next commit, we want to teach Git's test suite to optionally
output test results in JUnit-style .xml files. These files contain
information about the time spent. So we need a way to measure time.
While we could use `date +%s` for that, this will give us only seconds,
i.e. very coarse-grained timings.
GNU `date` supports `date +%s.%N` (i.e. nanosecond-precision output),
but there is no equivalent in BSD `date` (read: on macOS, we would not
be able to obtain precise timings).
So let's introduce `test-tool date getnanos`, with an optional start
time, that outputs preciser values.
Granted, it is a bit pointless to try measuring times accurately in
shell scripts, certainly to nanosecond precision. But it is better than
second-granularity.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
It seems that on some build agents, there might be some transient file
locking issues, so when we cannot delete the trash directory, let's be
gentle and try again five seconds later, and only error out if it still
fails the second time.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Symbolic links are still not quite as easy to use on Windows as on Linux
(for example, on versions older than Windows 10, only administrators can
create symlinks, and on Windows 10 you still need to be in developer
mode for regular users to have permission), but NTFS junctions can give
us a way out.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Let's not decide in the generic ci/ script how many jobs to run in
parallel; it is easy enough to hand that information down via the
`MAKEFLAGS`.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The upcoming patches will allow building git.git via Azure Pipelines
(i.e. Azure DevOps' Continuous Integration), where variable names and
URLs look a bit different than in Travis CI.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The name is hard-coded to reflect that we use Travis CI for continuous
testing.
In the next commits, we will extend this to be able use Azure DevOps,
too.
So let's adjust the name to make it more generic.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This speeds up the tests by a bit on Windows, where running Unix shell
scripts (and spawning processes) is not exactly a cheap operation.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
When building Git with RUNTIME_PREFIX and starting a test helper from
t/helper/, it fails to detect the system prefix correctly.
This is the reason that the warning
RUNTIME_PREFIX requested, but prefix computation failed. [...]
to be printed.
In t0061, we did not expect that to happen, and it actually did not
happen in the normal case, because bin-wrappers/test-tool specifically
sets GIT_TEXTDOMAINDIR (and as a consequence, nothing in test-tool wants
to know about the runtime prefix).
However, with --with-dashes, bin-wrappers/test-tool is no longer called,
but t/helper/test-tool is called directly.
So let's just ignore the RUNTIME_PREFIX warning.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
We really need to be able to find the test helpers... Really. This
change was forgotten when we moved the test helpers into t/helper/
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Technically, this topic branch would need a little bit more polishing.
However, it consists largely of fixup! commits which will be squashed
into the original `azure-pipelines` branch, anyway. And v2.20.1 is out
already, so the rebase is imminent. So...
There are two things to do, still, before submitting a new iteration of
that patch series:
- We need a better way to encode the text we paste into the XML.
Probably we should extend the test-tool.
- We will also want to replace the perl invocation to retrieve the file
size by a test-tool command.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
A recent update accidentally squelched an error message when the
run_command API failed to run a missing command, which has been
corrected.
* jc/run-command-report-exec-failure-fix:
run-command: report exec failure
"git help -a" did not work well when an overly long alias is
defined, which has been corrected.
* js/help-commands-verbose-by-default-fix:
help -a: handle aliases with long names gracefully
help.h: fix coding style
The new test_oid machinery in the test library requires reading
some information from t/oid-info/hash-info and t/oid-info/oid.
The logic to read from these files in shell uses built-in "read"
command, which leaves CR at the end of these text files when they
are checked out with CRLF line endings, at least when run with bash
shipped with Git for Windows. This results in an unexpected value
in the variable these lines are read into, leading the tests to
fail.
Mark them to be checked out always with the LF line endings.
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The oneline notwithstanding, 13374987dd (completion: use _gitcompbuiltin
for format-patch, 2018-11-03) changed also the way send-email options
are completed, by asking the git send-email command itself what options
it offers.
Necessarily, this must fail when built with NO_PERL because send-email
itself is a Perl script. Which means that we need the PERL prerequisite
for the send-email test case in t9902.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
As Unix shell scripting comes at a hefty price on Windows, we have to
see where we can save some time to run the test suite.
Let's skip the chain linting and the bin-wrappers/ redirection on
Windows; this seems to shave of anywhere between 10-30% from the overall
runtime.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Actually, seems like \x1c is never valid in an XML... so let's use the
replacement character for such values.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The t9902 test puts funny characters into its output, like \x1c. Let's
translate them into XML attributes, too.
At some stage, we will just want to add a function to the `test-tool` to
perform this for us, instead of that ugly shell script construct that is
incomplete to begin with.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
It is actually better not to modify (or even truncate) the .out files
because the ci/ scripts want to provide a full output in the log.
So let's just remember the file offset at the beginning of each test
case, and use `cut <offset>-` to use that specifically.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
As we use the common ci/ scripts to build and test on Windows, we now
have to make sure that there are no untracked, unignored files after the
build. That includes the SDK, and the test-cache directory.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Let's really use the common ci/ scripts to build and test on Windows,
meaning that we do not source `ci/lib.sh` ourselves.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
It might be that mkpasswd cannot help speed up the tests on Windows, but
then, it won't hurt.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Ignore failures to mount the network share. It is purely to speed up
recurrent builds, and it is only in effect in non-forked builds, anyway,
so it is in a way optional.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Let's split the Windows phase into multiple, logical tasks so that we
can quickly determine whether the build failed, or the test suite, or
whether there was a problem with initializing the SDK.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
It seems that on some build agents, there might be some transient file
locking issues, so when we cannot delete the trash directory, let's be
gentle and try again five seconds later, and only error out if it still
fails the second time.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Symbolic links are still not quite as easy to use on Windows as on Linux
(for example, on versions older than Windows 10, only administrators can
create symlinks, and on Windows 10 you still need to be in developer
mode for regular users to have permission), but NTFS junctions can give
us a way out.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Let's not decide in the generic ci/ script how many jobs to run in
parallel; it is easy enough to hand that information down via the
`MAKEFLAGS`.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Turns out that there are problems with that approach that are totally
not related to Azure Pipelines, so in order to focus on the latter,
let's skip the test_atexit patches for now.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
