From 40f3f25c94ae84d9f506b4312cd51dc64588e184 Mon Sep 17 00:00:00 2001 From: Johannes Schindelin Date: Wed, 23 Nov 2016 20:59:44 +0100 Subject: [PATCH 1/2] Export the preload_index() function The purpose of this function is to stat() the files listed in the index in a multi-threaded fashion. It is called directly after reading the index in the read_index_preloaded() function. However, in some cases we may want to separate the index reading from the preloading step, e.g. in builtin/add.c, where we need to load the index before we parse the pathspecs (which needs to error out if one of the pathspecs refers to a path within a submodule, for which the index must have been read already), and only then will we want to preload, possibly limited by the just-parsed pathspecs. So let's just export that function to allow calling it separately. Signed-off-by: Johannes Schindelin --- cache.h | 1 + preload-index.c | 6 ++---- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/cache.h b/cache.h index f49d9fbac8..c0319669ed 100644 --- a/cache.h +++ b/cache.h @@ -659,6 +659,7 @@ extern int daemonize(void); /* Initialize and use the cache information */ struct lock_file; extern int read_index(struct index_state *); +extern void preload_index(struct index_state *, const struct pathspec *pathspec); extern int read_index_preload(struct index_state *, const struct pathspec *pathspec); extern int do_read_index(struct index_state *istate, const char *path, int must_exist); /* for testting only! */ diff --git a/preload-index.c b/preload-index.c index 9c22a0705f..6857573ce6 100644 --- a/preload-index.c +++ b/preload-index.c @@ -7,8 +7,7 @@ #include "fsmonitor.h" #ifdef NO_PTHREADS -static void preload_index(struct index_state *index, - const struct pathspec *pathspec) +void preload_index(struct index_state *index, const struct pathspec *pathspec) { ; /* nothing */ } @@ -73,8 +72,7 @@ static void *preload_thread(void *_data) return NULL; } -static void preload_index(struct index_state *index, - const struct pathspec *pathspec) +void preload_index(struct index_state *index, const struct pathspec *pathspec) { int threads, i, work, offset; struct thread_data data[MAX_PARALLEL]; From 08a7003e669df5ab58adb4ddfc2fdacee1316286 Mon Sep 17 00:00:00 2001 From: Jeff Hostetler Date: Tue, 22 Nov 2016 11:26:38 -0500 Subject: [PATCH 2/2] add: use preload-index and fscache for performance Teach "add" to use preload-index and fscache features to improve performance on very large repositories. During an "add", a call is made to run_diff_files() which calls check_remove() for each index-entry. This calls lstat(). On Windows, the fscache code intercepts the lstat() calls and builds a private cache using the FindFirst/FindNext routines, which are much faster. Somewhat independent of this, is the preload-index code which distributes some of the start-up costs across multiple threads. We need to keep the call to read_cache() before parsing the pathspecs (and hence cannot use the pathspecs to limit any preload) because parse_pathspec() is using the index to determine whether a pathspec is, in fact, in a submodule. If we would not read the index first, parse_pathspec() would not error out on a path that is inside a submodule, and t7400-submodule-basic.sh would fail with not ok 47 - do not add files from a submodule We still want the nice preload performance boost, though, so we simply call read_cache_preload(&pathspecs) after parsing the pathspecs. Signed-off-by: Jeff Hostetler Signed-off-by: Johannes Schindelin --- builtin/add.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/builtin/add.c b/builtin/add.c index 9916498a29..91333724f6 100644 --- a/builtin/add.c +++ b/builtin/add.c @@ -461,6 +461,10 @@ int cmd_add(int argc, const char **argv, const char *prefix) die_path_inside_submodule(&the_index, &pathspec); + enable_fscache(1); + /* We do not really re-read the index but update the up-to-date flags */ + preload_index(&the_index, &pathspec); + if (add_new_files) { int baselen;