It makes a huge difference.
For what it's worth, I'm working on a project with about 1000 dependencies and incremental debug build time is about 4 seconds on my laptop.
That's already pretty good.
Number of dependencies likely won't affect incremental build times except for linking and replacing it might offer some good gains for incremental builds.
Default linker is quite a bit slower.
It makes a huge difference.
For what it's worth, I'm working on a project with about 1000 dependencies and incremental debug build time is about 4 seconds on my laptop.
That's already pretty good.