Gulp.src is terribly slow

I noticed gulp-jshint was 20 times slower than native jshint on my directory tree. I though the problem was due to plugin itself so I kept removing lines until I was left with:

// bench.js
module.exports = function (opt) {
  var i = 0;
  return through.obj(function (file, enc, cb) {
      console.log('got', file.path);

      if (i) {
        console.timeEnd(i);
      }

      i++;
      console.time(i);

      return cb();
  });
};

// gulpfile.js
gulp.task('bench', function () {
  return gulp.src('src/scripts/**/*.js')
    .pipe(bench());

Even doing nothing, it would take 20 seconds on my directory tree. I though the selling point of Gulp was speed so I’m somewhat confused by the results.

I’m not sure if this is related. Of course it may be an issue with node-glob but any performance enhancements Gulp brings to the table are out of the equation if it lags so much enumerating files. So I think this is an important issue for Gulp.

Here’s the benchmark I got. Interestingly, the delays are non-linear: it speeds up at the very end.

[15:11:07] Starting 'bench'...
got src/scripts/app.js
got src/scripts/history.js
1: 567ms
got src/scripts/index.js
2: 618ms
got src/scripts/router.js
3: 124ms
got src/scripts/common/collections.js
4: 15ms
got src/scripts/generated/templates.js
5: 23ms
got src/scripts/common/collections/collection.js
6: 25ms
got src/scripts/common/collections/media_collection.js
7: 17ms
got src/scripts/common/collections/paginated_collection.js
8: 45ms
got src/scripts/common/collections/section_collection.js
9: 35ms
got src/scripts/common/collections/service_account_collection.js
10: 42ms
got src/scripts/common/collections/stamp_collection.js
11: 61ms
got src/scripts/common/collections/user_collection.js
12: 33ms
got src/scripts/common/collections/zine_collection.js
13: 37ms
got src/scripts/common/helpers/ajax.js
14: 44ms
got src/scripts/common/helpers/analytics.js
15: 40ms
got src/scripts/common/helpers/analytics_client.js
16: 36ms
got src/scripts/common/helpers/analytics_events.js
17: 29ms
got src/scripts/common/helpers/closest.js
18: 59ms
got src/scripts/common/helpers/connect_service.js
19: 67ms
got src/scripts/common/helpers/contenteditable.js
20: 60ms
got src/scripts/common/helpers/drag_drop_manager.js
21: 75ms
got src/scripts/common/helpers/drag_drop_operation.js
22: 114ms
got src/scripts/common/helpers/draggable.js
23: 66ms
got src/scripts/common/helpers/droppable.js
24: 67ms
got src/scripts/common/helpers/file_uploader.js
25: 79ms
got src/scripts/common/helpers/focus_polyfill.js
26: 178ms
got src/scripts/common/helpers/get_scroll_y.js
27: 199ms
got src/scripts/common/helpers/gradient.js
28: 195ms
got src/scripts/common/helpers/image_loader.js
29: 196ms
got src/scripts/common/helpers/misc.js
30: 240ms
got src/scripts/common/helpers/noop.js
31: 167ms
got src/scripts/common/helpers/promise_util.js
32: 183ms
got src/scripts/common/helpers/render_icon.js
33: 177ms
got src/scripts/common/helpers/shallow_equal.js
34: 247ms
got src/scripts/common/helpers/share_service.js
35: 542ms
got src/scripts/common/helpers/vendor_prefix.js
36: 444ms
got src/scripts/common/helpers/view_state_manager.js
37: 453ms
got src/scripts/common/models/abstract_model.js
38: 469ms
got src/scripts/common/models/cover_section.js
39: 745ms
got src/scripts/common/models/gallery_section.js
40: 408ms
got src/scripts/common/models/image_media.js
41: 414ms
got src/scripts/common/models/media.js
42: 425ms
got src/scripts/common/models/model.js
43: 436ms
got src/scripts/common/models/poster_section.js
44: 544ms
got src/scripts/common/models/section.js
45: 504ms
got src/scripts/common/models/service_account.js
46: 513ms
got src/scripts/common/models/stamp.js
47: 579ms
got src/scripts/common/models/stamp_content.js
48: 570ms
got src/scripts/common/models/text_section.js
49: 629ms <-------------------------------------------------------------------------------- WAT
got src/scripts/common/models/user.js
50: 284ms
got src/scripts/common/models/zine.js
51: 174ms
got src/scripts/common/models/zine_stamp_state.js
52: 191ms
got src/scripts/common/views/collection_view.js
53: 190ms
got src/scripts/common/views/container_view.js
54: 287ms
got src/scripts/common/views/dim_view.js
55: 362ms
got src/scripts/common/views/menu_view.js
56: 209ms
got src/scripts/common/views/modal_view.js
57: 220ms
got src/scripts/common/views/routing_container_view.js
58: 239ms
got src/scripts/common/views/tab_bar_view.js
59: 232ms
got src/scripts/common/views/tab_container_view.js
60: 216ms
got src/scripts/common/views/view.js
61: 232ms
got src/scripts/components/mixins/anchor_scroll_mixin.js
62: 33ms
got src/scripts/components/mixins/back_color_mixin.js
63: 156ms
got src/scripts/components/mixins/backbone_state_mixin.js
64: 43ms
got src/scripts/components/mixins/class_name_mixin.js
65: 161ms
got src/scripts/components/mixins/delay_action_mixin.js
66: 41ms
got src/scripts/components/mixins/delay_preload_mixin.js
67: 179ms
got src/scripts/components/mixins/form_mixin.js
68: 31ms
got src/scripts/components/mixins/image_queue_mixin.js
69: 153ms
got src/scripts/components/mixins/linked_value_utils.js
70: 35ms
got src/scripts/components/mixins/list_mixin.js
71: 152ms
got src/scripts/components/mixins/pin_it_mixin.js
72: 42ms
got src/scripts/components/mixins/popover_mixin.js
73: 41ms
got src/scripts/components/mixins/pure_render_mixin.js
74: 132ms
got src/scripts/components/mixins/raf_mixin.js
75: 6ms
got src/scripts/components/mixins/responsive_image_mixin.js
76: 6ms
got src/scripts/components/mixins/save_props_mixin.js
77: 7ms
got src/scripts/components/mixins/scroll_state_mixin.js
78: 6ms
got src/scripts/components/mixins/smooth_scroll_mixin.js
79: 11ms
got src/scripts/components/mixins/stamp_link_mixin.js
80: 8ms
got src/scripts/components/mixins/stamp_list_mixin.js
81: 6ms
got src/scripts/components/mixins/transition_events_mixin.js
82: 8ms
got src/scripts/components/mixins/user_link_mixin.js
83: 7ms
got src/scripts/components/mixins/viewport_test_mixin.js
84: 133ms
got src/scripts/components/mixins/zine_link_mixin.js
85: 8ms
got src/scripts/components/mixins/zine_list_mixin.js
86: 10ms
got src/scripts/components/utils/form_link.js
87: 1ms
got src/scripts/editor/collections/draft_collection.js
88: 1ms
got src/scripts/editor/collections/grabbox_collection.js
89: 0ms
got src/scripts/editor/mixins/drag_image_preloader_mixin.js
90: 1ms
got src/scripts/editor/mixins/file_input_owner_mixin.js
91: 1ms
got src/scripts/editor/mixins/tooltip_mixin.js
92: 0ms
got src/scripts/editor/models/grabbox_item.js
93: 3ms
got src/scripts/editor/views/canvas_view.js
94: 2ms
got src/scripts/editor/views/compose_view.js
95: 2ms
got src/scripts/editor/views/grabbox_view.js
96: 2ms
got src/scripts/editor/views/preview_view.js
97: 2ms
got src/scripts/social/presenters/editor_presenter.js
98: 4ms
got src/scripts/social/presenters/social_presenter.js
99: 3ms
got src/scripts/social/presenters/stamp_presenter.js
100: 5ms
got src/scripts/social/views/app_view.js
101: 1ms
got src/scripts/social/views/editor_wrapper_view.js
102: 1ms
got src/scripts/social/views/home_summary_view.js
103: 1ms
got src/scripts/social/views/profile_summary_view.js
104: 0ms
got src/scripts/social/views/social_wrapper_view.js
105: 1ms
got src/scripts/social/views/stamp_wrapper_view.js
106: 1ms
got src/scripts/social/views/zine_summary_view.js
107: 1ms
got src/scripts/social/views/zines_summary_view.js
108: 3ms
got src/scripts/viewer/helpers/font_loader.js
109: 3ms
got src/scripts/viewer/mixins/fading_calculator_mixin.js
110: 2ms
got src/scripts/viewer/mixins/lightbox_mixin.js
111: 2ms
got src/scripts/viewer/mixins/section_mixin.js
112: 3ms
got src/scripts/viewer/mixins/stamp_image_mixin.js
113: 3ms
got src/scripts/viewer/mixins/stamp_scroll_mixin.js
114: 2ms
got src/scripts/viewer/mixins/stamp_transition_mixin.js
115: 133ms
got src/scripts/editor/views/items/add_section_view.js
116: 2ms
got src/scripts/editor/views/items/cover_section_view.js
117: 1ms
got src/scripts/editor/views/items/gallery_media_view.js
118: 0ms
got src/scripts/editor/views/items/gallery_section_view.js
119: 0ms
got src/scripts/editor/views/items/grabbox_dummy_item_view.js
120: 0ms
got src/scripts/editor/views/items/grabbox_item_view.js
121: 1ms
got src/scripts/editor/views/items/poster_background_view.js
122: 0ms
got src/scripts/editor/views/items/poster_section_view.js
123: 0ms
got src/scripts/editor/views/items/section_view.js
124: 0ms
got src/scripts/editor/views/items/text_section_view.js
125: 0ms
[15:11:24] Finished 'bench' after 16 s

[gaearon]

@contra

I was looking for a practical solution that I could integrate today that would help me convince my team to switch to Gulp—it’s a tough sell with such a dramatic performance drop.

I was under impression that the problem lied in readFile/writeFile/stat calls because I put console.time right before fs calls and console.timeEnd inside their callbacks, and it was clearly showing the delays I experienced. That’s why I thought streams weren’t to blame, and fs async call performance was. (And as I said it did help.)

Is there a fault in my reasoning? If so, I’d appreciate pointing me in the right direction. Is highWaterMark somehow involved in inner workings of readFile? I’m new to Node and I thought stream API was more higher-level than fs.readFile and if fs.readFile itself is slow, it’s no use optimizing on higher levels. Am I completely missing the point? I guess fs is using streams under the hood? Man, I should really go RFTM.

I totally dig async-first—but does this only happen on my system? I was actually surprised sync versions worked so well—guess, my pipeline isn’t large enough.

[yocontra]

Okay, well that sounds like an issue with nunjucks. Open an issue with them if their watch stuff is on by default and slowing things down.