Buildgen doesn't work on Spindle?

It looks like buildgen in fsqio doesn’t seem to be updating dependencies in src/thrift:

Omers-MacBook-Pro:fsqio omer$ git status
On branch master
Your branch is up-to-date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   src/thrift/io/fsq/twofishes/BUILD

no changes added to commit (use "git add" and/or "git commit -a")
Omers-MacBook-Pro:fsqio omer$ git diff src/thrift/io/fsq/twofishes/BUILD
diff --git a/src/thrift/io/fsq/twofishes/BUILD b/src/thrift/io/fsq/twofishes/BUILD
index 2846792..dbcaa37 100644
--- a/src/thrift/io/fsq/twofishes/BUILD
+++ b/src/thrift/io/fsq/twofishes/BUILD
@@ -3,9 +3,6 @@
 scala_record_library(
   name = 'twofishes',
   dependencies = [
-    '3rdparty:bson',
-    '3rdparty:twitter-util',
-    'src/jvm/io/fsq/spindle/runtime',
   ],
   sources = globs('*.thrift'),
   provides=scala_artifact(
Omers-MacBook-Pro:fsqio omer$ ./pants buildgen

22:31:09 00:00 [main]
               (To run a reporting server: ./pants server)
22:31:09 00:00   [setup]
22:31:09 00:00     [parse]
               Executing tasks in goals: tag -> bootstrap -> imports -> unpack-jars -> validate -> deferred-sources -> build-spindle -> jvm-platform-validate -> gen -> webpack -> map-java-exported-symbols -> map-scala-exported-symbols -> map-jvm-symbol-to-source-tree -> resolve -> map-third-party-jar-symbols -> map-scala-used-symbols -> map-sources-to-addresses-mapper -> map-scala-library-used-addresses -> map-python-exported-symbols -> map-derived-targets -> buildgen
22:31:10 00:01   [tag]
22:31:10 00:01     [tag]
22:31:10 00:01   [bootstrap]
22:31:10 00:01     [substitute-aliased-targets]
22:31:10 00:01     [bootstrap-jvm-tools]
22:31:10 00:01     [provide-tools-jar]
22:31:10 00:01     [global-jar-dependency-management]
22:31:10 00:01   [imports]
22:31:10 00:01     [ivy-imports]
22:31:10 00:01   [unpack-jars]
22:31:11 00:02     [unpack-jars]
22:31:11 00:02   [validate]
22:31:11 00:02     [validate]
22:31:11 00:02   [deferred-sources]
22:31:11 00:02     [deferred-sources]
22:31:11 00:02   [build-spindle]
22:31:11 00:02     [build-spindle]
22:31:11 00:02   [jvm-platform-validate]
22:31:11 00:02     [jvm-platform-validate]
22:31:11 00:02   [gen]
22:31:11 00:02     [antlr-java]
22:31:11 00:02     [antlr-py]
22:31:11 00:02     [jaxb]
22:31:11 00:02     [protoc]
22:31:11 00:02     [ragel]
22:31:11 00:02     [thrift-java]
22:31:11 00:02     [thrift-py]
22:31:11 00:02     [wire]
22:31:11 00:02     [go-thrift]
22:31:11 00:02     [spindle]
22:31:11 00:02     [validate-graph]
22:31:11 00:02   [webpack]
22:31:11 00:02     [webpack-resolve]
22:31:11 00:02     [webpack-gen]
22:31:12 00:03   [map-java-exported-symbols]
22:31:12 00:03     [map-java-exported-symbols]
22:31:12 00:03   [map-scala-exported-symbols]
22:31:12 00:03     [map-scala-exported-symbols]
22:31:12 00:03   [map-jvm-symbol-to-source-tree]
22:31:12 00:03     [map-jvm-symbol-to-source-tree]
22:31:12 00:03   [resolve]
22:31:12 00:03     [ivy]
22:31:13 00:04     [go]
22:31:13 00:04   [map-third-party-jar-symbols]
22:31:13 00:04     [map-third-party-jar-symbols]
22:31:13 00:04   [map-scala-used-symbols]
22:31:13 00:04     [map-scala-used-symbols]
22:31:13 00:04   [map-sources-to-addresses-mapper]
22:31:13 00:04     [map-sources-to-addresses-mapper]
22:31:13 00:04   [map-scala-library-used-addresses]
22:31:13 00:04     [map-scala-library-used-addresses]
22:31:13 00:04   [map-python-exported-symbols]
22:31:13 00:04     [map-python-exported-symbols]
22:31:13 00:04   [map-derived-targets]
22:31:13 00:04     [map-derived-targets]
22:31:13 00:04   [buildgen]
22:31:13 00:04     [go]
22:31:13 00:04     [buildgen]
22:31:13 00:04     [aggregate-targets]
22:31:13 00:04     [scala]WARN] BuildFileManipulator would have added test/thrift/io/fsq/spindle/codegen/parser/test:test as a dependency of test/jvm/io/fsq/spindle/codegen/binary/test:test, but that dependency was already forced with a comment.

22:31:13 00:04     [python]
22:31:14 00:05   [complete]
               SUCCESS

I recall it working there and see there’s some code in buildgen referring to Spindle.

[mateor]

How I tested that in Fsqio.

1. git pull fsqio master

2. Add the above pants.ini change (it already has the BUILD.tools part in the BUILD.opensource file)

3. Delete some dependencies from scala_record_library files:

4. Changed some scala_record_library and buildgen_bag BUILD files and run buildgen

    modified:   src/thrift/io/fsq/BUILD
    modified:   src/thrift/io/fsq/common/thrift/descriptors/services/gen/BUILD
   
   $ git pull fsqio master
   $ rm -rf src/python/fsqio/pants/buildgen
   $ rm -rf  test/python/fsqio_test/pants/buildgen
   $ rm -rf ~/.cache/fsqio
   $ ./pants buildgen
   

[mateor]

👋

Yeah, I forgot honestly, although I thought I had gotten enough of the general info in that thread, all that was missing was the exact current package name.

Anyway, it was due. I updated fsqio tonight and also configured a PyPi module for fsqio.pants.buildgen.spindle.

If you check out Fsq.io master, you should be able to fully demo consuming buildgen from hosted repo with the following small diff

mateo-2:fsqio mateo$ git diff --cached
diff --git a/BUILD.tools b/BUILD.tools
new file mode 100644
index 00000000..c9980895
--- /dev/null
+++ b/BUILD.tools
@@ -0,0 +1,15 @@
+
+jar_library(
+  name = 'buildgen-emit-exported-symbols',
+  jars = [
+    scala_jar(org = 'io.fsq', name = 'buildgen-emit-exported-symbols', rev = '1.2.0'),
+  ],
+)
+
+
+jar_library(
+  name = 'buildgen-emit-used-symbols',
+  jars = [
+    scala_jar(org = 'io.fsq', name = 'buildgen-emit-used-symbols', rev = '1.2.0'),
+  ],
+)
\ No newline at end of file
diff --git a/pants.ini b/pants.ini
index 50241c8a..332325c1 100644
--- a/pants.ini
+++ b/pants.ini
@@ -12,6 +12,13 @@ pants_version: 1.3.1rc1
 verify_config: True
 kill_nailguns: True

+plugins: [
+    "fsqio.pants.buildgen.core==1.3.1",
+    "fsqio.pants.buildgen.jvm==1.3.1",
+    "fsqio.pants.buildgen.python==1.3.1",
+    "fsqio.pants.buildgen.spindle==1.3.1",
+  ]
+
 backend_packages: +[
     'pants.contrib.go',
     'pants.contrib.node',