Running pivy-importer fails on Mac

I attempted to run pivy-importer on my MacBook Pro 2015, Siera v. 10.12.3, Java 1.8.0_112 and ended up with this error

MUSRS186002-942:tmp rs186002$ java -jar pivy-importer-0.4.1-all.jar --repo ~/tmp/pypi virtualenv:15.0.1 pip:7.1.2 --replace alabaster:0.7=alabaster:0.7.1,pytz:0a=pytz:2016.4,Babel:0.8=Babel:1.0,sphinx_rtd_theme:0.1=sphinx_rtd_theme:0.1.1
alabaster:0.7=alabaster:0.7.1
pytz:0a=pytz:2016.4
Babel:0.8=Babel:1.0
sphinx_rtd_theme:0.1=sphinx_rtd_theme:0.1.1
12:53:02.579 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in virtualenv:15.0.1
12:53:04.256 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in pip:7.1.2
12:53:04.777 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in pytest:3.0.6
12:53:05.196 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in virtualenv:1.10
12:53:05.872 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in scripttest:1.3
12:53:05.936 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in mock:2.0.0
12:53:06.671 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in py:1.4.29
12:53:06.825 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in setuptools:34.1.1
12:53:06.940 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in argparse:1.4.0
12:53:07.010 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in colorama:0.3.7
12:53:07.076 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in pbr:0.11.0
12:53:07.161 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in six:1.9.0
12:53:07.228 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in funcsigs:1.0.0
12:53:07.354 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in Sphinx:1.5.2
12:53:09.066 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in Jinja2:2.7
12:53:09.386 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in Pygments:2.0
12:53:10.152 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in Sphinx:1.3
12:53:11.369 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in unittest2:1.1.0
12:53:11.506 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in pip:9.0.1
12:53:11.748 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in ordereddict:1.1
12:53:11.811 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in six:1.5.0
12:53:11.883 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in Jinja2:2.3
12:53:12.307 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in docutils:0.11
12:53:12.514 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in snowballstemmer:1.1.0
12:53:12.637 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in Babel:1.3
12:53:13.104 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in alabaster:0.7.1
12:53:13.169 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in imagesize:0.7.1
12:53:13.232 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in requests:2.4.0
12:53:13.403 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in colorama:0.3.5
12:53:13.473 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in nose:1.3.7
12:53:13.568 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in simplejson:3.10.0
12:53:13.638 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in html5lib:1.0b10
Exception in thread "main" java.lang.RuntimeException: Unable to find version 2.2 for chardet
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:80)
	at org.codehaus.groovy.reflection.CachedConstructor.doConstructorInvoke(CachedConstructor.java:74)
	at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrap.callConstructor(ConstructorSite.java:84)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:60)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:235)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:247)
	at com.linkedin.python.importer.pypi.ProjectDetails.maybeFixVersion(ProjectDetails.groovy:55)
	at com.linkedin.python.importer.pypi.ProjectDetails$maybeFixVersion.call(Unknown Source)
	at com.linkedin.python.importer.distribution.SourceDistPackage$_parseRequiresText_closure1.doCall(SourceDistPackage.groovy:81)
	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019)
	at groovy.lang.Closure.call(Closure.java:426)
	at groovy.lang.Closure.call(Closure.java:442)
	at org.codehaus.groovy.runtime.DefaultGroovyMethods.callClosureForLine(DefaultGroovyMethods.java:5236)
	at org.codehaus.groovy.runtime.StringGroovyMethods.eachLine(StringGroovyMethods.java:615)
	at org.codehaus.groovy.runtime.StringGroovyMethods.eachLine(StringGroovyMethods.java:636)
	at org.codehaus.groovy.runtime.StringGroovyMethods.eachLine(StringGroovyMethods.java:595)
	at org.codehaus.groovy.runtime.dgm$1060.invoke(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
	at com.linkedin.python.importer.distribution.SourceDistPackage.parseRequiresText(SourceDistPackage.groovy:50)
	at com.linkedin.python.importer.distribution.SourceDistPackage.getDependencies(SourceDistPackage.groovy:41)
	at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.MethodMetaProperty$GetBeanMethodMetaProperty.getProperty(MethodMetaProperty.java:76)
	at org.codehaus.groovy.runtime.callsite.GetEffectivePogoPropertySite.callGroovyObjectGetProperty(GetEffectivePogoPropertySite.java:68)
	at com.linkedin.python.importer.deps.DependencyDownloader.downloadDependency(DependencyDownloader.groovy:72)
	at com.linkedin.python.importer.deps.DependencyDownloader$downloadDependency.callCurrent(Unknown Source)
	at com.linkedin.python.importer.deps.DependencyDownloader.download(DependencyDownloader.groovy:50)
	at com.linkedin.python.importer.ImporterCLI.main(ImporterCLI.java:73)

[bostone]

For project that advertise itself as dependency resolution solution I found it strange that I have to jump over multiple hoops just to import a single library. It’s not a critique but rather an observation. I really think this needs to be described in more details right away so not to build false expectation. In my personal opinion this project can only really take off if it provides easy way of plugging into official and/or Artifactory-based PyPi repo. Initially I thought that it was enough to provide some definition in the “repositories” section. Then I red in the documentation that I need to use pivy-importer and now both of these things are not (essentially) true. It’s really frustrating. For light-medium Python project it seems to me that I’m better off just writing few gradle tasks that would run some python scripts and that’s a pity because I really like the premise of this project

[zvezdan]

@bostone The pyvi-importer is just an example to get people started quickly and provides just the packages that were needed for our CI runs triggered by commits here.

Internally, at LinkedIn, we have a Python product that uploads external packages into our artifactory + Ivy files created by analyzing the dependencies. Thus, when our developers need a new package, we can easily get it (and its transitive dependencies) into our artifactory (if they are not already there).

Being a Python program, this tool is much better at analyzing the needs of Python packages than pyvi-importer and it has various options. This is a much more sophisticated program and highly customized for internal LinkedIn use, so we did not open-source it. We expect that enterprise users will be able to build something similar for their own needs. I had been contacted by some who used pyvi-importer as a guidance to build a very simple Python program that uploads the dependencies to their custom artifact repository.

If the community feels the need for a tool like this, we can look into producing a simpler open-source version of it, but the timeline for that would depend on our priorities.