Config Server cannot pull from remote repo

Hi,

After almost 3 months without any issues the cloud config server suddenly failed pulling from the repo. This happened silently on Saturday and on Monday we found out after checking the logs that the new config changes were not being picked from the config-server.

Exception thrown: Cannot pull from remote https://repo.git, the working tree is not clean.

No changes were pushed to the repo or new version of cloud server deployed anytime close to the silent crash. The good thing is that the issue can be resolved just by restarting the cloud config server(s). After that everything goes back to normal but now the question is how to monitor and get notified for such issues. It would be nice if there is a HealthIndicator reporting the health of the config server taking into consideration such state or/and the logging level changed to ERROR.

Detailed exception:

2016-12-10 03:47:09,118 [http-nio-8888-exec-5] INFO  o.s.c.a.AnnotationConfigApplicationContext Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@498613ae: startup date [Sat Dec 10 03:47:09 CET 2016]; root of context hierarchy
2016-12-10 03:47:09,123 [http-nio-8888-exec-5] INFO  o.s.c.c.s.e.NativeEnvironmentRepository Adding property source: file:/tmp/config-repo-7239745589329148613/application.properties
2016-12-10 03:47:09,123 [http-nio-8888-exec-5] INFO  o.s.c.a.AnnotationConfigApplicationContext Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@498613ae: startup date [Sat Dec 10 03:47:09 CET 2016]; root of context hierarchy
2016-12-10 03:48:08,949 [http-nio-8888-exec-10] INFO  o.s.c.c.s.e.MultipleJGitEnvironmentRepository Cannot pull from remote https://repo.git, the working tree is not clean.
2016-12-10 03:48:08,983 [http-nio-8888-exec-10] INFO  o.s.c.a.AnnotationConfigApplicationContext Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@11af7f6b: startup date [Sat Dec 10 03:48:08 CET 2016]; root of context hierarchy
2016-12-10 03:48:08,988 [http-nio-8888-exec-10] INFO  o.s.c.c.s.e.NativeEnvironmentRepository Adding property source: file:/tmp/config-repo-7239745589329148613/application.properties
2016-12-10 03:48:08,988 [http-nio-8888-exec-10] INFO  o.s.c.a.AnnotationConfigApplicationContext Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@11af7f6b: startup date [Sat Dec 10 03:48:08 CET 2016]; root of context hierarchy
2016-12-10 03:49:08,954 [http-nio-8888-exec-8] INFO  o.s.c.c.s.e.MultipleJGitEnvironmentRepository Cannot pull from remote https://repo.git, the working tree is not clean.
2016-12-10 03:49:08,981 [http-nio-8888-exec-8] INFO  o.s.c.a.AnnotationConfigApplicationContext Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@7f1733bd: startup date [Sat Dec 10 03:49:08 CET 2016]; root of context hierarchy

Thanks

[redcastleiron]

I tried with ‘force-pull’ property. But problem still being occured. “org.springframework.cloud.config.server.environment.EnvironmentController.defaultLabel(java.lang.String,java.lang.String)]: org.springframework.cloud.config.server.environment.NoSuchRepositoryException: Cannot clone or checkout repository:” Isn’t there any solution?

[marbon87]

@spencergibb Any update on this? Are you working on this problem?

It happened a few times in the last two months that our two instances of the config server were not able to pull the latest version from the git repository and we have to restart the servers. The Health-Indicator actually does not help to monitor this problem, which is a huge problem.