Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Internal error processing get_table_req
See original GitHub issueHere is the environment.
- waggle-dance version : 2.3.7 (also tested on 2.3.6)
- waggle-dance configuration
- database-resolution: PREFIXED
- primary-meta-store :
- hive & metastore version : 2.3.0
thrift://localhost:9083
- remote-meta-store
- hive & metastore version : 2.1.0
thrift://remote-ip:9083
I can see the remote db and tables with ‘prod_’ prefix which I set. (I executed use prod_test;)
But when I execute a select query for a table(select * from test_1;), it makes error.
Here is the logs (and stack trace)
2018-07-25 23:25:29.500 INFO 28387 --- [ main] o.h.v.i.x.ValidationBootstrapParameters : HV000006: Using org.hibernate.validator.HibernateValidator as validation provider.
2018-07-25 23:25:29.921 INFO 28387 --- [ main] s.w.s.m.m.a.RequestMappingHandlerAdapter : Looking for @ControllerAdvice: org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@4f4a7090: startup date [Wed Jul 25 23:25:24 UTC 2018]; root of context hierarchy
2018-07-25 23:25:30.031 INFO 28387 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/api/admin/federations],methods=[GET]}" onto public java.util.List<com.hotels.bdp.waggledance.api.model.AbstractMetaStore> com.hotels.bdp.waggledance.rest.endpoint.FederationsAdminController.federations()
2018-07-25 23:25:30.033 INFO 28387 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/api/admin/federations],methods=[POST]}" onto public void com.hotels.bdp.waggledance.rest.endpoint.FederationsAdminController.add(com.hotels.bdp.waggledance.api.model.AbstractMetaStore)
2018-07-25 23:25:30.033 INFO 28387 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/api/admin/federations/{name}],methods=[DELETE]}" onto public void com.hotels.bdp.waggledance.rest.endpoint.FederationsAdminController.remove(java.lang.String)
2018-07-25 23:25:30.033 INFO 28387 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/api/admin/federations/{name}],methods=[GET]}" onto public com.hotels.bdp.waggledance.api.model.AbstractMetaStore com.hotels.bdp.waggledance.rest.endpoint.FederationsAdminController.read(java.lang.String)
2018-07-25 23:25:30.035 INFO 28387 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error],produces=[text/html]}" onto public org.springframework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
2018-07-25 23:25:30.036 INFO 28387 --- [ main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error]}" onto public org.springframework.http.ResponseEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.boot.autoconfigure.web.BasicErrorController.error(javax.servlet.http.HttpServletRequest)
2018-07-25 23:25:30.111 INFO 28387 --- [ main] o.s.w.s.h.SimpleUrlHandlerMapping : Mapped URL path [/webjars/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2018-07-25 23:25:30.111 INFO 28387 --- [ main] o.s.w.s.h.SimpleUrlHandlerMapping : Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2018-07-25 23:25:30.200 INFO 28387 --- [ main] o.s.w.s.h.SimpleUrlHandlerMapping : Mapped URL path [/**/favicon.ico] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2018-07-25 23:25:31.236 INFO 28387 --- [ main] o.s.b.a.e.m.EndpointHandlerMapping : Mapped "{[/beans || /beans.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-07-25 23:25:31.238 INFO 28387 --- [ main] o.s.b.a.e.m.EndpointHandlerMapping : Mapped "{[/env/{name:.*}],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EnvironmentMvcEndpoint.value(java.lang.String)
2018-07-25 23:25:31.239 INFO 28387 --- [ main] o.s.b.a.e.m.EndpointHandlerMapping : Mapped "{[/env || /env.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-07-25 23:25:31.240 INFO 28387 --- [ main] o.s.b.a.e.m.EndpointHandlerMapping : Mapped "{[/configprops || /configprops.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-07-25 23:25:31.241 INFO 28387 --- [ main] o.s.b.a.e.m.EndpointHandlerMapping : Mapped "{[/trace || /trace.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-07-25 23:25:31.244 INFO 28387 --- [ main] o.s.b.a.e.m.EndpointHandlerMapping : Mapped "{[/dump || /dump.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-07-25 23:25:31.246 INFO 28387 --- [ main] o.s.b.a.e.m.EndpointHandlerMapping : Mapped "{[/autoconfig || /autoconfig.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-07-25 23:25:31.247 INFO 28387 --- [ main] o.s.b.a.e.m.EndpointHandlerMapping : Mapped "{[/auditevents || /auditevents.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public org.springframework.http.ResponseEntity<?> org.springframework.boot.actuate.endpoint.mvc.AuditEventsMvcEndpoint.findByPrincipalAndAfterAndType(java.lang.String,java.util.Date,java.lang.String)
2018-07-25 23:25:31.248 INFO 28387 --- [ main] o.s.b.a.e.m.EndpointHandlerMapping : Mapped "{[/health || /health.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.invoke(javax.servlet.http.HttpServletRequest,java.security.Principal)
2018-07-25 23:25:31.250 INFO 28387 --- [ main] o.s.b.a.e.m.EndpointHandlerMapping : Mapped "{[/info || /info.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-07-25 23:25:31.251 INFO 28387 --- [ main] o.s.b.a.e.m.EndpointHandlerMapping : Mapped "{[/metrics/{name:.*}],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.MetricsMvcEndpoint.value(java.lang.String)
2018-07-25 23:25:31.251 INFO 28387 --- [ main] o.s.b.a.e.m.EndpointHandlerMapping : Mapped "{[/metrics || /metrics.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-07-25 23:25:31.252 INFO 28387 --- [ main] o.s.b.a.e.m.EndpointHandlerMapping : Mapped "{[/mappings || /mappings.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-07-25 23:25:31.254 INFO 28387 --- [ main] o.s.b.a.e.m.EndpointHandlerMapping : Mapped "{[/heapdump || /heapdump.json],methods=[GET],produces=[application/octet-stream]}" onto public void org.springframework.boot.actuate.endpoint.mvc.HeapdumpMvcEndpoint.invoke(boolean,javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException,javax.servlet.ServletException
2018-07-25 23:25:31.259 INFO 28387 --- [ main] o.s.b.a.e.m.EndpointHandlerMapping : Mapped "{[/loggers/{name:.*}],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.LoggersMvcEndpoint.get(java.lang.String)
2018-07-25 23:25:31.259 INFO 28387 --- [ main] o.s.b.a.e.m.EndpointHandlerMapping : Mapped "{[/loggers/{name:.*}],methods=[POST],consumes=[application/vnd.spring-boot.actuator.v1+json || application/json],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.LoggersMvcEndpoint.set(java.lang.String,java.util.Map<java.lang.String, java.lang.String>)
2018-07-25 23:25:31.260 INFO 28387 --- [ main] o.s.b.a.e.m.EndpointHandlerMapping : Mapped "{[/loggers || /loggers.json],methods=[GET],produces=[application/vnd.spring-boot.actuator.v1+json || application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2018-07-25 23:25:32.433 INFO 28387 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Registering beans for JMX exposure on startup
2018-07-25 23:25:32.454 INFO 28387 --- [ main] o.s.c.s.DefaultLifecycleProcessor : Starting beans in phase 0
2018-07-25 23:25:32.808 INFO 28387 --- [ main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 18000 (http)
2018-07-25 23:25:32.814 INFO 28387 --- [ main] c.h.b.w.s.MetaStoreProxyServer : Starting WaggleDance on port 48869
Starting WaggleDance on port 48869
2018-07-25 23:25:32.839 INFO 28387 --- [ main] c.h.b.w.s.MetaStoreProxyServer : Starting WaggleDance Server
2018-07-25 23:25:32.846 INFO 28387 --- [ main] c.h.b.w.s.MetaStoreProxyServer : Started the new WaggleDance on port [48869]...
2018-07-25 23:25:32.847 INFO 28387 --- [ main] c.h.b.w.s.MetaStoreProxyServer : Options.minWorkerThreads = 200
2018-07-25 23:25:32.847 INFO 28387 --- [ main] c.h.b.w.s.MetaStoreProxyServer : Options.maxWorkerThreads = 1000
2018-07-25 23:25:32.847 INFO 28387 --- [ main] c.h.b.w.s.MetaStoreProxyServer : TCP keepalive = true
2018-07-25 23:25:57.209 INFO 28387 --- [pool-4-thread-1] c.h.b.w.m.m.MetaStoreMappingFactoryImpl : Mapping databases with name 'primary' to metastore: thrift://localhost:9083
2018-07-25 23:25:57.451 INFO 28387 --- [pool-4-thread-1] c.h.b.w.m.m.MetaStoreMappingFactoryImpl : Mapping databases with name 'prod' to metastore: thrift://10.10.127.175:9083
2018-07-25 23:25:57.732 INFO 28387 --- [pool-4-thread-1] c.j.a.a.NamedThreads : jcabi-aspects 0.22.6/3f0a1f7 started new daemon thread jcabi-loggable for watching of @Loggable annotated methods
2018-07-25 23:25:57.736 INFO 28387 --- [pool-4-thread-1] c.h.b.w.c.ThriftMetastoreClient : Trying to connect to metastore with URI thrift://localhost:9083
2018-07-25 23:25:57.766 INFO 28387 --- [pool-4-thread-1] c.h.b.w.c.ThriftMetastoreClient : Opened a connection to metastore 'thrift://localhost:9083', total current connections to all metastores: 1
2018-07-25 23:25:57.766 INFO 28387 --- [pool-4-thread-1] c.h.b.w.c.ThriftMetastoreClient : Connected to metastore.
2018-07-25 23:25:57.766 INFO 28387 --- [pool-4-thread-1] c.h.b.w.c.ThriftMetastoreClient : Trying to connect to metastore with URI thrift://10.10.127.175:9083
2018-07-25 23:25:57.769 INFO 28387 --- [pool-4-thread-1] c.h.b.w.c.ThriftMetastoreClient : Opened a connection to metastore 'thrift://10.10.127.175:9083', total current connections to all metastores: 2
2018-07-25 23:25:57.769 INFO 28387 --- [pool-4-thread-1] c.h.b.w.c.ThriftMetastoreClient : Connected to metastore.
2018-07-25 23:25:58.077 INFO 28387 --- [pool-4-thread-1] c.h.b.w.s.FederatedHMSHandler : Fetching database prod_test
2018-07-25 23:25:58.085 INFO 28387 --- [pool-4-thread-1] o.h.v.i.x.ValidationXmlParser : HV000007: META-INF/validation.xml found. Parsing XML based configuration.
2018-07-25 23:25:58.096 INFO 28387 --- [pool-4-thread-1] o.h.v.i.x.ValidationBootstrapParameters : HV000006: Using org.hibernate.validator.HibernateValidator as validation provider.
2018-07-25 23:25:58.114 INFO 28387 --- [pool-4-thread-1] c.h.b.w.s.FederatedHMSHandler : Mapping is 'prod_'
2018-07-25 23:25:58.185 INFO 28387 --- [pool-4-thread-2] c.h.b.w.m.m.MetaStoreMappingFactoryImpl : Mapping databases with name 'primary' to metastore: thrift://localhost:9083
2018-07-25 23:25:58.188 INFO 28387 --- [pool-4-thread-1] c.h.b.w.c.ThriftMetastoreClient : Closed a connection to metastore, current connections: 1
2018-07-25 23:25:58.188 INFO 28387 --- [pool-4-thread-1] c.h.b.w.c.ThriftMetastoreClient : Closed a connection to metastore, current connections: 0
2018-07-25 23:25:58.245 INFO 28387 --- [pool-4-thread-2] c.h.b.w.m.m.MetaStoreMappingFactoryImpl : Mapping databases with name 'prod' to metastore: thrift://10.10.127.175:9083
2018-07-25 23:25:58.281 INFO 28387 --- [pool-4-thread-2] c.h.b.w.c.ThriftMetastoreClient : Trying to connect to metastore with URI thrift://localhost:9083
2018-07-25 23:25:58.282 INFO 28387 --- [pool-4-thread-2] c.h.b.w.c.ThriftMetastoreClient : Opened a connection to metastore 'thrift://localhost:9083', total current connections to all metastores: 1
2018-07-25 23:25:58.282 INFO 28387 --- [pool-4-thread-2] c.h.b.w.c.ThriftMetastoreClient : Connected to metastore.
2018-07-25 23:25:58.282 INFO 28387 --- [pool-4-thread-2] c.h.b.w.c.ThriftMetastoreClient : Trying to connect to metastore with URI thrift://10.10.127.175:9083
2018-07-25 23:25:58.285 INFO 28387 --- [pool-4-thread-2] c.h.b.w.c.ThriftMetastoreClient : Opened a connection to metastore 'thrift://10.10.127.175:9083', total current connections to all metastores: 2
2018-07-25 23:25:58.285 INFO 28387 --- [pool-4-thread-2] c.h.b.w.c.ThriftMetastoreClient : Connected to metastore.
2018-07-25 23:25:58.305 INFO 28387 --- [pool-4-thread-2] c.h.b.w.s.FederatedHMSHandler : Fetching database prod_test
2018-07-25 23:25:58.308 INFO 28387 --- [pool-4-thread-2] c.h.b.w.s.FederatedHMSHandler : Mapping is 'prod_'
2018-07-25 23:25:58.322 INFO 28387 --- [pool-4-thread-2] c.h.b.w.s.FederatedHMSHandler : Fetching database prod_test
2018-07-25 23:25:58.325 INFO 28387 --- [pool-4-thread-2] c.h.b.w.s.FederatedHMSHandler : Mapping is 'prod_'
2018-07-25 23:26:13.054 ERROR 28387 --- [pool-4-thread-2] o.a.h.h.m.RetryingHMSHandler : org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
at org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:1563)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1550)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.hotels.bdp.waggledance.client.DefaultMetaStoreClientFactory$ReconnectingMetastoreClientInvocationHandler.invoke(DefaultMetaStoreClientFactory.java:67)
at com.sun.proxy.$Proxy134.get_table_req(Unknown Source)
at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_table_req_aroundBody618(FederatedHMSHandler.java:1481)
at com.hotels.bdp.waggledance.server.FederatedHMSHandler$AjcClosure619.run(FederatedHMSHandler.java:1)
at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
at com.hotels.bdp.waggledance.metrics.MonitoredAspect.monitor(MonitoredAspect.java:57)
at com.hotels.bdp.waggledance.metrics.MonitoredAspect.monitor(MonitoredAspect.java:47)
at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_table_req_aroundBody620(FederatedHMSHandler.java:1479)
at com.hotels.bdp.waggledance.server.FederatedHMSHandler$AjcClosure621.run(FederatedHMSHandler.java:1)
at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:213)
at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1)
at com.jcabi.aspects.aj.MethodLogger.wrapMethod(MethodLogger.java:169)
at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_table_req(FederatedHMSHandler.java:1479)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.hotels.bdp.waggledance.server.ExceptionWrappingHMSHandler.invoke(ExceptionWrappingHMSHandler.java:50)
at com.sun.proxy.$Proxy137.get_table_req(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at com.sun.proxy.$Proxy137.get_table_req(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:11457)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:11441)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hadoop.hive.metastore.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:48)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-07-25 23:26:13.055 ERROR 28387 --- [pool-4-thread-2] o.a.t.ProcessFunction : Internal error processing get_table_req
org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
at org.apache.thrift.TApplicationException.read(TApplicationException.java:111) ~[libthrift-0.9.3.jar!/:0.9.3]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) ~[libthrift-0.9.3.jar!/:0.9.3]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:1563) ~[hive-metastore-2.3.0.jar!/:2.3.0]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1550) ~[hive-metastore-2.3.0.jar!/:2.3.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_171]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_171]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
at com.hotels.bdp.waggledance.client.DefaultMetaStoreClientFactory$ReconnectingMetastoreClientInvocationHandler.invoke(DefaultMetaStoreClientFactory.java:67) ~[classes!/:?]
at com.sun.proxy.$Proxy134.get_table_req(Unknown Source) ~[?:?]
at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_table_req_aroundBody618(FederatedHMSHandler.java:1481) ~[classes!/:?]
at com.hotels.bdp.waggledance.server.FederatedHMSHandler$AjcClosure619.run(FederatedHMSHandler.java:1) ~[classes!/:?]
at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) ~[aspectjweaver-1.8.10.jar!/:1.8.10]
at com.hotels.bdp.waggledance.metrics.MonitoredAspect.monitor(MonitoredAspect.java:57) ~[classes!/:?]
at com.hotels.bdp.waggledance.metrics.MonitoredAspect.monitor(MonitoredAspect.java:47) ~[classes!/:?]
at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_table_req_aroundBody620(FederatedHMSHandler.java:1479) ~[classes!/:?]
at com.hotels.bdp.waggledance.server.FederatedHMSHandler$AjcClosure621.run(FederatedHMSHandler.java:1) ~[classes!/:?]
at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) ~[aspectjweaver-1.8.10.jar!/:1.8.10]
at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:213) ~[jcabi-aspects-0.22.6.jar!/:?]
at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1) ~[jcabi-aspects-0.22.6.jar!/:?]
at com.jcabi.aspects.aj.MethodLogger.wrapMethod(MethodLogger.java:169) ~[jcabi-aspects-0.22.6.jar!/:?]
at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_table_req(FederatedHMSHandler.java:1479) ~[classes!/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_171]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_171]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
at com.hotels.bdp.waggledance.server.ExceptionWrappingHMSHandler.invoke(ExceptionWrappingHMSHandler.java:50) ~[classes!/:?]
at com.sun.proxy.$Proxy137.get_table_req(Unknown Source) ~[?:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_171]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_171]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148) ~[hive-metastore-2.3.0.jar!/:2.3.0]
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107) ~[hive-metastore-2.3.0.jar!/:2.3.0]
at com.sun.proxy.$Proxy137.get_table_req(Unknown Source) ~[?:?]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:11457) ~[hive-metastore-2.3.0.jar!/:2.3.0]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:11441) ~[hive-metastore-2.3.0.jar!/:2.3.0]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) [libthrift-0.9.3.jar!/:0.9.3]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) [libthrift-0.9.3.jar!/:0.9.3]
at org.apache.hadoop.hive.metastore.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:48) [hive-metastore-2.3.0.jar!/:2.3.0]
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) [libthrift-0.9.3.jar!/:0.9.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
2018-07-25 23:26:14.066 INFO 28387 --- [pool-4-thread-2] c.h.b.w.c.ThriftMetastoreClient : Closed a connection to metastore, current connections: 1
2018-07-25 23:26:14.066 INFO 28387 --- [pool-4-thread-2] c.h.b.w.c.ThriftMetastoreClient : Closed a connection to metastore, current connections: 0
2018-07-25 23:26:14.067 INFO 28387 --- [pool-4-thread-3] c.h.b.w.m.m.MetaStoreMappingFactoryImpl : Mapping databases with name 'primary' to metastore: thrift://localhost:9083
2018-07-25 23:26:14.105 INFO 28387 --- [pool-4-thread-3] c.h.b.w.m.m.MetaStoreMappingFactoryImpl : Mapping databases with name 'prod' to metastore: thrift://10.10.127.175:9083
2018-07-25 23:26:14.136 INFO 28387 --- [pool-4-thread-3] c.h.b.w.c.ThriftMetastoreClient : Trying to connect to metastore with URI thrift://localhost:9083
2018-07-25 23:26:14.138 INFO 28387 --- [pool-4-thread-3] c.h.b.w.c.ThriftMetastoreClient : Opened a connection to metastore 'thrift://localhost:9083', total current connections to all metastores: 1
2018-07-25 23:26:14.138 INFO 28387 --- [pool-4-thread-3] c.h.b.w.c.ThriftMetastoreClient : Connected to metastore.
2018-07-25 23:26:14.138 INFO 28387 --- [pool-4-thread-3] c.h.b.w.c.ThriftMetastoreClient : Trying to connect to metastore with URI thrift://10.10.127.175:9083
2018-07-25 23:26:14.141 INFO 28387 --- [pool-4-thread-3] c.h.b.w.c.ThriftMetastoreClient : Opened a connection to metastore 'thrift://10.10.127.175:9083', total current connections to all metastores: 2
2018-07-25 23:26:14.141 INFO 28387 --- [pool-4-thread-3] c.h.b.w.c.ThriftMetastoreClient : Connected to metastore.
2018-07-25 23:26:14.150 ERROR 28387 --- [pool-4-thread-3] o.a.h.h.m.RetryingHMSHandler : org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
at org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:1563)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1550)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.hotels.bdp.waggledance.client.DefaultMetaStoreClientFactory$ReconnectingMetastoreClientInvocationHandler.invoke(DefaultMetaStoreClientFactory.java:67)
at com.sun.proxy.$Proxy134.get_table_req(Unknown Source)
at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_table_req_aroundBody618(FederatedHMSHandler.java:1481)
at com.hotels.bdp.waggledance.server.FederatedHMSHandler$AjcClosure619.run(FederatedHMSHandler.java:1)
at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
at com.hotels.bdp.waggledance.metrics.MonitoredAspect.monitor(MonitoredAspect.java:57)
at com.hotels.bdp.waggledance.metrics.MonitoredAspect.monitor(MonitoredAspect.java:47)
at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_table_req_aroundBody620(FederatedHMSHandler.java:1479)
at com.hotels.bdp.waggledance.server.FederatedHMSHandler$AjcClosure621.run(FederatedHMSHandler.java:1)
at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:213)
at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1)
at com.jcabi.aspects.aj.MethodLogger.wrapMethod(MethodLogger.java:169)
at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_table_req(FederatedHMSHandler.java:1479)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.hotels.bdp.waggledance.server.ExceptionWrappingHMSHandler.invoke(ExceptionWrappingHMSHandler.java:50)
at com.sun.proxy.$Proxy137.get_table_req(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at com.sun.proxy.$Proxy137.get_table_req(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:11457)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:11441)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hadoop.hive.metastore.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:48)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-07-25 23:26:14.150 ERROR 28387 --- [pool-4-thread-3] o.a.t.ProcessFunction : Internal error processing get_table_req
org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
at org.apache.thrift.TApplicationException.read(TApplicationException.java:111) ~[libthrift-0.9.3.jar!/:0.9.3]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) ~[libthrift-0.9.3.jar!/:0.9.3]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:1563) ~[hive-metastore-2.3.0.jar!/:2.3.0]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1550) ~[hive-metastore-2.3.0.jar!/:2.3.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_171]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_171]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
at com.hotels.bdp.waggledance.client.DefaultMetaStoreClientFactory$ReconnectingMetastoreClientInvocationHandler.invoke(DefaultMetaStoreClientFactory.java:67) ~[classes!/:?]
at com.sun.proxy.$Proxy134.get_table_req(Unknown Source) ~[?:?]
at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_table_req_aroundBody618(FederatedHMSHandler.java:1481) ~[classes!/:?]
at com.hotels.bdp.waggledance.server.FederatedHMSHandler$AjcClosure619.run(FederatedHMSHandler.java:1) ~[classes!/:?]
at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) ~[aspectjweaver-1.8.10.jar!/:1.8.10]
at com.hotels.bdp.waggledance.metrics.MonitoredAspect.monitor(MonitoredAspect.java:57) ~[classes!/:?]
at com.hotels.bdp.waggledance.metrics.MonitoredAspect.monitor(MonitoredAspect.java:47) ~[classes!/:?]
at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_table_req_aroundBody620(FederatedHMSHandler.java:1479) ~[classes!/:?]
at com.hotels.bdp.waggledance.server.FederatedHMSHandler$AjcClosure621.run(FederatedHMSHandler.java:1) ~[classes!/:?]
at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) ~[aspectjweaver-1.8.10.jar!/:1.8.10]
at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:213) ~[jcabi-aspects-0.22.6.jar!/:?]
at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1) ~[jcabi-aspects-0.22.6.jar!/:?]
at com.jcabi.aspects.aj.MethodLogger.wrapMethod(MethodLogger.java:169) ~[jcabi-aspects-0.22.6.jar!/:?]
at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_table_req(FederatedHMSHandler.java:1479) ~[classes!/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_171]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_171]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
at com.hotels.bdp.waggledance.server.ExceptionWrappingHMSHandler.invoke(ExceptionWrappingHMSHandler.java:50) ~[classes!/:?]
at com.sun.proxy.$Proxy137.get_table_req(Unknown Source) ~[?:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_171]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_171]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148) ~[hive-metastore-2.3.0.jar!/:2.3.0]
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107) ~[hive-metastore-2.3.0.jar!/:2.3.0]
at com.sun.proxy.$Proxy137.get_table_req(Unknown Source) ~[?:?]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:11457) ~[hive-metastore-2.3.0.jar!/:2.3.0]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:11441) ~[hive-metastore-2.3.0.jar!/:2.3.0]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) [libthrift-0.9.3.jar!/:0.9.3]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) [libthrift-0.9.3.jar!/:0.9.3]
at org.apache.hadoop.hive.metastore.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:48) [hive-metastore-2.3.0.jar!/:2.3.0]
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) [libthrift-0.9.3.jar!/:0.9.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Issue Analytics
- State:
- Created 5 years ago
- Comments:5 (3 by maintainers)
Top Results From Across the Web
Internal error processing get_table_req · Issue #110 - GitHub
we are seeing similar error when running show create table, configuration details. hive cli version: 2.3.3(EMR 5.15) waggle-dance version: 2.3.1
Read more >Internal error processing completeApplyMigrations
New install of Tableau Server on RHEL fails at the Running Migrations step of initialization. This job failed due to unexpected error: ' ......
Read more >Resolving the "Internal error when processing the request ...
This type of error can generally be resolved by running the Verify and Rebuild functions from the File > Utilities menu in QuickBooks....
Read more >com.amazonaws.services.keyspaces.model (AWS SDK for Java ...
Amazon Keyspaces was unable to fully process this request because of an internal server error. ResourceNotFoundException. The operation tried to access a ...
Read more >org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore ...
Please understand that we have to compensate our server costs. ... TException; public GetTableResult get_table_req(GetTableRequest req) throws MetaException ...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
The relevant Circus Train ticket and PR that appears to be equivalent to this is:
https://github.com/HotelsDotCom/circus-train/issues/32 https://github.com/HotelsDotCom/circus-train/pull/38
This has been fixed in Waggle Dance 3.0.0 which was released yesterday. We’d appreciate if everyone who experienced this issue could upgrade and let us know if this resolves it for you.