[Proposal]Metadata(URL) problems and possible solutions

Hello, community

We have previously roughly discussed the problems with Dubbo’s metadata in GitHub issues and mailing list, here when saying metadata I mainly refer to URL in Dubbo. Now I think it’s time to bring the discussion onto the table and consider solving it. Below is the proposal I’ve drafted:

Problems

URL works as the carrier for Dubbo internal data transfer at runtime, and is also works as a protocol integrating with the external system: registration center, service query, service management, etc., resulting in serious coupling between the configuration and the registration center, and the data load pushed by the registration center is too large.

Targets

• Keep URL unchanged at runtime, and even enrich the runtime URL; Keep the immutability of the URL during a single invoke from start to end.
• Simplify Registry url as much as possible.
• Refactor the way of doing service configuration, rely on Registry currently, switch to third-party Config Center while keeping compatible with old Overrides protocol.
• Some data in URL are for service query purpose, we should register this part to another place instead of Registry. And it should be realized as extensible and users can decide to enable it or not.

Solutions

Here is a typical URL in Registry:

dubbo://10.122.111.22:20880/com.xxx.compose.ic.service.vas.ValueAddServiceCompose?anyhost=true&application=goods-compose&default.actives=400&default.delay=-1&default.dispatcher=all&default.group=online&default.loadbalance=leastactive&default.service.filter=-monitor&default.threads=400&default.timeout=2000&default.version=1.0&delay=1&dubbo=2.6.2&environment=product&interface=com.xxx.compose.ic.service.vas.ValueAddServiceCompose&logger=slf4j&methods=queryValueAddServiceEditVOs,auditPassByValueAddServiceIdList,getValueAddServiceVOByServiceIds,getValueAddServiceInfoByGoodsIdList,getRuleMatchVOByGoodsIdList,getValueAddServiceEditVOById,getValueAddServiceEditVOByIds,queryValueAddServiceVOs,disableValueAddService,auditRejectByValueAddServiceIdList,saveValueAddServiceEditVO,enableValueAddService&organization=someorg&owner=somebody&pid=54812&revision=1.18.0628.3&side=provider&timestamp=1530671441040

We can see this URL showed here is really long and redundant, here are my suggestions to refactor it step by step:

1. Only keep necessary part in Registry For example:

dubbo://10.122.111.22:20880/com.xxx.compose.ic.service.vas.ValueAddServiceCompose?timeout=1000&group=online&version=1.0&weight=100&timestamp=1530671441040

1. Find a new place for those parameters only for query purpose For example:

delay=1&dubbo=2.6.2&interface=com.xxx.compose.ic.service.vas.ValueAddServiceCompose&logger=slf4j&methods=queryValueAddServiceEditVOs,auditPassByValueAddServiceIdList,getValueAddServiceVOByServiceIds,getValueAddServiceInfoByGoodsIdList,getRuleMatchVOByGoodsIdList,getValueAddServiceEditVOById,getValueAddServiceEditVOByIds,queryValueAddServiceVOs,disableValueAddService,auditRejectByValueAddServiceIdList,saveValueAddServiceEditVO,enableValueAddService&organization=someorg&owner=somebody&pid=54812&revision=1.18.0628.3&side=provider

1. Deprecate the Override protocol in Registry Typical Override protocol to change timeout value at runtime:

   override://0.0.0.0/com.xxx.compose.ic.service.vas.ValueAddServiceCompose?category=configurators&dynamic=false&application=foo&timeout=1000

We can instead integrate with thirdparty Config Center to do that, like Apollo, Archaius, Diamond and so on, these systems are more likely to be used in a miscroservice architecture.

Furthermore, we can also consider separate the routing rule config from Registry and switch to Config Center, using Config Center to dynamically add new group policies that can group service instances into different groups according to environments.

Kindly ask for your suggestions, don’t hesitate to let us know if you have any different ideas.

[chickenlj]

Integrate other Config Center make operation and maintenance complex. It means that we should deploy a Config Center to use dubbo, not very suitable for me

We need to keep compatibility with current Overrides policies. I listed ‘Apollo’, ‘Archaius’, ‘Diamond’ above, but Zookeeper can still be an option for configuration, Archaius itself also doesn’t need too much effort to deploy. So it’s not mandatory to deploy a Config Center.

Besides, Config Center seems to be a common system for a Microservice architecture, users usually don’t need to deploy it for Dubbo purpose, it is usually already there. Consider a cluster with a certain scale, it may have lots of rules that need to be stored and notified at runtime, a separate configuration center may be more scalable.

What do you think now, @carryxyh ?

[chickenlj]

Parameters need to transfer to Registry

The URL for provider parameters comes from:

1. ApplicationConfig
2. ModuleConfig
3. ProviderConfig
4. ProtocolConfig
5. ServiceConfig
6. MethodConfig&ArgumentConfig
7. others: side/dubbo/timestamp/pid/generic/methods/revision/default/token/notify/dynamic

ApplicationConfig

application(name) need to transfer.

ModuleConfig

Module info may not need to transfer to registry.

ProviderConfig

ProviderConfig ‘ keys are subsets of keys in ServiceConfig and ProtocolConfig, And It’s default. prefix should be removed.

ProtocolConfig

key	C use	transfer
accepts	N	N
accesslog	N	N
buffer	Y	N
charset	N	N
client	Y	N
codec	Y	Y
contextpath	N	N
default	N	N
dispatcher	N	N
dispather	N	N
exchanger	Y	Y
extension	Y	N
heartbeat	N	N
host	Y	Y
id	N	N
iothreads	N	N
keep-alive	N	N
name	Y	Y
networker	p2p?	?
optimizer	for kryo?	?
path	N	N
payload	Y	N?
port	Y	N not in parameters
prompt	N	N
queues	N	N
register	N	N
serialization	Y	Y
server	N	N
status	N	N
telnet	N	N
threadpool	N	N
threads	N	N
transporter	Y	N

ServiceConfig

key	C use	transfer
accesslog	N	N
actives	Y	Y?
async	Y	N
cache	Y	N?
callbacks	callback limits	N?
cluster	Y	Y
connections	Y	Y
delay	N	N
deprecated	Y	Y
document	N	N
dynamic	N	N
executes	N	N
export	N	N
filter service.filter	N	N
generic	Y	N?
group	Y	Y
id	N	N
interface	Y	N, use path
layer	N	N
listener	N	N
loadbalance	Y	Y
local	same with stub	??
merger	Y	N
mock	Y	Y
monitor	N	N
onconnect	N	N
ondisconnect	N	N
owner	N	N
path	Y	Y
proxy	N	N
ref	N	N
register	Y	Y
retries	Y	N?
scope	N	N
sent	Y	N?
serialization	Y	Y
stub	Y	N
timeout	Y	Y
token	Y	Y
validation	Y	N
version	Y	Y
warmup	Y	Y
weight	Y	Y

MethodConfig&ArgumentConfig

Method & Argument Config should not transfer to registry. Use configurators instead.

All transfer to Consumer side

others

others: side/dubbo/timestamp/pid/generic/methods/revision/default/notify/dynamic

the dubbo? & timestamp should be transfered.

TranferKeys

• application
• codec
• exchanger
• host
• protocol
• payload
• port
• serialization
• cluster
• connections
• deprecated
• group
• loadbalance
• mock
• path
• timeout
• token
• version
• warmup
• weight
• dubbo?
• timestamp
• pid
• Any method level configuration of the listed keys above

Parameters to search center

• All parameters should be stored to search center
• Further enrich method signature information: method name, param name, JSON string that fully reflects the parameter structure.