Question about blockEntityLinkage

I have a question about the following line :

 val linkedResults = LuceneRDD.blockEntityLinkage(a, b,
      linker,
      Array("blocker"),
      Array("blocker"),
      500
    )

How will it link a and b?

a ) Search for all the entries of a in b , b) All the entries of b in a c) combine a and b and search both?

Ideally , I’d like to archive the option b) , where I have a small table “b” that I’d like to link with a larger table “a”

Thank you.

[zouzias]

You can do:

val topK = 5
val linkedResults = LuceneRDD.blockEntityLinkage(B, A, linker, blockingFields, blockingFields, topK)

[zouzias]

The semantics of the block linkage is as follows:

1. Split a and b into blocks based on the value on column “blocker”. Say a value of the blocker column is “blocker_value_1”.
2. Then, approximate link all values between a and b that have value on column “blocker” equal to exactly “blocker_value_1”.

You could use blocking on any column where you know that a and b have common values and you can assume that you can condition on such a column.

See Slide 15 here: http://helios.mi.parisdescartes.fr/~themisp/publications/PapadakisPalpanas-TutorialWWW2018.pdf