Koalas DataFrame should only have Pandas corresponding APIs (not Spark APIs)

This is partially of https://github.com/databricks/koalas/issues/119

I was thinking Koalas DataFrame strictly should have Pandas corresponding APIs, although there might be few exceptions with some strong reasons.

Meaning Koalas DataFrame should not have Spark DataFrame specific APIs like explain() or selectExpr(). My current thought is we have a API like koalas_df.to_spark() (borrowed from @ueshin’s idea via offline discussion) so that users can use Spark APIs.

Koalas API usages

koalas_df.loc(...)
koalas_df.drop(...)  # works as Pandas API

Spark API usages

koalas_df.to_spark().explain()
koalas_df.to_spark().selectExpr()
koalas_df.to_spark().drop(...)  # works as Spark API

This can clearly define what Koalas APIs expect and Spark APIs expect.

[thunterdb]

I think that users should be able to pick what they need, in particular things like .cache() or .repartition() which can be necessary. Doing koala_df.to_spark().cache().to_koalas() looses the index and metadata, so it is a no-go. Other users should also chime in though.

[rxin]

Closing this as it will be part of the design philosophy doc.