question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Outputs vary for Koalas Series vs Pandas Series when we have np.nan

See original GitHub issue

The behavior of the Series functions changes when I introduced a np.nan in the series. But Pandas seem to handle this just fine. Some of the functions and their outputs.

def usingSeries():
    s = ks.Series([11, 3, 5, 6, 8,np.nan])
    print(s.max())
    print(s.min())
    print(s.kurtosis())
    print(s.std())
nan
3.0
nan
nan
def usingPdSeries():
    s = pd.Series([11, 3, 5, np.nan, 6, 8])
    print(s.max())
    print(s.min())
    print(s.kurtosis())
    print(s.std())
11.0
3.0
-0.0034686090877542597
3.049590136395381

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:5 (3 by maintainers)

github_iconTop GitHub Comments

2reactions
floschacommented, May 2, 2019

When running

s = ks.Series([11, 3, 5, 6, 8, np.nan])
print(s.max())
print(s.min())
print(s.kurtosis())
print(s.std())

with Koalas 0.2, I get the following output:

11.0
3.0
-1.0008671522719388
3.0495901363953815

Has this issue been fixed with the latest Koalas release or is something wrong with @akhilanandbv003’s setup?

1reaction
shrilcommented, May 2, 2019

It works fine for me.

>>> s = ks.Series([11, 3, 5, 6, 8, np.nan])
>>> print(s.max())
11.0
>>> print(s.min())
3.0
>>> print(s.kurtosis())
-1.0008671522719395
>>> print(s.std())
3.0495901363953815
Read more comments on GitHub >

github_iconTop Results From Across the Web

How to convert np.where() while converting pandas to koalas?
The code is working if I convert the koalas back to pandas using toPandas() or from_pandas() , but due to performance and scalability...
Read more >
Series — Koalas 1.8.2 documentation - Read the Docs
Conform Series to new index with optional filling logic, placing NA/NaN in locations having no value in the previous index. Series.reindex_like (other).
Read more >
10 Minutes from pandas to Koalas on Apache Spark - Databricks
A Koalas Series can be created by passing a list of values, the same way as a pandas Series. A Koalas Series can...
Read more >
16 Underrated Pandas Series Methods And When To Use Them
In this article, we're going to explore some lesser-known but very useful pandas methods for manipulating Series objects.
Read more >
Koalas: Easy Transition between pandas and Spark - Medium
If you are a data scientist you might have used pandas data-frame API to get your ... Output comparison between Pandas and Koalas...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found