Parallelism not working as expected
See original GitHub issueWe recently found the weird behavior of parallelism with freestyle. Details below:
Algebra:
@free
trait Validation[F[_]] {
def minSize(n: Int): FreeS.Par[F, Boolean]
def hasNumber: FreeS.Par[F, Boolean]
}
Interpreter:
import cats.data.Kleisli
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
object Interpreters {
import Algebra._
type ParValidator[A] = Kleisli[Future, String, A]
implicit val interpreter = new Validation.Handler[ParValidator] {
def minSize(n: Int): ParValidator[Boolean] =
Kleisli { s =>
Future {
println("x before")
Thread.sleep(4000)
println("x after")
s.size >= n
}
}
def hasNumber: ParValidator[Boolean] =
Kleisli { s =>
Future {
println("y before")
Thread.sleep(100)
println("y after")
s.exists(c => "0123456789".contains(c))
}
}
}
}
Test1:
"program with for-expression running in nondeterminism style" in {
import cats.implicits._
import scala.concurrent.ExecutionContext.Implicits.global
import Algebra._
import Interpreters._
import freestyle.nondeterminism._
import freestyle.implicits._
val validation = Validation[Validation.Op]
import validation._
val k = for {
one <- (minSize(3) |@| hasNumber).map(_ :: _ :: Nil)
}yield (one)
val validator = k.exec[ParValidator]
Await.result(validator.run("a"), Duration.Inf) shouldBe List(false, false)
}
Output:(nondeterminism)
x before
y before
y after
x after
Tes2:
"program with for-expression running in determinism style because of `tupled.freeS`" in {
import cats.implicits._
import scala.concurrent.ExecutionContext.Implicits.global
import Algebra._
import Interpreters._
import freestyle.nondeterminism._
import freestyle.implicits._
val validation = Validation[Validation.Op]
import validation._
val k = for {
one <- (minSize(3) |@| hasNumber).tupled.freeS
}yield (one)
val validator = k.exec[ParValidator]
Await.result(validator.run("a"), Duration.Inf) shouldBe (false, false)
}
Output:(determinism)
x before
x after
y before
y after
We found that hasNumber
is blocked till minSize
is completed. So I think the issue is between map
and tupled.freeS
.
If you want to run a complete program, you can clone it from here https://github.com/abdheshkumar/freestyle-example/blob/master/src/test/scala/FreeFutureTest.scala
Issue Analytics
- State:
- Created 6 years ago
- Comments:5 (3 by maintainers)
Top Results From Across the Web
OLAP Cube Parallelism not working as expected
We are having a OLAP SSAS cube setup and the Cube Processing is triggered from SQL Server Agent Job (on SQL ...
Read more >azurerm_storage_blob parallelism not working as expected?
Hello, I'm running terraform v0.11.7 with the following plugins azurerm v1.15.0_x4 and azure v0.1.1_x4. I'm trying to upload a VHD image to ...
Read more >21 Diagnosing Parallel Execution Performance Problems
If performance is as you expected, can you justify the notion that there is a performance problem? Perhaps you have a desired outcome...
Read more >Parallel Structure - Purdue OWL
Parallel structure means using the same pattern of words to show that two or more ideas have the same level of importance. This...
Read more >Ten Flink Gotchas we wish we had known
1 - Workloads skew due to parallelism settings. Let's start with a simple problem: when investigating a task's subtasks in the Flink UI,...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Now that the Cats PR is merged, I am going ahead and close this issue.
If you want to try this before the next Cats release, you can provide your own
Monad
instance forKleisli
and usefoldMap
with a bit more work thanexec
.This actually has nothing to do with
map
versustupled
, but with usingfreeS
or not using.freeS
.Calling
freeS
on(minSize(3) |@| hasNumber).tupled
lifts theFreeS.Par[Validation.Op, (Boolean, Boolean)]
into aFreeS[Validation.Op, (Boolean, Boolean)]
.The difference between calling
exec[ParValidator]
onFreeS.Par
orFreeS
is that the first one uses theApplicative[ParValidator]
instance where the second one uses theMonad[ParValidator]
instance.The problem is that the
Monad
instance ofKleisli
doesn’t override theproduct
method, which is used bymap
andtupled
of the cartesian builder. This means that the twoFuture
operations are executed sequentially.PR https://github.com/typelevel/cats/pull/1618 should solve this in the next Cats version.