pandas on spark apply_batch/transform_batch broken? (tl;dr; No – but it isn’t well documented)
Arnon Rotem-Gal-Oz
Using pypark’s pandas integration via apply_batch and transform_batch is very powerful but lacking documentation can cause hard to trace bugs – hopefully my experience (below) can save you some debugging when you’d use it yourself. It has a been something like 3 years since I last used...