I think you could play with sticky footers, sticky headers, and some active styling indicators like opacity and background plus highlights to let the user know they are in scroll mode. Really fun idea, love this! Wish Apple Music UI for moving songs on playlists had that. Holding my dumb while dragging is suboptimal comparing to click and scroll.