are you familiar with how to create mass actions for related lists for Classic? It's the same.

In a nutshell, go to Setup, create a Custom List Button that is of type Visualforce page. Javascript buttons are not supported. URL buttons sometimes work, but if they involve parameters being added to a salesforce URL many of those are not supported in Lightning Experience. Once you have that button, go to the Page Layout Editor, for the related list you want, click the gear icon and add the custom button. Remember to create the Custom button for the object type of the related list, not of the parent.

It is on the roadmap to allow you to use a Lightning Component based button as well, but that currently does not exist yet.