It was not a background video, I used this code to embed the video:
< video width="100%" height="340" controls>
<source src="/ABSOLUTE URL" type="video/mp4">
Your browser does not support the video tag.
</video >
It is a video of 9 MB. I uploaded it within the Joomla's images folder and then i tried to upload it from Amazon S3.
In both occasions the video would load just fine from desktop and on mobile it would play for just half a second then stop.
I ended up using the extension All Video Share by using a shortcode inside a custom module, inside SP Page Builder. Now it works. It seems like standard HTML5 code isn't working.