PHP/JS/WebDev Zip generation of downloads

[Ralle]

When you download a bundle consisting of more than one file on Hive, you are given a zip file. This file is generated on the server and cached for a number of days. This system works fine for small downloads (<100mb) but recently, with the Re-Reforged project it came to my attention that it's nearing 1 GB. We had some issues where the PHP ZIP generation would take more than the maximum 120 seconds to generate (seconds where the user is actually waiting to download) and would fail and generate a corrupted zip.

I fixed this by increasing the maximum seconds to 240, but it's far longer than any user should have to wait to download something.

So I have been looking into my options. I see two solutions.

1. Download all files in browser and use JSZip locally to serve the combined file to the users browser. The downside of this is that 1 GB is still a bit steep and I envision a future where you may pick and choose multiple bundles and download them all at once, so we'd quickly reach multiple GBs for a single download. I don't want to push more than 1 GB onto the browser for it to zip and then download. It will cause problems when people don't have much RAM.
   
   
2. Streaming zip server-side. I am limited a bit by PHP but there are ways to stream a file. If I wasn't using the standard ZipArchive and wrote my own ZIP maker, I could stream the zip rather than having the user wait for it to be zipped. I do one day plan to use blob storage on AWS S3 or Backblaze B2 instead so server-side zip generation will not be feasible here.

I definitely want to go with option 1, but there are a few obstacles.

I talked a bit with @GhostWolf and what I really want is file streaming onto disk. The only solutions I've seen requires you to generate the whole file in browser and when you have the full size in RAM (could be multiple GBs) you can pass it on to be downloaded. I need a WritableStream to a File object somehow. But that doesn't exist.

After sleeping on it, I have a theory of a potential solution. Most browsers support something called Service Workers now. A service worker can have many roles, one of them being to intercept requests made on the website. So my theory is that I can make a special service worker that intercepts requests of the format:

www.hiveworkshop.com/special-download-intercept-url/<bundle-id>

Which would then craft a response where it would fetch each file sequentially from the server and stream them into the response using JSZip to combine all files there. What the browser sees is a single download which happens to be a zip. The zip will be generated by the service worker. And it would all be streaming and low memory usage.

I still need to find out and test but it's interesting and might work. I have yet to find out why it wouldn't

• Service Worker API - Web APIs | MDN
  
• FetchEvent - Web APIs | MDN
• Download files like mega.co.nz
• Ziping a filezipe of 3.5 GB on a browser in a 1 GB RAM machine results in "Your computer is low on memory" error · Issue #530 · Stuk/jszip
• JavaScript: Writing to download stream

It's a project on my list of things for our post-XF2 migration coming up.

 

[Almia]

Interesting. Have you considered compression algorithms such as Brotli? I have found it to be able to reduce sizes up to 80% of the original file size, not sure how effective it is for file sizes over 100 mb. If the idea of downloading files through browser, Brotli or GZip might be helpful, as browsers are capable of decompressing it through Content-Encoding header. Although streaming chunks and reconstructing it is also a good idea.

 

[Ralle]

I actually don't have much problems with the size, just the time to generate the download, and if we used brotli, people would have to install something to unzip unlike how it is now.