1. Triumph has risen from these uncharted shores. The 34th Modeling Contest Results are out!
    Dismiss Notice
  2. Awaken what lies in the heart of your swarm. The 17th Techtree Contest has arrived!
    Dismiss Notice
  3. The Hive Workshop is launching its first HD modelling contest. How HD should it be?
    Dismiss Notice
  4. Check out the Staff Job Openings thread.
    Dismiss Notice
Dismiss Notice
Hive 3 Remoosed BETA - NOW LIVE. Go check it out at BETA Hive Workshop! Post your feedback in this new forum BETA Feedback.
Dismiss Notice
60,000 passwords have been reset on July 8, 2019. If you cannot login, read this.

PHP/JS/WebDev Zip generation of downloads

Discussion in 'Programming' started by Ralle, Feb 19, 2021.

  1. Ralle

    Ralle

    Owner

    Joined:
    Oct 6, 2004
    Messages:
    7,732
    Resources:
    21
    Tools:
    2
    Maps:
    5
    Tutorials:
    14
    Resources:
    21
    When you download a bundle consisting of more than one file on Hive, you are given a zip file. This file is generated on the server and cached for a number of days. This system works fine for small downloads (<100mb) but recently, with the Re-Reforged project it came to my attention that it's nearing 1 GB. We had some issues where the PHP ZIP generation would take more than the maximum 120 seconds to generate (seconds where the user is actually waiting to download) and would fail and generate a corrupted zip.

    I fixed this by increasing the maximum seconds to 240, but it's far longer than any user should have to wait to download something.

    So I have been looking into my options. I see two solutions.

    1. Download all files in browser and use JSZip locally to serve the combined file to the users browser. The downside of this is that 1 GB is still a bit steep and I envision a future where you may pick and choose multiple bundles and download them all at once, so we'd quickly reach multiple GBs for a single download. I don't want to push more than 1 GB onto the browser for it to zip and then download. It will cause problems when people don't have much RAM.

    2. Streaming zip server-side. I am limited a bit by PHP but there are ways to stream a file. If I wasn't using the standard ZipArchive and wrote my own ZIP maker, I could stream the zip rather than having the user wait for it to be zipped. I do one day plan to use blob storage on AWS S3 or Backblaze B2 instead so server-side zip generation will not be feasible here.

    I definitely want to go with option 1, but there are a few obstacles.

    I talked a bit with @GhostWolf and what I really want is file streaming onto disk. The only solutions I've seen requires you to generate the whole file in browser and when you have the full size in RAM (could be multiple GBs) you can pass it on to be downloaded. I need a WritableStream to a File object somehow. But that doesn't exist.

    After sleeping on it, I have a theory of a potential solution. Most browsers support something called Service Workers now. A service worker can have many roles, one of them being to intercept requests made on the website. So my theory is that I can make a special service worker that intercepts requests of the format:
    Code (Text):
    www.hiveworkshop.com/special-download-intercept-url/<bundle-id>
    Which would then craft a response where it would fetch each file sequentially from the server and stream them into the response using JSZip to combine all files there. What the browser sees is a single download which happens to be a zip. The zip will be generated by the service worker. And it would all be streaming and low memory usage.

    I still need to find out and test but it's interesting and might work. I have yet to find out why it wouldn't


    It's a project on my list of things for our post-XF2 migration coming up.
     
    Last edited: Feb 19, 2021