New Features in WoltLab Suite 6.1: Upload Pipeline

Files can be uploaded in a wide variety of places in the software: file attachments in a post, a picture in the gallery, your own avatar or a file in WoltLab Suite Filebase. This list can be extended almost indefinitely and this is exactly what has happened consistently for a long time: files can now be uploaded to so many places in the software that even we find it difficult to name them all straight away.

Upload Files: Significant Effort and Little Flexibility

However, this high degree of flexibility and customizability also comes at a price: all these uploaded files must be received, processed and, if successful, saved for later use. For this purpose, there are two different components in the software, the file attachments and a somewhat more general implementation for file uploads.

While the file attachments represent a self-contained and quite comprehensive system, the more general implementation is best described as a simple base implementation than a ready-made solution. As a result, both we and other developers have to put in some effort for the implementation and ongoing maintenance. Additional functions, such as scaling images before uploading them to the browser, have to be re-implemented each time.

In other areas, not all the capabilities that are available today are fully taken advantage of. PHP, for example, still relies on an almost archaic limit of 2 MB for uploaded files, forcing every site owner to adapt the configuration sooner or later. Another aspect is the long time it takes to rebuild the data after switching to a new version of the software: updating the preview images in particular is extremely time-consuming.

Foundation for the Future: The New Upload Pipeline

With WoltLab Suite 6.1, we have spent months meticulously developing a new system for uploading, processing, storing and delivering files. We incorporated our own experience as well as feedback from customers and developers into the design in order to find a reliable solution for typical use cases. The new upload pipeline offers a flexible basis for file uploads, takes care of most of the work and still leaves the developer sufficient design freedom for integration.

The heart of the new system is the uploading of files as a “chunked upload”: files that are larger than the limit set in PHP are automatically split into smaller segments in the browser and uploaded one after the other. The file is reassembled correctly on the server and then saved; files larger than 2.1 GB are also supported.

More Efficient Delivery of Images

The delivery of images, such as preview images, is a special feature. In the case of file attachments, for example, these were always delivered via a PHP call, which is significantly slower and blocks a PHP work process during retrieval. This was a huge problem, especially for forums with a lot of images, and caused many a visitor to have to think for a second when accessing the page.

With WoltLab Suite 6.1, these images are stored separately and referenced directly. They are served directly by the web server, which is optimized for the mass delivery of static files and works extremely resource-efficiently. There are no changes for all other files, which are still delivered via PHP - including for security reasons.

Creating and Updating Preview Images

Depending on the type of file upload, there is a need for preview images in a wide variety of formats. The new upload pipeline offers a flexible configuration of the required preview images with a wide range of dimensions. A preview image is only generated if the original image is larger than the default on at least one axis.

Of particular interest to operators: the specifications used to generate a preview image are also saved alongside a checksum of the generated file. When rebuilding the data, the system checks whether the required preview images are complete, correspond to the specifications and the checksums match in order to detect damaged files. A new preview image is only generated if one of the aforementioned criteria is not met, otherwise no new image is generated and the check continues with the next file.

Uses of the Upload Pipeline in WoltLab Suite 6.1

Switching from file uploads to the new upload pipeline involves a considerable one-off effort, as it requires the entire component to be replaced. With WoltLab Suite 6.1, we have completely modernized the file attachment system and redeveloped it on top of the upload pipeline.

Our second focus was on providing these file uploads as a ready-made component in the Form Builder, our flexible “construction kit” system for forms. This enables easy use in new functions as well as quick integration into existing pages based on the Form Builder. Developers now have the choice of using the previous UploadFormField, which is available unchanged, or the new FormProcessorFormField.

WoltLab Suite Filebase 6.1 has also been revised by us and all forms have been converted to the Form Builder. In addition to a standardized user interface and much easier extensibility through plugins, WoltLab Suite Filebase 6.1 benefits in particular from the chunked uploads and the removal of the technical limitation to files with a maximum size of 2.1 GB.

With the upcoming versions of WoltLab Suite, we will gradually implement the new upload pipeline in more and more places and also expand the functionality offered as required. We look forward to receiving feedback from developers on GitHub on the use of the new API as well as suggestions for additional use cases that would be suitable for the upload pipeline.