Large Files Management in Plone

Plone is a leading open source CMS, that is well-suited for documents and files management. Plone is very flexible and allows to customize its configuration for work with large files. Unfortunately, for the time being default Plone has some problems with managing large volumes of PDF, media, video and other files.

There are solutions to these problems, it is possible to customize Plone to avoid any difficulties, but what concerns out-of-the-box Plone, it has limitation on work with large files. These are, first of all, Plone-beginners and ordinary Plone users that face default Plone drawbacks. That can be really frustrating when working with such a powerful CMS. That is why lack of a universal tool for managing big files successfully is really topical issue. A sophisticated CMS system, such as Plone is, should provide a transparent way to upload big files and handle them smoothly on the server side.

This is multimedia that often involves large files, so products are being developed aimed to manage large video files efficiently. The need is to improve the handling of multimedia files in Plone, which includes the uploading audio/video files process as well as their storage.

File Storage

There are several places, where files can be stored in Plone. These are:

ZODB (Zope Object Database)
File System
External Servers

1. ZODB's FileStorage (Data.fs)

for ZODB versions < 3.8

The first problem when dealing with big files is the upload procedure. Large files upload can cause RAM shortage, which in their turn involve performance failure in general. Due to the way the ZODB works (default Plone has all its object stored in Zope Object Database (ZODB) after upload), large files can be problematic to work with. There are solutions to this problem, which usually involve storing some content outside the ZODB.

2. Plone FileSystem Storage

Several Plone products were developed 2 years ago to handle external storage of files on the FileSystem. They are probably not as transparent as would perhaps be desirable, because they require the use of specific content types, but they appeared to work well. Although the following products aimed to solve large files management problems, they can not be suitable enough to overcome present default Plone drawbacks.

CMFExtFile
author: Stefan H. Holek

description source: CMFExtFile project page

CMFExtFile replaces the stock File and Image portal types with ExtFile and ExtImage respectively.
PloneExFile
author: ingeniweb

description source: PloneExFile project page
PloneExFile is a replacement content type for the usual File (ATCT or CMF) with additional features for the file types supported by AttachmentField (MS Office, PDF, OpenOffice):
- File content indexing
- File preview through the browser
ATManagedFile
author: Enfold Systems

description source: ATManagedFile project page

ATManagedFile stores files on the file system while keeping a content representation in Plone. ATManagedFile allows you to store metadata in the database while keeping File assets on the servers file system. Content is treated like a first-class citizen in Plone, it can be transitioned (workflow), edited, copy/pasted. Metadata for content can be described through Archetype schema's. A managed_files tool enables administrators to specify where content is stored, the deletion policy, as well as the stored filesystem naming convention.
ExternalStorage
author: Plone Solutions

description source: ExternalStorage README.txt

ExternalStorage is an additional storage implementation for Archetypes, to support storing the fields content outside the ZODB. It works in a fashion like ExternalFile or similar products.
FileSystemStorage
author: ingeniweb

description source: FileSystemStorage project page

FileSystemStorage (FSS) is an Archetypes storage for storing fields raw values on the file system. This storage is used to avoid unnecessary growth of the ZODB's FileStorage (Data.fs) when using a lot of large files.
z3c.extfile
author: Lovely Systems

description source: z3c.extfile README.txt

Filestorage package offers large file handling solutions for zope3. The first implementation is based on properties, the second on wsgi. The property implementation runs on plain zope 3.
This package provides a wsgi filter that upon upload replaces the content of the upload with the sha digest of the content and stores the file on the filesystem. Upon download it looks if it has a digest and returns the according file directly from the filesystem.
Tramline
author: Infrae

description source: Tramline project page

Tramline is an upload and download accelerator that plugs into Apache. Its aim is to make downloading and uploading large media to an application server easy and fast, without overloading the application server with large amounts of binary data.

Tramline integrates into Apache using mod_python. The application server is assumed to sit behind Apache, for instance hooked up using mod_proxy or mod_rewrite. Tramline takes over uploading and downloading files, handling these within Apache. Only a small configuration change in Apache should be necessary to enable Tramline.

The application server remains in complete control over security, page and form rendering, and everything else. Minimal changes are necessary to any application to enable it to work with Tramline; in fact it’s just setting two response headers in a few places in the code.

Even though these products were accepted as a right solutions once, now they are not good enough. Large files management problem remains topical and it was among other topics to be considered on 2008 Plone Strategic Planning Summit in Mountain View, California, USA.

ZODB Blob Support

As a result of the Summit, a ticket was opened, aimed to find a solution to the problem with large files management in Plone 3. This was 7822 ticket concerning file content type BLOB support - Make standard file content types use ZODB BLOB support.

It was decided to create an integration package for ZODB BLOB support for Plone aimed to handle large file more efficiently than it was in previous Plone releases.

The idea with ZODB blob support is not new, it was initiated long ago, and there are several solutions for present time:

blob
Authors: Christian Theune, Chris McDonough
This product is for Zope developers, not end users. It provides an alternative to storing large binary chunks of arbitrary data in ZODB, allowing this kind of data to live instead in files on a FileSystem. Unlike a "roll your own" solution, it provides transactional integrity while doing so. It also provides a simple UNIX-like API for dealing with file objects.
ATBlobField
Author: Enfold Systems
plone.app.blob
Author: Andreas Zeidler
plone.app.blob is supposed to provide a new content type that can be used as a drop-in replacement for ATFile. Plone.app.blob is a package integrating ZODB 3.8's blob support. This allows large binary data to be managed by the ZODB, but separately from the usual Data.fs FileStorage database.

3. Storage at External Servers

Files can be hosted on remote sharing sites (e.g. Youtube, Google Video, Blip.tv, Revver.com etc.), not in your FileSystem. This is the easiest way to deal with video on your instance. All you should do is to paste a piece of <embed> code into Plone. You can insert this code even into a usual document (for that you must have HTML filter disabled), or create a simple content type with the corresponding field for <embed> code. In this case large files are physically located in some external sources, not in your instance.
Example: http://www.bmctv.com, developed by Quintagroup.

_____
tags: Plone 3.0 | Plone products | ZODB | blob

Monday, May 19, 2008 | Permalink | Comments (0)

Document Actions

Print this

Sections

Personal tools