Binary Large Objects (BLOBs)

BLOBs are very large, variable length, binary or character data objects. BLOBS are typically used to store documents, pictures, or other variably structured (or unstructured) items.

Note

Related terms, LOB (large object) and CLOB (character large object in Oracle), are used to refer to large objects consisting entirely of text data.

When a DataFlex text field is too large for the native type, the Mertech drivers automatically use the BLOB data type during the migration. For example in MS SQL Server the driver automatically switches from varchar (<string length>) to varchar (max) when database limitations are hit.

As long as the size of the object fits in a regular DataFlex string, the object is bound as any regular TEXT column. The Mertech drivers transparently handle this, defaulting to non-BLOB data types if possible.

Note

Mertech suggests using non-BLOB data types, whenever possible, if designing database tables outside of Mertech’s tools. For example, when using available SQL modeling tools to manage your database structure.

BLOBs can either be stored inside the database or in the file system. Storing BLOB data in the database offers a number of advantages:

  • Security measures used to control access to the database can also be used to control access to BLOBs.

  • Backup routines used to backup the database can be used to capture BLOB data as well, simplifying maintenance.

  • Transaction control, if required, is built into the database.

  • Search operations can be performed against formatted text-based data contained within BLOB fields—for example, Microsoft Word or Microsoft Excel documents.

The file system may be a better storage choice for BLOBs when:

  • The object requires significant overhead to process, e.g., streaming video.

  • The cost of storage space is a consideration.

  • The object will be modified frequently. The file system may handle fragmentation better than the database server.

For all databases, there is a slight performance penalty fetching BLOB data, because BLOB data is not stored in the same internal page as the rest of the record.

MS SQL Server

There are special considerations when using MS SQL Server. If you use embedded SQL and access LOB columns, you have to specify a server side cursor in your call to SQL_PREPARE_STMT. MS SQL Server enforces cursor limitations on LOB columns and transactions.

MySQL & MariaDB

By default, the max_allowed_packet size attribute is set to 16MB for MySQL 8.x and MariaDB 10.x.

max_allowed_packet=16M

To use the improved LOB commands, you must edit the my.ini file and set this attribute to at least 32MB.

max_allowed_packet=32M

Note

The MySQL Configuration Wizard places the my.ini file in the MySQL server installation directory.

PostgreSQL

The BLOB implementation in PostgreSQL uses the Large Object feature. When a BLOB is created, and an OID (object identifier) is saved in a column in your table and the BLOB is stored separately. Physically an OID is an unsigned 32 bit integer that is a reference to a binary file object.

Note

MERTECH.INC includes a definition for data type OID: ePgSQL_OID

The contents referenced by an OID cannot be read using any SQL functions or directives. In order to read the contents of the BLOB file, one has to read the OID from a database record, open the BLOB file, do the reading and then close it.

The OID field has to be nullable. A NULL value means an empty BLOB. If you write something into that BLOB using LOB_WRITE or LOB_APPEND, the driver automatically creates a new LOB object and writes its OID into the corresponding database field.

There are several significant things that differ in the PostgreSQL BLOB implementation from all other drivers:

  • BLOB columns cannot be converted from regular text columns like VARCHAR etc.

  • You cannot read the data stored within BLOBs using regular SAVE_RECORD / UPDATE_RECORD calls. You can only manipulate raw OIDs themselves.

  • All BLOB related calls must happen within a transaction even if no modifications are performed. In each BLOB API call the driver checks if there is a transaction in effect. If not, it starts it and commits it after the call.

PostgreSQL Example

Below is an example of turning a PostgreSQL TEXT column into an OID column.

Note

To turn a PostgreSQL TEXT column into an OID column you must delete and then recreate the column*.

MERTECH_WARNING_MESSAGE DISABLED
Open "CUSTOMER.INT" as CUSTOMER
Move CUSTOMER.File_Number to iFile
Structure_Start iFile "MDSPGSQL
    Move 13 to iField // CUSTOMER.COMMENTS, note this is the last column in the SQL table
    Delete_Field iFile iField
    Create_Field iFile At iField
    Set_Attribute DF_FIELD_NAME of iFile iField to "COMMENTS
    Set_Attribute DF_FIELD_TYPE of iFile iField to DF_ASCII
    Set_Attribute DF_FIELD_NATIVE_TYPE of iFile iField to ePgSQL_OID
    Set_Attribute DF_FIELD_LENGTH of iFile iField to 8016
    Set_Attribute DF_FIELD_NULL of iFile iField to 1
Structure_End iFile DF_STRUCTEND_OPT_NONE "." 0
Close CUSTOMER

Improved LOB Access

In version 14, Mertech added new LOB commands that are independent of the field buffer and allow reading and writing from/to a LOB field in chunks up to 64MB (the previous maximum was 16KB). The new access method is also much faster (20x) than the old method. The new LOB commands introduced in version 14.0 are:

Command

Description

SQL_SET_LOB

Save data to a LOB field. If the field is larger than what is allowed by DataFlex, SQL_ADD_LOB_CHUNK is used in a loop to add all the data to the field. The save is performed separately.

SQL_ADD_LOB_CHUNK

Add additional chunks to the buffer before saving the record. The save is performed separately.

SQL_GET_LOB

Retrieve a chunk of data from a specified LOB field.

SQL_GET_LOB_CHUNK

Retrieve a chunk of data from a LOB field, specifying the offset and length of the chunk to be retrieved.

SQL_GET_LOB_LENGTH

Get the length of the specified LOB field.

SQL_SET_LOB_NULL

Set a LOB column to null.

Example reading a LOB field:

Procedure LoadImage
    Integer iImageLen
    Integer iOldVal iChunk iSize ISoFar iChunksize
    Move 2097152 to ChunkSize // 2 MB chunks
    SQL_GET_LOB_LENGTH salesp.icture to iImageLen
    If (iImageLen > 0) Begin
        Get_Argument_Size to iOldVal
        Set_Argument_Size iImageLen
        String sChunk SBitmap
        UChar[] ucFileAs Array
        Address pStr
        //The Repeat loop is shown below to demonstrate how chunking works. This loop could //have been replaced with a call to SQL_GET_LOB salesp.picture sBitmap
        Repeat
            SQL_GET_LOB_CHUNK salesp.picture to sChunk OFFSET (iChunkSize \* iChunk) Length iChunkSize
            Move (sBitmap + sChuck) to sBitmap
            Increment iChunk
            Add iChunkSize to iSoFar
        Until (iSoFar > iImageLen
        //Code follows to convert the image from sBitmap to the type expected by the image control
    end
End_Procedure

Note: MERTECH.INC contains a full description and examples of each command. You can also find additional examples and information on the YouTube video Mertech Campfire – Introduction to Flex2SQLv14.

Note: The old LOB commands are still supported.