Binary Large Objects (BLOBs)¶
BLOBs are very large, variable length, binary or character data objects. BLOBS are typically used to store documents, pictures, or other variably structured (or unstructured) items.
Note
Related terms, LOB (large object) and CLOB (character large object in Oracle), are used to refer to large objects consisting entirely of text data.
When a DataFlex text field is too large for the native type, the Mertech drivers automatically use the BLOB data type during the migration. For example in MS SQL Server the driver automatically switches from varchar (<string length>) to varchar (max) when database limitations are hit.
As long as the size of the object fits in a regular DataFlex string, the object is bound as any regular TEXT column. The Mertech drivers transparently handle this, defaulting to non-BLOB data types if possible.
Note
Mertech suggests using non-BLOB data types, whenever possible, if designing database tables outside of Mertech’s tools. For example, when using available SQL modeling tools to manage your database structure.
BLOBs can either be stored inside the database or in the file system. Storing BLOB data in the database offers a number of advantages:
Security measures used to control access to the database can also be used to control access to BLOBs.
Backup routines used to backup the database can be used to capture BLOB data as well, simplifying maintenance.
Transaction control, if required, is built into the database.
Search operations can be performed against formatted text-based data contained within BLOB fields—for example, Microsoft Word or Microsoft Excel documents.
The file system may be a better storage choice for BLOBs when:
The object requires significant overhead to process, e.g., streaming video.
The cost of storage space is a consideration.
The object will be modified frequently. The file system may handle fragmentation better than the database server.
For all databases, there is a slight performance penalty fetching BLOB data, because BLOB data is not stored in the same internal page as the rest of the record.
MS SQL Server¶
There are special considerations when using MS SQL Server. If you use embedded SQL and access LOB columns, you have to specify a server side cursor in your call to SQL_PREPARE_STMT. MS SQL Server enforces cursor limitations on LOB columns and transactions.
MySQL & MariaDB¶
By default, the max_allowed_packet size attribute is set to 16MB for MySQL 8.x and MariaDB 10.x.
max_allowed_packet=16M
To use the improved LOB commands, you must edit the my.ini file and set this attribute to at least 32MB.
max_allowed_packet=32M
Note
The MySQL Configuration Wizard places the my.ini file in the MySQL server installation directory.
PostgreSQL¶
The BLOB implementation in PostgreSQL uses the Large Object feature. When a BLOB is created, and an OID (object identifier) is saved in a column in your table and the BLOB is stored separately. Physically an OID is an unsigned 32 bit integer that is a reference to a binary file object.
Note
MERTECH.INC includes a definition for data type OID: ePgSQL_OID
The contents referenced by an OID cannot be read using any SQL functions or directives. In order to read the contents of the BLOB file, one has to read the OID from a database record, open the BLOB file, do the reading and then close it.
The OID field has to be nullable. A NULL value means an empty BLOB. If you write something into that BLOB using LOB_WRITE or LOB_APPEND, the driver automatically creates a new LOB object and writes its OID into the corresponding database field.
There are several significant things that differ in the PostgreSQL BLOB implementation from all other drivers:
BLOB columns cannot be converted from regular text columns like VARCHAR etc.
You cannot read the data stored within BLOBs using regular SAVE_RECORD / UPDATE_RECORD calls. You can only manipulate raw OIDs themselves.
All BLOB related calls must happen within a transaction even if no modifications are performed. In each BLOB API call the driver checks if there is a transaction in effect. If not, it starts it and commits it after the call.
PostgreSQL Example¶
Below is an example of turning a PostgreSQL TEXT column into an OID column.
Note
To turn a PostgreSQL TEXT column into an OID column you must delete and then recreate the column*.
MERTECH_WARNING_MESSAGE DISABLED
Open "CUSTOMER.INT" as CUSTOMER
Move CUSTOMER.File_Number to iFile
Structure_Start iFile "MDSPGSQL
Move 13 to iField // CUSTOMER.COMMENTS, note this is the last column in the SQL table
Delete_Field iFile iField
Create_Field iFile At iField
Set_Attribute DF_FIELD_NAME of iFile iField to "COMMENTS
Set_Attribute DF_FIELD_TYPE of iFile iField to DF_ASCII
Set_Attribute DF_FIELD_NATIVE_TYPE of iFile iField to ePgSQL_OID
Set_Attribute DF_FIELD_LENGTH of iFile iField to 8016
Set_Attribute DF_FIELD_NULL of iFile iField to 1
Structure_End iFile DF_STRUCTEND_OPT_NONE "." 0
Close CUSTOMER
Improved LOB Access¶
In version 14, Mertech added new LOB commands that are independent of the field buffer and allow reading and writing from/to a LOB field in chunks up to 64MB (the previous maximum was 16KB). The new access method is also much faster (20x) than the old method. The new LOB commands introduced in version 14.0 are:
Command |
Description |
---|---|
SQL_SET_LOB |
Save data to a LOB field. If the field is larger than what is allowed by DataFlex, SQL_ADD_LOB_CHUNK is used in a loop to add all the data to the field. The save is performed separately. |
SQL_ADD_LOB_CHUNK |
Add additional chunks to the buffer before saving the record. The save is performed separately. |
SQL_GET_LOB |
Retrieve a chunk of data from a specified LOB field. |
SQL_GET_LOB_CHUNK |
Retrieve a chunk of data from a LOB field, specifying the offset and length of the chunk to be retrieved. |
SQL_GET_LOB_LENGTH |
Get the length of the specified LOB field. |
SQL_SET_LOB_NULL |
Set a LOB column to null. |
Example reading a LOB field:
Procedure LoadImage
Integer iImageLen
Integer iOldVal iChunk iSize ISoFar iChunksize
Move 2097152 to ChunkSize // 2 MB chunks
SQL_GET_LOB_LENGTH salesp.icture to iImageLen
If (iImageLen > 0) Begin
Get_Argument_Size to iOldVal
Set_Argument_Size iImageLen
String sChunk SBitmap
UChar[] ucFileAs Array
Address pStr
//The Repeat loop is shown below to demonstrate how chunking works. This loop could //have been replaced with a call to SQL_GET_LOB salesp.picture sBitmap
Repeat
SQL_GET_LOB_CHUNK salesp.picture to sChunk OFFSET (iChunkSize \* iChunk) Length iChunkSize
Move (sBitmap + sChuck) to sBitmap
Increment iChunk
Add iChunkSize to iSoFar
Until (iSoFar > iImageLen
//Code follows to convert the image from sBitmap to the type expected by the image control
end
End_Procedure
Note: MERTECH.INC contains a full description and examples of each command. You can also find additional examples and information on the YouTube video Mertech Campfire – Introduction to Flex2SQLv14.
Note: The old LOB commands are still supported.