1 of 8

v1.1.0

Model9 Gravity

Monetize unlocked mainframe data in business intelligence (BI), analytics and cloud applications

Leverage any disk data or historical tape data for use by analytics services and BI tools. Mainframe data migrated to object storage on-premises or in the cloud can be transformed to standard data formats without requiring any access to the mainframe, instantly providing it for use in cloud applications and analytics tools.

For any questions, contact us at support@model9.io

Model9 website: www.model9.io

Installation

Instructions for installing Model9 Gravity in an on-premise environment.

Introduction

The following guides provides details on how to install and configure the Model9 Gravity application.

Prerequisites

8 vCPU, 32gb RAM instance, 1TB of storage (preferably SSD).
- AWS instance types:
  - m5d.2xlarge
  - m5ad.2xlarge
- docker is expected to be installed on the instance.
  - A user with docker privileges will be required for running docker commands.

Installation

Depending on the system's configuration, root access might be required for some steps. Using a root user for the installation is recommended.

Download Model9 Gravity installation package file

Obtain the Model9 Gravity installation package from Model9 and move it to the target installation system.

The recommended location for the package files on the installation system is:

/data/model9/gravity/packages

Prepare installation

Set version environment variable

export GRAVITY_VERSION=<version>

In the snippet above, <version> represents the package version number. E.g., "1.0.0".

Exported environment variables will be used by the following installation steps.

Extract the package file

export GRAVITY_PACKAGES=/data/model9/gravity/packages
cd $GRAVITY_PACKAGES
tar -xzvf model9-gravity-$GRAVITY_VERSION.tar.gz
export GRAVITY_EXTRACTED=$(pwd)/model9-gravity-$GRAVITY_VERSION
export GRAVITY_BUILD=$(cat $GRAVITY_EXTRACTED/build-id.txt)

Create installation target folder structure

export GRAVITY_HOME=/data/model9/gravity
mkdir -p $GRAVITY_HOME/config
mkdir -p $GRAVITY_HOME/keys
mkdir -p $GRAVITY_HOME/logs
mkdir -p $GRAVITY_HOME/work

$GRAVITY_HOME/work is the work directory used by the service for data processing. It should be mounted on a block-device on which enough space is available.

Create Model9 Gravity PostgreSQL database container

A user with docker privileges or root access will be required for the following steps.

Create docker network bridge

docker network create gravity-bridge

Load the PostgreSQL docker image

docker load --input $GRAVITY_EXTRACTED/postgres-14.2-x86.docker.gz

Run docker PostgreSQL container

docker run --detach \
    --net gravity-bridge \
    --volume $GRAVITY_HOME/db/data:/var/lib/postgresql/data:z \
    --publish 0.0.0.0:5432:5432 \
    --name gravitydb \
    --restart unless-stopped \
    --env POSTGRES_PASSWORD=model9 \
    --env POSTGRES_DB=gravitydb postgres:14.2

model9 is chosen here as the default database password.

Create database

docker exec gravitydb psql -h 127.0.0.1 -p 5432 -U postgres -c 'CREATE DATABASE gravity;'

Configure Model9 Gravity

Copy key stores into the `$GRAVITY_HOME/keys` folder

cp $GRAVITY_EXTRACTED/keys/* $GRAVITY_HOME/keys

The key stores provided in the package include the default self-signed Model9 certificates for setting up TLS in the Gravity service. If required, these key stores can be replaced with key stores provided by the organization.

Create the Model9 Gravity configuration file

cp $GRAVITY_EXTRACTED/application.properties $GRAVITY_HOME/config

After creating the application.properties file edit it and fill in the missing values:

vi $GRAVITY_HOME/config/application.properties

vi is usually available by default on Linux systems. If more convenient, other editors such as nano can also be used.

# ----------------------------------------
# Model9 Gravity Configuration
# Documentation: https://docs.model9.io
# Support: support@model9.io
# ----------------------------------------
model9.gravity.global.license-key=
logging.level.io.model9=INFO

If the default database password has been changed, use the model9.gravity.datasource.password configuration option to change it.

For details of the configuration options and their default values, see the Configuration page.

Start the Model9 Gravity service

Load the Model9 Gravity docker image

docker load --input $GRAVITY_EXTRACTED/model9-gravity-$GRAVITY_VERSION-build-$GRAVITY_BUILD.docker.gz

Start the Model9 Gravity container

docker run --detach --publish 443:443 \
    --net gravity-bridge \
    --volume $GRAVITY_HOME/config:/data/model9/config:z \
    --volume $GRAVITY_HOME/keys:/data/model9/keys:z \
    --volume $GRAVITY_HOME/logs:/data/model9/logs:z \
    --volume $GRAVITY_HOME/work:/data/model9/work:z \
    --env "TZ=America/New_York" \
    --env "JAVA_OPTS=-Xmx28g" \
    --restart unless-stopped \
    --name model9gravity-v$GRAVITY_VERSION model9/gravity:$GRAVITY_VERSION.$GRAVITY_BUILD

Change the TZ environment variable as appropriate.
Change the -Xmx value to reflect the amount of available memory for the service (in GBs) - Remember to leave a few free GBs for the operating system.
443 is the default secure port, if running the service on another port, this value should be changed as well.

Configuration

Server configuration

Database Configuration

Transformation Configuration

All configuration options

User Guide

Service Limitations

Data size

There is no limit for the input data set size, as long as the output data set size will utilize under 20 GB of storage.

Maximum elapsed time

The service is synchronous and can run for up to an hour.

Deployment and Initial Usage

This guide will instruct you how to set up and invoke the Model9 Gravity service from the mainframe, using JCL. The service will transform a Model9 data set backup copy / archive / import into a readable file in the cloud. Once transformed, the readable file can be accessed directly or via data analytics tools.

Using Gravity

Step 1: Prerequisites

Verify that Model9 Cloud Backup and Recovery for z/OS is installed

Model9 is responsible for delivering the data set from the mainframe to the cloud / on-premises storage. The data set is delivered as a backup copy, an archive or imported tape data set, and provides the input to the transform service.

Download z/OS cURL

This free tool will allow you to invoke the transform service from z/OS. If cURL is not installed under /usr/bin, edit line 4 and add the path where the cURL module resides.

Step 2: Copy the following script

Copy the following script to /usr/lpp/model9/gravity-run.sh:

#!/bin/sh
json=$1
url='https://<gravity-url>:<port>/transform'
export PATH=/usr/bin:/bin
export _EDC_ADD_ERRNO2=1
cnvto="$(locale codeset)"
headers="Content-Type:application/json"
echo "Running Model9 Gravity"
output=$(curl -H "$headers" -s -k -X POST --data "$json" $url)
if ! [ -z "$output" ]; then
   echo "Transformation ended with the following output:"
   # If the answer is in ASCII then convert to EBCDIC
   firstChar=$(echo $output | cut -c1)
   if [ "$firstChar" = "#" ]; then
      convOutput="$(echo $output | iconv -f ISO8859-1 -t $cnvto)"
   else
      convOutput=$output
   fi
   echo "$convOutput"
fi
status=$(echo $convOutput | tr -s " " | cut -d, -f1 | cut -d" " -f3)
   echo "Transformation ended with status: $status"
if [ "$status" = '"OK"' ];then
   exit 0
else if [ "$status" = '"WARNING"' ]; then
   exit 4
else
   exit 8
fi
fi

Step 3: Copy the following JCL

Copy the following JCL to a local library, update the JOBCARD according to your site standards:

//M9TRNSFM JOB 'ACCT#',REGION=0M,CLASS=A,NOTIFY=&SYSUID
//EXTRACT  EXEC PGM=BPXBATCH
//STDOUT   DD SYSOUT=*
//STDERR   DD SYSOUT=*
//STDPARM  DD *
SH /usr/lpp/model9/gravity-run.sh
//         DD *,SYMBOLS=EXECSYS
'{
    "input": {
        "name"   : "<DATA-SET>",
        "complex": "group-&SYSPLEX",
        "type": "<BACKUP|ARCHIVE|IMPORT>"
    },
    "output": {
        "prefix"      : "/transform/&LYR4/&LMON/&LDAY",
        "compression" : "none",
        "format"      : "text"
    },
    "source": {
        "url"     : "<URL>",
        "api"     : "<API>",
        "bucket"  : "<BUCKET>",
        "user"    : "<USER>",
        "password": "<PASSWORD>"
    }
}'
/*
//

Step 4: Customize the JCL

Update the object storage details

Copy the following object storage variables from the Model9 agent configuration file:

<URL>
<API>
<BUCKET>
<USER>
<PASSWORD>

Update the complex name

The "complex” name represents the group of resources that the Model9 agent can access. By default, this group is named group-<SYSPLEX> and it is shared by all the agents in the same sysplex. The transform JCL specifies the default, using the z/OS system symbol &SYSPLEX.

If the default was kept for "complex" in the Model9 agent configuration file, no change is needed

If the "complex” name was changed in the Model9 agent configuration file, change the "complex” in the JCL accordingly.

Update the transform prefix

By default, the JCL will create a transformed copy of your input data set, in the same bucket, with the prefix: /transform/&LYR4/&LMON/&LDAY. The prefix is using the following z/OS system symbols:

&LYR4 - The year in 4 digits, e.g. 2019
&LMON - The month in 2 digits, e.g. 08
&LDAY - The day in the month in 2 digits, e.g. 10 You can change the prefix according to your needs.

Step 5: Choose the data set to be transformed

The data set to be transformed should be a backup copy, an archive or imported tape data set, delivered by the Model9 agent:

<DATA-SET> - the name of the data set
<BACKUP|ARCHIVE|IMPORT> - if the data set is a Model9 backup, archive or import

To change the attributes of the input and the output, and for a full description of the service parameters, see Service parameters.

Step 6: Run the JCL

Submit the job and view the output. See Service response and log samples for sample output.

Step 7: Access the transformed data

Based on the returned response, the outputName will point to the path inside the bucket where the transformed data resides. See Service response and log samples.

User Guide

Service Parameters

To run the service, specify the source object storage and identify the input data set.

REQUIRED: "source"

Identify the transform source object storage, where the input resides. The source object storage details appear in the Model9 agent configuration file.

Required Keywords for "source"

{
  "source": {
    "url": "<URL>",
    "api": "<API>",
    "bucket": "<USER_BUCKET>",
    "user": "<USERID>",
    "password": "<PASSWORD>"
  }
}

Optional Keywords for "source"

{
  "source": {
    "useS3V4Signatures": "false|true"
  }
}

OPTIONAL: "target"

Identify the transform target object storage. Values not specified will be taken from the "source" parameter.

{
  "target": {
    "url": "<URL>",
    "api": "<API>",
    "bucket": "<USER-BUCKET>",
    "user": "<USERID>",
    "password": "<PASSWORD>",
    "useS3V4Signatures": "false|true"
  }
}

REQUIRED: "input"

If you specify VSAM keywords for a sequential input data set, the transform will be performed and a warning message will be issued

Required Keywords for "input"

{
  "input": {
    "name": "<DSN>",
    "complex": "<group-SYSPLEX>"
  }
}

Optional Keywords for "input"

{
  "input": {
    "type": "backup|archive|import",
    "entry": "0|<N>",
    "prefix": "model9|<USER-PREFIX>",
    "recordBinary": "false|true",
    "recordCharset": "<CHARSET>",
    "vsam": {
      "keyBinary": "false|true",
      "keyCharset": "<CHARSET>"
    }
  }
}

OPTIONAL: "output"

The output is the transformed data of the MF data set, accessible as S3 object

When transforming a file with the same name as an existing file in the target, the existing file will be replaced by the newly transformed file.
Note that the service does not delete previously transformed files but rather overwrites files with the same name, so when re-transforming a file using the "split" function, ensure to remove any previously transformed files to avoid having split files of different versions.
When splitting a file, wait for the successful completion of the transform function before continuing with the processing, to insure all the parts of a the file were created.
Specifying "text" format for a "binary" input will cause the transform to fail.

{
  "output": {
    "prefix": "model9|<USER-PREFIX>",
    "compression": "none|gzip",
    "format": "JSON|text|CSV|RAW",
    "charset": "UTF8",
    "endWithNewLine": "false|true",
    "splitBySize": "<nnnnb/m/g>",
    "splitByRecords": "<n>"
  }
}

Service parameters samples

Transforming a plain text data set

Transform the latest backup of a plain text data set, charset IBM-1047, converted to UTF8 and compressed.

{
  "input": {
    "name": "SAMPLE.TEXT",
    "complex": "group-PLEX1"
  },
  "output": {
    "format": "text"
  },
  "source": {
    "url": "https://s3.amazonaws.com",
    "api": "aws-s3",
    "bucket": "prod-bucket",
    "user": "sdsdDVDCsxadA43TERVGFBSDSSDff",
    "password": "ddferdscsdW4REFEBA33DSffss344gbs4efe7"
  }
}

Transforming an unloaded DB2 table

Transform the latest backup of an unloaded DB2 table, charset IBM-1047, converted to UTF8 and compressed, located with a specific prefix:

{
  "input": {
    "name": "DB2.UNLOADED.SEQ",
    "complex": "group-PLEX1"
  },
  "output": {
    "format": "text"
  },
  "source": {
    "url": "https://s3.amazonaws.com",
    "api": "aws-s3",
    "bucket": "prod-bucket",
    "user": "sdsdDVDCsxadA43TERVGFBSDSSDff",
    "password": "ddferdscsdW4REFEBA33DSffss344gbs4efe7"
  },
  "output": {
    "prefix": "DBprodCustomers"
  }
}

Transforming a VSAM file using the defaults

When transforming a VSAM file, the defaults are a text key and binary data, transforming to a JSON output file:

{
  "input": {
    "name": "SAMPLE.VSAM",
    "complex": "group-PLEX1"
  },
  "source": {
    "url": "https://s3.amazonaws.com",
    "api": "aws-s3",
    "bucket": "prod-bucket",
    "user": "sdsdDVDCsxadA43TERVGFBSDSSDff",
    "password": "ddferdscsdW4REFEBA33DSffss344gbs4efe7"
  }
}

Transforming a VSAM text file to CSV

Specify a text data, transforming to a CSV output file:

{
  "input": {
    "name": "SAMPLE.VSAM",
    "complex": "group-PLEX1"
  },
  "vsam": {
    "keyBinary": "false|true",
    "keyCharset": "<CHARSET>"
  },
  "output": {
    "format": "CSV"
  },
  "source": {
    "url": "https://s3.amazonaws.com",
    "api": "aws-s3",
    "bucket": "prod-bucket",
    "user": "sdsdDVDCsxadA43TERVGFBSDSSDff",
    "password": "ddferdscsdW4REFEBA33DSffss344gbs4efe7"
  }
}

Transforming on Azure Storage using OAuth2

When transforming data on Azure blob storage with OAuth2 set the "api" to azureblob-oauth2 and use the azureOauth section to specify Azure OAuth arguments as follows:

{
  "input": {
    "name": "SAMPLE.PS",
    "complex": "group-PLEX1"
  },
  "vsam": {
    "keyBinary": "false|true",
    "keyCharset": "<CHARSET>"
  },
  "output": {
    "format": "CSV"
  },
  "source": {
    "api": "azureblob-oauth2",
    "url": "https://<azure-storage-account>.blob.core.windows.net",
    "bucket": "<azure-container-name>",
    "user": "<azure-application-uuid>",
    "password": "<azure-application-client-secret>",
    "azureOauth": {
      "oauthEndpoint": "<azure-oauth-endpoint>",
      "storageAccount": "<azure-storage-account>",
      "oauthAudience": "<azure-oauth-audience>",
      "credentialType": "<azure-credential-type>"
    }
  }
}

Table: Azure OAuth2 Arguments

Service response and log

The transform service is invoked as an HTTP request. It returns:

HTTP status

HTTP response

{
  "status": "OK|WARNING|ERROR",
  "outputName": "<OUTPUT-NAME>",
  "inputName": "<DSN>",
  "outputCompression": "none|gzip",
  "outputSizeInBytes": "<SIZE-IN_BYTES>",
  "outputFormat": "JSON|text|CSV"
}

In case of a WARNING or an ERROR - the HTTP response will also contain log messages.

Informational messages are printed only to service log and not to the HTTP response. The service log can be viewed on the AWS console when executing the service from AWS, or the docker log, when executing the service on-premises.

Log

{
  "log": [
    "<INFO-MESSAGE>",
    "<WARNING-MESSAGE>",
    "<ERROR-MESSAGE>"
  ]
}

Service response and log samples

Status OK sample

{
  "status": "OK",
  "outputName": "transform/QA.SMS.MCBK.SG1QNOBK.DSERV.TXT.TMPPS!uuid=a641d670-2d05-41e7-9dd3-7815e1b2d4c4",
  "inputName": "QA.SMS.MCBK.SG1QNOBK.DSERV.TXT.TMPPS",
  "outputCompression": "NONE",
  "outputSizeInBytes": 97,
  "outputFormat": "JSON"
}

Status WARNING sample

{
  "log": [
    "ZM9K001I Transform service started",
    "ZM9K108W Specifying input parameter vsam is ignored for input data set with DSORG PS",
    "ZM9K002I Transform service completed successfully, output is transform/QA.SMS.MCBK.SG1QNOBK.DSERV.TXT.TMPPS!uuid=d779fbf9-da6b-495b-b6b9-de7583905f19"
  ],
  "status": "WARNING",
  "outputName": "transform/QA.SMS.MCBK.SG1QNOBK.DSERV.TXT.TMPPS!uuid=d779fbf9-da6b-495b-b6b9-de7583905f19",
  "inputName": "QA.SMS.MCBK.SG1QNOBK.DSERV.TXT.TMPPS",
  "outputCompression": "NONE",
  "outputSizeInBytes": 97,
  "outputFormat": "JSON"
}

Status ERROR sample

{
  "status": "ERROR",
  "log": [
    "ZM9K001I Transform service started",
    "ZM9K008E The input was not found: name QA.SMS.MCBK.DSERV.TXT.NON, archive false, entry (0)"
  ]
}

Input format support

Supported formats

SMS-managed data sets
Non-SMS managed data sets
Sequential and extended-sequential data sets with the following RECFM:
- V
- VB
- F
- FB
- FBA
Non-extended VSAM KSDS data sets

Unsupported formats

RRDS, VRRDS, LINEAR, ESDS
Extended format data sets with compression or encryption
PDS data sets
RECFM not mentioned above (U)

Output format support

Text
JSON
CSV

DB2 Image Copy Transform Guide

Configuration

Make sure that <M9_HOME>/scripts/transform-service.sh has execute permissions. If not, add it by using chmod a+x <M9_HOME>/scripts/transform-service.sh.
Copy M9XFDB2 from Model9's SAMPLIB data set to a PDS data set of your choosing.
Edit M9XFDB2 and replace the placeholders enclosed with angle brackets with the following:

Table: Placeholders

Replace the remaining placeholders in the JCL as described in this manual.

Execute and verify results

When done, submit the job and make sure it ends with MAXCC of 0.

Via SDSF, verify that the transform service was in fact called and completed successfully. Successful output would look something like this:

{
  "status": "OK",
  "outputNames": [
    "transform-output/M9.SHY.DB2.IMGCPY.M9DB.M9SEG4"
  ],
  "inputName": "M9.SHY.DB2.IMGCPY.M9DB.M9SEG4",
  "outputCompression": "NONE",
  "outputSizeInBytes": 1064,
  "outputFormat": "CSV"
}

Supported DB2 Column Types

Table 5. Supported DB2 Column Types for Transformation

Deployment and Initial Usage

Using Gravity

Step 1: Prerequisites

Verify that Model9 Cloud Backup and Recovery for z/OS is installed

Download z/OS cURL

This free tool will allow you to invoke the transform service from z/OS. If cURL is not installed under /usr/bin, edit line 4 and add the path where the cURL module resides.

Step 2: Copy the following script

Copy the following script to /usr/lpp/model9/gravity-run.sh:

#!/bin/sh
json=$1
url='https://<gravity-url>:<port>/transform'
export PATH=/usr/bin:/bin
export _EDC_ADD_ERRNO2=1
cnvto="$(locale codeset)"
headers="Content-Type:application/json"
echo "Running Model9 Gravity"
output=$(curl -H "$headers" -s -k -X POST --data "$json" $url)
if ! [ -z "$output" ]; then
   echo "Transformation ended with the following output:"
   # If the answer is in ASCII then convert to EBCDIC
   firstChar=$(echo $output | cut -c1)
   if [ "$firstChar" = "#" ]; then
      convOutput="$(echo $output | iconv -f ISO8859-1 -t $cnvto)"
   else
      convOutput=$output
   fi
   echo "$convOutput"
fi
status=$(echo $convOutput | tr -s " " | cut -d, -f1 | cut -d" " -f3)
   echo "Transformation ended with status: $status"
if [ "$status" = '"OK"' ];then
   exit 0
else if [ "$status" = '"WARNING"' ]; then
   exit 4
else
   exit 8
fi
fi

Step 3: Copy the following JCL

Copy the following JCL to a local library, update the JOBCARD according to your site standards:

//M9TRNSFM JOB 'ACCT#',REGION=0M,CLASS=A,NOTIFY=&SYSUID
//EXTRACT  EXEC PGM=BPXBATCH
//STDOUT   DD SYSOUT=*
//STDERR   DD SYSOUT=*
//STDPARM  DD *
SH /usr/lpp/model9/gravity-run.sh
//         DD *,SYMBOLS=EXECSYS
'{
    "input": {
        "name"   : "<DATA-SET>",
        "complex": "group-&SYSPLEX",
        "type": "<BACKUP|ARCHIVE|IMPORT>"
    },
    "output": {
        "prefix"      : "/transform/&LYR4/&LMON/&LDAY",
        "compression" : "none",
        "format"      : "text"
    },
    "source": {
        "url"     : "<URL>",
        "api"     : "<API>",
        "bucket"  : "<BUCKET>",
        "user"    : "<USER>",
        "password": "<PASSWORD>"
    }
}'
/*
//

Step 4: Customize the JCL

Update the object storage details

Copy the following object storage variables from the Model9 agent configuration file:

<URL>
<API>
<BUCKET>
<USER>
<PASSWORD>

Update the complex name

If the default was kept for "complex" in the Model9 agent configuration file, no change is needed

If the "complex” name was changed in the Model9 agent configuration file, change the "complex” in the JCL accordingly.

Update the transform prefix

By default, the JCL will create a transformed copy of your input data set, in the same bucket, with the prefix: /transform/&LYR4/&LMON/&LDAY. The prefix is using the following z/OS system symbols:

&LYR4 - The year in 4 digits, e.g. 2019
&LMON - The month in 2 digits, e.g. 08
&LDAY - The day in the month in 2 digits, e.g. 10 You can change the prefix according to your needs.

Step 5: Choose the data set to be transformed

The data set to be transformed should be a backup copy, an archive or imported tape data set, delivered by the Model9 agent:

<DATA-SET> - the name of the data set
<BACKUP|ARCHIVE|IMPORT> - if the data set is a Model9 backup, archive or import

To change the attributes of the input and the output, and for a full description of the service parameters, see Service parameters.

Step 6: Run the JCL

Submit the job and view the output. See Service response and log samples for sample output.

Step 7: Access the transformed data

Based on the returned response, the outputName will point to the path inside the bucket where the transformed data resides. See Service response and log samples.

Installation

Instructions for installing Model9 Gravity in an on-premise environment.

Introduction

The following guides provides details on how to install and configure the Model9 Gravity application.

Prerequisites

8 vCPU, 32gb RAM instance, 1TB of storage (preferably SSD).
- AWS instance types:
  - m5d.2xlarge
  - m5ad.2xlarge
- docker is expected to be installed on the instance.
  - A user with docker privileges will be required for running docker commands.

Installation

Depending on the system's configuration, root access might be required for some steps. Using a root user for the installation is recommended.

Download Model9 Gravity installation package file

Obtain the Model9 Gravity installation package from Model9 and move it to the target installation system.

The recommended location for the package files on the installation system is:

/data/model9/gravity/packages

Prepare installation

Set version environment variable

export GRAVITY_VERSION=<version>

In the snippet above, <version> represents the package version number. E.g., "1.0.0".

Exported environment variables will be used by the following installation steps.

Extract the package file

export GRAVITY_PACKAGES=/data/model9/gravity/packages
cd $GRAVITY_PACKAGES
tar -xzvf model9-gravity-$GRAVITY_VERSION.tar.gz
export GRAVITY_EXTRACTED=$(pwd)/model9-gravity-$GRAVITY_VERSION
export GRAVITY_BUILD=$(cat $GRAVITY_EXTRACTED/build-id.txt)

Create installation target folder structure

export GRAVITY_HOME=/data/model9/gravity
mkdir -p $GRAVITY_HOME/config
mkdir -p $GRAVITY_HOME/keys
mkdir -p $GRAVITY_HOME/logs
mkdir -p $GRAVITY_HOME/work

$GRAVITY_HOME/work is the work directory used by the service for data processing. It should be mounted on a block-device on which enough space is available.

Create Model9 Gravity PostgreSQL database container

A user with docker privileges or root access will be required for the following steps.

Create docker network bridge

docker network create gravity-bridge

Load the PostgreSQL docker image

docker load --input $GRAVITY_EXTRACTED/postgres-14.2-x86.docker.gz

Run docker PostgreSQL container

docker run --detach \
    --net gravity-bridge \
    --volume $GRAVITY_HOME/db/data:/var/lib/postgresql/data:z \
    --publish 0.0.0.0:5432:5432 \
    --name gravitydb \
    --restart unless-stopped \
    --env POSTGRES_PASSWORD=model9 \
    --env POSTGRES_DB=gravitydb postgres:14.2

model9 is chosen here as the default database password.

Create database

docker exec gravitydb psql -h 127.0.0.1 -p 5432 -U postgres -c 'CREATE DATABASE gravity;'

Configure Model9 Gravity

Copy key stores into the `$GRAVITY_HOME/keys` folder

cp $GRAVITY_EXTRACTED/keys/* $GRAVITY_HOME/keys

Create the Model9 Gravity configuration file

cp $GRAVITY_EXTRACTED/application.properties $GRAVITY_HOME/config

After creating the application.properties file edit it and fill in the missing values:

vi $GRAVITY_HOME/config/application.properties

vi is usually available by default on Linux systems. If more convenient, other editors such as nano can also be used.

# ----------------------------------------
# Model9 Gravity Configuration
# Documentation: https://docs.model9.io
# Support: support@model9.io
# ----------------------------------------
model9.gravity.global.license-key=
logging.level.io.model9=INFO

If the default database password has been changed, use the model9.gravity.datasource.password configuration option to change it.

For details of the configuration options and their default values, see the Configuration page.

Start the Model9 Gravity service

Load the Model9 Gravity docker image

docker load --input $GRAVITY_EXTRACTED/model9-gravity-$GRAVITY_VERSION-build-$GRAVITY_BUILD.docker.gz

Start the Model9 Gravity container

docker run --detach --publish 443:443 \
    --net gravity-bridge \
    --volume $GRAVITY_HOME/config:/data/model9/config:z \
    --volume $GRAVITY_HOME/keys:/data/model9/keys:z \
    --volume $GRAVITY_HOME/logs:/data/model9/logs:z \
    --volume $GRAVITY_HOME/work:/data/model9/work:z \
    --env "TZ=America/New_York" \
    --env "JAVA_OPTS=-Xmx28g" \
    --restart unless-stopped \
    --name model9gravity-v$GRAVITY_VERSION model9/gravity:$GRAVITY_VERSION.$GRAVITY_BUILD

Change the TZ environment variable as appropriate.
Change the -Xmx value to reflect the amount of available memory for the service (in GBs) - Remember to leave a few free GBs for the operating system.
443 is the default secure port, if running the service on another port, this value should be changed as well.

User Guide

Service Parameters

To run the service, specify the source object storage and identify the input data set.

REQUIRED: "source"

Identify the transform source object storage, where the input resides. The source object storage details appear in the Model9 agent configuration file.

Required Keywords for "source"

{
  "source": {
    "url": "<URL>",
    "api": "<API>",
    "bucket": "<USER_BUCKET>",
    "user": "<USERID>",
    "password": "<PASSWORD>"
  }
}

Optional Keywords for "source"

{
  "source": {
    "useS3V4Signatures": "false|true"
  }
}

Keyword

Description

Required

Default

OPTIONAL: "target"

Identify the transform target object storage. Values not specified will be taken from the "source" parameter.

{
  "target": {
    "url": "<URL>",
    "api": "<API>",
    "bucket": "<USER-BUCKET>",
    "user": "<USERID>",
    "password": "<PASSWORD>",
    "useS3V4Signatures": "false|true"
  }
}

Keyword

Description

Required

Default

REQUIRED: "input"

If you specify VSAM keywords for a sequential input data set, the transform will be performed and a warning message will be issued

Required Keywords for "input"

{
  "input": {
    "name": "<DSN>",
    "complex": "<group-SYSPLEX>"
  }
}

Optional Keywords for "input"

{
  "input": {
    "type": "backup|archive|import",
    "entry": "0|<N>",
    "prefix": "model9|<USER-PREFIX>",
    "recordBinary": "false|true",
    "recordCharset": "<CHARSET>",
    "vsam": {
      "keyBinary": "false|true",
      "keyCharset": "<CHARSET>"
    }
  }
}

Description

Default

OPTIONAL: "output"

The output is the transformed data of the MF data set, accessible as S3 object

When transforming a file with the same name as an existing file in the target, the existing file will be replaced by the newly transformed file.
Note that the service does not delete previously transformed files but rather overwrites files with the same name, so when re-transforming a file using the "split" function, ensure to remove any previously transformed files to avoid having split files of different versions.
When splitting a file, wait for the successful completion of the transform function before continuing with the processing, to insure all the parts of a the file were created.
Specifying "text" format for a "binary" input will cause the transform to fail.

{
  "output": {
    "prefix": "model9|<USER-PREFIX>",
    "compression": "none|gzip",
    "format": "JSON|text|CSV|RAW",
    "charset": "UTF8",
    "endWithNewLine": "false|true",
    "splitBySize": "<nnnnb/m/g>",
    "splitByRecords": "<n>"
  }
}

Keyword

Description

Default

Service parameters samples

Transforming a plain text data set

Transform the latest backup of a plain text data set, charset IBM-1047, converted to UTF8 and compressed.

{
  "input": {
    "name": "SAMPLE.TEXT",
    "complex": "group-PLEX1"
  },
  "output": {
    "format": "text"
  },
  "source": {
    "url": "https://s3.amazonaws.com",
    "api": "aws-s3",
    "bucket": "prod-bucket",
    "user": "sdsdDVDCsxadA43TERVGFBSDSSDff",
    "password": "ddferdscsdW4REFEBA33DSffss344gbs4efe7"
  }
}

Transforming an unloaded DB2 table

Transform the latest backup of an unloaded DB2 table, charset IBM-1047, converted to UTF8 and compressed, located with a specific prefix:

{
  "input": {
    "name": "DB2.UNLOADED.SEQ",
    "complex": "group-PLEX1"
  },
  "output": {
    "format": "text"
  },
  "source": {
    "url": "https://s3.amazonaws.com",
    "api": "aws-s3",
    "bucket": "prod-bucket",
    "user": "sdsdDVDCsxadA43TERVGFBSDSSDff",
    "password": "ddferdscsdW4REFEBA33DSffss344gbs4efe7"
  },
  "output": {
    "prefix": "DBprodCustomers"
  }
}

Transforming a VSAM file using the defaults

When transforming a VSAM file, the defaults are a text key and binary data, transforming to a JSON output file:

{
  "input": {
    "name": "SAMPLE.VSAM",
    "complex": "group-PLEX1"
  },
  "source": {
    "url": "https://s3.amazonaws.com",
    "api": "aws-s3",
    "bucket": "prod-bucket",
    "user": "sdsdDVDCsxadA43TERVGFBSDSSDff",
    "password": "ddferdscsdW4REFEBA33DSffss344gbs4efe7"
  }
}

Transforming a VSAM text file to CSV

Specify a text data, transforming to a CSV output file:

{
  "input": {
    "name": "SAMPLE.VSAM",
    "complex": "group-PLEX1"
  },
  "vsam": {
    "keyBinary": "false|true",
    "keyCharset": "<CHARSET>"
  },
  "output": {
    "format": "CSV"
  },
  "source": {
    "url": "https://s3.amazonaws.com",
    "api": "aws-s3",
    "bucket": "prod-bucket",
    "user": "sdsdDVDCsxadA43TERVGFBSDSSDff",
    "password": "ddferdscsdW4REFEBA33DSffss344gbs4efe7"
  }
}

Transforming on Azure Storage using OAuth2

When transforming data on Azure blob storage with OAuth2 set the "api" to azureblob-oauth2 and use the azureOauth section to specify Azure OAuth arguments as follows:

{
  "input": {
    "name": "SAMPLE.PS",
    "complex": "group-PLEX1"
  },
  "vsam": {
    "keyBinary": "false|true",
    "keyCharset": "<CHARSET>"
  },
  "output": {
    "format": "CSV"
  },
  "source": {
    "api": "azureblob-oauth2",
    "url": "https://<azure-storage-account>.blob.core.windows.net",
    "bucket": "<azure-container-name>",
    "user": "<azure-application-uuid>",
    "password": "<azure-application-client-secret>",
    "azureOauth": {
      "oauthEndpoint": "<azure-oauth-endpoint>",
      "storageAccount": "<azure-storage-account>",
      "oauthAudience": "<azure-oauth-audience>",
      "credentialType": "<azure-credential-type>"
    }
  }
}

Table: Azure OAuth2 Arguments

Field Name

Description

Required

Default Value

Service response and log

The transform service is invoked as an HTTP request. It returns:

HTTP status

Code

Description

HTTP response

{
  "status": "OK|WARNING|ERROR",
  "outputName": "<OUTPUT-NAME>",
  "inputName": "<DSN>",
  "outputCompression": "none|gzip",
  "outputSizeInBytes": "<SIZE-IN_BYTES>",
  "outputFormat": "JSON|text|CSV"
}

Output keyword

Description

In case of a WARNING or an ERROR - the HTTP response will also contain log messages.

Log

{
  "log": [
    "<INFO-MESSAGE>",
    "<WARNING-MESSAGE>",
    "<ERROR-MESSAGE>"
  ]
}

Service response and log samples

Status OK sample

{
  "status": "OK",
  "outputName": "transform/QA.SMS.MCBK.SG1QNOBK.DSERV.TXT.TMPPS!uuid=a641d670-2d05-41e7-9dd3-7815e1b2d4c4",
  "inputName": "QA.SMS.MCBK.SG1QNOBK.DSERV.TXT.TMPPS",
  "outputCompression": "NONE",
  "outputSizeInBytes": 97,
  "outputFormat": "JSON"
}

Status WARNING sample

{
  "log": [
    "ZM9K001I Transform service started",
    "ZM9K108W Specifying input parameter vsam is ignored for input data set with DSORG PS",
    "ZM9K002I Transform service completed successfully, output is transform/QA.SMS.MCBK.SG1QNOBK.DSERV.TXT.TMPPS!uuid=d779fbf9-da6b-495b-b6b9-de7583905f19"
  ],
  "status": "WARNING",
  "outputName": "transform/QA.SMS.MCBK.SG1QNOBK.DSERV.TXT.TMPPS!uuid=d779fbf9-da6b-495b-b6b9-de7583905f19",
  "inputName": "QA.SMS.MCBK.SG1QNOBK.DSERV.TXT.TMPPS",
  "outputCompression": "NONE",
  "outputSizeInBytes": 97,
  "outputFormat": "JSON"
}

Status ERROR sample

{
  "status": "ERROR",
  "log": [
    "ZM9K001I Transform service started",
    "ZM9K008E The input was not found: name QA.SMS.MCBK.DSERV.TXT.NON, archive false, entry (0)"
  ]
}

Input format support

Supported formats

SMS-managed data sets
Non-SMS managed data sets
Sequential and extended-sequential data sets with the following RECFM:
- V
- VB
- F
- FB
- FBA
Non-extended VSAM KSDS data sets

Unsupported formats

RRDS, VRRDS, LINEAR, ESDS
Extended format data sets with compression or encryption
PDS data sets
RECFM not mentioned above (U)

Output format support

Text
JSON
CSV

DB2 Image Copy Transform Guide

Configuration

Make sure that <M9_HOME>/scripts/transform-service.sh has execute permissions. If not, add it by using chmod a+x <M9_HOME>/scripts/transform-service.sh.
Copy M9XFDB2 from Model9's SAMPLIB data set to a PDS data set of your choosing.
Edit M9XFDB2 and replace the placeholders enclosed with angle brackets with the following:

Table: Placeholders

Placeholder Name

Replace with ...

Replace the remaining placeholders in the JCL as described in this manual.

Execute and verify results

When done, submit the job and make sure it ends with MAXCC of 0.

Via SDSF, verify that the transform service was in fact called and completed successfully. Successful output would look something like this:

{
  "status": "OK",
  "outputNames": [
    "transform-output/M9.SHY.DB2.IMGCPY.M9DB.M9SEG4"
  ],
  "inputName": "M9.SHY.DB2.IMGCPY.M9DB.M9SEG4",
  "outputCompression": "NONE",
  "outputSizeInBytes": 1064,
  "outputFormat": "CSV"
}

Supported DB2 Column Types

Table 5. Supported DB2 Column Types for Transformation

DB2 SQL Code

Name

v1.1.0

Model9 Gravity

Installation

Installation

Introduction

Prerequisites

Installation

Download Model9 Gravity installation package file

Prepare installation

Set version environment variable

Extract the package file

Create installation target folder structure

Create Model9 Gravity PostgreSQL database container

Create docker network bridge

Load the PostgreSQL docker image

Run docker PostgreSQL container

Create database

Configure Model9 Gravity

Copy key stores into the $GRAVITY_HOME/keys folder

Create the Model9 Gravity configuration file

Start the Model9 Gravity service

Load the Model9 Gravity docker image

Start the Model9 Gravity container

Configuration

Server configuration

Database Configuration

Transformation Configuration

All configuration options

User Guide

Service Limitations

Data size

Maximum elapsed time

Deployment and Initial Usage

Using Gravity

Step 1: Prerequisites

Verify that Model9 Cloud Backup and Recovery for z/OS is installed

Download z/OS cURL

Step 2: Copy the following script

Step 3: Copy the following JCL

Step 4: Customize the JCL

Update the object storage details

Update the complex name

Update the transform prefix

Step 5: Choose the data set to be transformed

Step 6: Run the JCL

Step 7: Access the transformed data

User Guide

Service Parameters

REQUIRED: "source"

Required Keywords for "source"

Optional Keywords for "source"

OPTIONAL: "target"

REQUIRED: "input"

Required Keywords for "input"

Optional Keywords for "input"

OPTIONAL: "output"

Service parameters samples

Transforming a plain text data set

Transforming an unloaded DB2 table

Transforming a VSAM file using the defaults

Transforming a VSAM text file to CSV

Transforming on Azure Storage using OAuth2

Service response and log

HTTP status

HTTP response

Log

Service response and log samples

Status OK sample

Status WARNING sample

Status ERROR sample

Input format support

Supported formats

Unsupported formats

Output format support

DB2 Image Copy Transform Guide

Configuration

Execute and verify results

Supported DB2 Column Types

Service Limitations

Data size

Copy key stores into the `$GRAVITY_HOME/keys` folder

Copy key stores into the `$GRAVITY_HOME/keys` folder