Data size

There is no limit for the input data set size, as long as the output data set size will utilize under 20 GB of storage.

Maximum elapsed time

The service is synchronous and can run for up to an hour.

Server configuration

Database Configuration

Transformation Configuration

All configuration options

This guide will instruct you how to set up and invoke the Model9 Gravity service from the mainframe, using JCL. The service will transform a Model9 data set backup copy / archive / import into a readable file in the cloud. Once transformed, the readable file can be accessed directly or via data analytics tools.

Using Gravity

Step 1: Prerequisites

Verify that Model9 Cloud Backup and Recovery for z/OS is installed

Model9 is responsible for delivering the data set from the mainframe to the cloud / on-premises storage. The data set is delivered as a backup copy, an archive or imported tape data set, and provides the input to the transform service.

Download z/OS cURL

This free tool will allow you to invoke the transform service from z/OS. If cURL is not installed under /usr/bin, edit line 4 and add the path where the cURL module resides.

Step 2: Copy the following script

Copy the following script to /usr/lpp/model9/gravity-run.sh:

#!/bin/sh
json=$1
url='https://<gravity-url>:<port>/transform'
export PATH=/usr/bin:/bin
export _EDC_ADD_ERRNO2=1
cnvto="$(locale codeset)"
headers="Content-Type:application/json"
echo "Running Model9 Gravity"
output=$(curl -H "$headers" -s -k -X POST --data "$json" $url)
if ! [ -z "$output" ]; then
   echo "Transformation ended with the following output:"
   # If the answer is in ASCII then convert to EBCDIC
   firstChar=$(echo $output | cut -c1)
   if [ "$firstChar" = "#" ]; then
      convOutput="$(echo $output | iconv -f ISO8859-1 -t $cnvto)"
   else
      convOutput=$output
   fi
   echo "$convOutput"
fi
status=$(echo $convOutput | tr -s " " | cut -d, -f1 | cut -d" " -f3)
   echo "Transformation ended with status: $status"
if [ "$status" = '"OK"' ];then
   exit 0
else if [ "$status" = '"WARNING"' ]; then
   exit 4
else
   exit 8
fi
fi

Step 3: Copy the following JCL

Copy the following JCL to a local library, update the JOBCARD according to your site standards:

//M9TRNSFM JOB 'ACCT#',REGION=0M,CLASS=A,NOTIFY=&SYSUID
//EXTRACT  EXEC PGM=BPXBATCH
//STDOUT   DD SYSOUT=*
//STDERR   DD SYSOUT=*
//STDPARM  DD *
SH /usr/lpp/model9/gravity-run.sh
//         DD *,SYMBOLS=EXECSYS
'{
    "input": {
        "name"   : "<DATA-SET>",
        "complex": "group-&SYSPLEX",
        "type": "<BACKUP|ARCHIVE|IMPORT>"
    },
    "output": {
        "prefix"      : "/transform/&LYR4/&LMON/&LDAY",
        "compression" : "none",
        "format"      : "text"
    },
    "source": {
        "url"     : "<URL>",
        "api"     : "<API>",
        "bucket"  : "<BUCKET>",
        "user"    : "<USER>",
        "password": "<PASSWORD>"
    }
}'
/*
//

Step 4: Customize the JCL

Update the object storage details

Copy the following object storage variables from the Model9 agent configuration file:

• <URL>

• <API>

• <BUCKET>

• <USER>

• <PASSWORD>

Update the complex name

The "complex” name represents the group of resources that the Model9 agent can access. By default, this group is named group-<SYSPLEX> and it is shared by all the agents in the same sysplex. The transform JCL specifies the default, using the z/OS system symbol &SYSPLEX.

If the "complex” name was changed in the Model9 agent configuration file, change the "complex” in the JCL accordingly.

Update the transform prefix

By default, the JCL will create a transformed copy of your input data set, in the same bucket, with the prefix: /transform/&LYR4/&LMON/&LDAY. The prefix is using the following z/OS system symbols:

• &LYR4 - The year in 4 digits, e.g. 2019

• &LMON - The month in 2 digits, e.g. 08

• &LDAY - The day in the month in 2 digits, e.g. 10 You can change the prefix according to your needs.

Step 5: Choose the data set to be transformed

The data set to be transformed should be a backup copy, an archive or imported tape data set, delivered by the Model9 agent:

• <DATA-SET> - the name of the data set

• <BACKUP|ARCHIVE|IMPORT> - if the data set is a Model9 backup, archive or import

To change the attributes of the input and the output, and for a full description of the service parameters, see Service parameters.

Step 6: Run the JCL

Submit the job and view the output. See Service response and log samples for sample output.

Step 7: Access the transformed data

Based on the returned response, the outputName will point to the path inside the bucket where the transformed data resides. See Service response and log samples.

Instructions for installing Model9 Gravity in an on-premise environment.

Introduction

The following guides provides details on how to install and configure the Model9 Gravity application.

Prerequisites

• 8 vCPU, 32gb RAM instance, 1TB of storage (preferably SSD).

  • AWS instance types:

    • m5d.2xlarge

    • m5ad.2xlarge

  • docker is expected to be installed on the instance.

    • A user with docker privileges will be required for running docker commands.

Installation

Download Model9 Gravity installation package file

Obtain the Model9 Gravity installation package from Model9 and move it to the target installation system.

The recommended location for the package files on the installation system is:

Prepare installation

Set version environment variable

Extract the package file

Create installation target folder structure

Create Model9 Gravity PostgreSQL database container

Create docker network bridge

Load the PostgreSQL docker image

Run docker PostgreSQL container

Create database

Configure Model9 Gravity

Copy key stores into the $GRAVITY_HOME/keys folder

Create the Model9 Gravity configuration file

After creating the application.properties file edit it and fill in the missing values:

Start the Model9 Gravity service

Load the Model9 Gravity docker image

Start the Model9 Gravity container

Service Parameters

To run the service, specify the source object storage and identify the input data set.

REQUIRED: "source"

Required Keywords for "source"

{
  "source": {
    "url": "<URL>",
    "api": "<API>",
    "bucket": "<USER_BUCKET>",
    "user": "<USERID>",
    "password": "<PASSWORD>"
  }
}

Optional Keywords for "source"

{
  "source": {
    "useS3V4Signatures": "false|true"
  }
}

OPTIONAL: "target"

{
  "target": {
    "url": "<URL>",
    "api": "<API>",
    "bucket": "<USER-BUCKET>",
    "user": "<USERID>",
    "password": "<PASSWORD>",
    "useS3V4Signatures": "false|true"
  }
}

REQUIRED: "input"

Required Keywords for "input"

{
  "input": {
    "name": "<DSN>",
    "complex": "<group-SYSPLEX>"
  }
}

Optional Keywords for "input"

{
  "input": {
    "type": "backup|archive|import",
    "entry": "0|<N>",
    "prefix": "model9|<USER-PREFIX>",
    "recordBinary": "false|true",
    "recordCharset": "<CHARSET>",
    "vsam": {
      "keyBinary": "false|true",
      "keyCharset": "<CHARSET>"
    }
  }
}

OPTIONAL: "output"

{
  "output": {
    "prefix": "model9|<USER-PREFIX>",
    "compression": "none|gzip",
    "format": "JSON|text|CSV|RAW",
    "charset": "UTF8",
    "endWithNewLine": "false|true",
    "splitBySize": "<nnnnb/m/g>",
    "splitByRecords": "<n>"
  }
} 

Service parameters samples

Transforming a plain text data set

Transform the latest backup of a plain text data set, charset IBM-1047, converted to UTF8 and compressed.

{
  "input": {
    "name": "SAMPLE.TEXT",
    "complex": "group-PLEX1"
  },
  "output": {
    "format": "text"
  },
  "source": {
    "url": "https://s3.amazonaws.com",
    "api": "aws-s3",
    "bucket": "prod-bucket",
    "user": "sdsdDVDCsxadA43TERVGFBSDSSDff",
    "password": "ddferdscsdW4REFEBA33DSffss344gbs4efe7"
  }
}

Transforming an unloaded DB2 table

Transform the latest backup of an unloaded DB2 table, charset IBM-1047, converted to UTF8 and compressed, located with a specific prefix:

{
  "input": {
    "name": "DB2.UNLOADED.SEQ",
    "complex": "group-PLEX1"
  },
  "output": {
    "format": "text"
  },
  "source": {
    "url": "https://s3.amazonaws.com",
    "api": "aws-s3",
    "bucket": "prod-bucket",
    "user": "sdsdDVDCsxadA43TERVGFBSDSSDff",
    "password": "ddferdscsdW4REFEBA33DSffss344gbs4efe7"
  },
  "output": {
    "prefix": "DBprodCustomers"
  }
}

Transforming a VSAM file using the defaults

When transforming a VSAM file, the defaults are a text key and binary data, transforming to a JSON output file:

{
  "input": {
    "name": "SAMPLE.VSAM",
    "complex": "group-PLEX1"
  },
  "source": {
    "url": "https://s3.amazonaws.com",
    "api": "aws-s3",
    "bucket": "prod-bucket",
    "user": "sdsdDVDCsxadA43TERVGFBSDSSDff",
    "password": "ddferdscsdW4REFEBA33DSffss344gbs4efe7"
  }
}

Transforming a VSAM text file to CSV

Specify a text data, transforming to a CSV output file:

{
  "input": {
    "name": "SAMPLE.VSAM",
    "complex": "group-PLEX1"
  },
  "vsam": {
    "keyBinary": "false|true",
    "keyCharset": "<CHARSET>"
  },
  "output": {
    "format": "CSV"
  },
  "source": {
    "url": "https://s3.amazonaws.com",
    "api": "aws-s3",
    "bucket": "prod-bucket",
    "user": "sdsdDVDCsxadA43TERVGFBSDSSDff",
    "password": "ddferdscsdW4REFEBA33DSffss344gbs4efe7"
  }
}

Transforming on Azure Storage using OAuth2

When transforming data on Azure blob storage with OAuth2 set the "api" to azureblob-oauth2 and use the azureOauth section to specify Azure OAuth arguments as follows:

{
  "input": {
    "name": "SAMPLE.PS",
    "complex": "group-PLEX1"
  },
  "vsam": {
    "keyBinary": "false|true",
    "keyCharset": "<CHARSET>"
  },
  "output": {
    "format": "CSV"
  },
  "source": {
    "api": "azureblob-oauth2",
    "url": "https://<azure-storage-account>.blob.core.windows.net",
    "bucket": "<azure-container-name>",
    "user": "<azure-application-uuid>",
    "password": "<azure-application-client-secret>",
    "azureOauth": {
      "oauthEndpoint": "<azure-oauth-endpoint>",
      "storageAccount": "<azure-storage-account>",
      "oauthAudience": "<azure-oauth-audience>",
      "credentialType": "<azure-credential-type>"
    }
  }
}

Table: Azure OAuth2 Arguments

Service response and log

The transform service is invoked as an HTTP request. It returns:

HTTP status

HTTP response

{
  "status": "OK|WARNING|ERROR",
  "outputName": "<OUTPUT-NAME>",
  "inputName": "<DSN>",
  "outputCompression": "none|gzip",
  "outputSizeInBytes": "<SIZE-IN_BYTES>",
  "outputFormat": "JSON|text|CSV"
}

In case of a WARNING or an ERROR - the HTTP response will also contain log messages.

Log

{
  "log": [
    "<INFO-MESSAGE>",
    "<WARNING-MESSAGE>",
    "<ERROR-MESSAGE>"
  ]
} 

Service response and log samples

Status OK sample

{
  "status": "OK",
  "outputName": "transform/QA.SMS.MCBK.SG1QNOBK.DSERV.TXT.TMPPS!uuid=a641d670-2d05-41e7-9dd3-7815e1b2d4c4",
  "inputName": "QA.SMS.MCBK.SG1QNOBK.DSERV.TXT.TMPPS",
  "outputCompression": "NONE",
  "outputSizeInBytes": 97,
  "outputFormat": "JSON"
}

Status WARNING sample

{
  "log": [
    "ZM9K001I Transform service started",
    "ZM9K108W Specifying input parameter vsam is ignored for input data set with DSORG PS",
    "ZM9K002I Transform service completed successfully, output is transform/QA.SMS.MCBK.SG1QNOBK.DSERV.TXT.TMPPS!uuid=d779fbf9-da6b-495b-b6b9-de7583905f19"
  ],
  "status": "WARNING",
  "outputName": "transform/QA.SMS.MCBK.SG1QNOBK.DSERV.TXT.TMPPS!uuid=d779fbf9-da6b-495b-b6b9-de7583905f19",
  "inputName": "QA.SMS.MCBK.SG1QNOBK.DSERV.TXT.TMPPS",
  "outputCompression": "NONE",
  "outputSizeInBytes": 97,
  "outputFormat": "JSON"
}

Status ERROR sample

{
  "status": "ERROR",
  "log": [
    "ZM9K001I Transform service started",
    "ZM9K008E The input was not found: name QA.SMS.MCBK.DSERV.TXT.NON, archive false, entry (0)"
  ]
}

Input format support

Supported formats

• SMS-managed data sets

• Non-SMS managed data sets

• Sequential and extended-sequential data sets with the following RECFM:

  • V

  • VB

  • F

  • FB

  • FBA

• Non-extended VSAM KSDS data sets

Unsupported formats

• RRDS, VRRDS, LINEAR, ESDS

• Extended format data sets with compression or encryption

• PDS data sets

• RECFM not mentioned above (U)

Output format support

• Text

• JSON

• CSV

DB2 Image Copy Transform Guide

Configuration

1. Make sure that <M9_HOME>/scripts/transform-service.sh has execute permissions. If not, add it by using chmod a+x <M9_HOME>/scripts/transform-service.sh.

2. Copy M9XFDB2 from Model9's SAMPLIB data set to a PDS data set of your choosing.

3. Edit M9XFDB2 and replace the placeholders enclosed with angle brackets with the following:

Table: Placeholders

1. Replace the remaining placeholders in the JCL as described in this manual.

Execute and verify results

When done, submit the job and make sure it ends with MAXCC of 0.

Via SDSF, verify that the transform service was in fact called and completed successfully. Successful output would look something like this:

{
  "status": "OK",
  "outputNames": [
    "transform-output/M9.SHY.DB2.IMGCPY.M9DB.M9SEG4"
  ],
  "inputName": "M9.SHY.DB2.IMGCPY.M9DB.M9SEG4",
  "outputCompression": "NONE",
  "outputSizeInBytes": 1064,
  "outputFormat": "CSV"
}

Supported DB2 Column Types

Table 5. Supported DB2 Column Types for Transformation

Monetize unlocked mainframe data in business intelligence (BI), analytics and cloud applications

Leverage any disk data or historical tape data for use by analytics services and BI tools. Mainframe data migrated to object storage on-premises or in the cloud can be transformed to standard data formats without requiring any access to the mainframe, instantly providing it for use in cloud applications and analytics tools.

For any questions, contact us at support@model9.io

Model9 website: www.model9.io