I’ve been developing a Java interface to GIMAPI and writing some sample reports. In the process I’ve been doing a deep dive into the some of the information in the SMP/E CSI.

HFS element delivery​

Delivery of HFS elements is one of the more confusing parts of SMP/E. So I took a look into how they are delivered and how the various parts fit together.

Let’s look at an example:

ENTRYNAME : IOEZH002
ENTRYTYPE : HFS
  ZONENAME : MVST
  DISTLIB : AIOEHLIB
  FMID : HZFS510
  HFSPARM : PATHMODE(0,7,5,5)
  INSTMODE : BINARY
  LASTUPD : HZFS510
  LASTUPDTYPE : ADD
  LINK : ../zfsadm
  RMID : HZFS510
  SYMLINK : ../../../../../../bin/zfsadm
  SYMPATH : ../usr/lpp/dfs/global/bin/zfsadm
  SYSLIB : SIOEHLMD

Where does this entry end up, and what’s the deal with all the ../../..?

Firstly, all paths are relative to the DDDEF specified by SYSLIB. If we look at the SIOEHLMD DDDEF:

ENTRYTYPE : DDDEF
  ENTRYNAME : SIOEHLMD
  ENTRYTYPE : DDDEF
    ZONENAME : MVST
    PATH : /sysc/usr/lpp/dfs/global/bin/IBM/
    PROTECT : NO
    WAITFORDSN : NO

The DDDEF should point to a location in your service HFS environment so you’re not updating live code. In this case the service filesystem lives under /sysc.

What we end up with:

The actual element installed by SMP/E is

/sysc/usr/lpp/dfs/global/bin/IBM/IOEZH002

SMP/E creates a hard link (another directory entry for the same inode in the filesystem):

/sysc/usr/lpp/dfs/global/bin/IBM/../zfsadm

SMP/E creates a symbolic link SYMLINK at:

/sysc/usr/lpp/dfs/global/bin/IBM/../../../../../../bin/zfsadm

Each ../ walks up one level in the directory tree. So ../../../../../.. goes up 6 levels from /sysc/usr/lpp/dfs/global/bin/IBM/ and results in /sysc/bin/zfsadm.

The symbolic link target SYMPATH is relative to the symbolic link, so it points at:

/sysc/usr/lpp/dfs/global/bin/IBM/../../../../../../bin/../usr/lpp/dfs/global/bin/zfsadm

I think I have that correct… did I mention it was confusing? It’s walking up and down the directory tree.

It’s too hard to figure out manually, so I wrote a program to resolve (using Path.normalize) and report the paths. We end up with:

Entry: IOEZH002   SYSLIB: SIOEHLMD   DDDEF PATH: /sysc/usr/lpp/dfs/global/bin/IBM/
     PATH: /sysc/usr/lpp/dfs/global/bin/IBM/IOEZH002
     LINK: /sysc/usr/lpp/dfs/global/bin/zfsadm
  SYMLINK: /sysc/bin/zfsadm
        -> ../usr/lpp/dfs/global/bin/zfsadm
        -> /sysc/usr/lpp/dfs/global/bin/zfsadm

The element installed by SMP/E:

/sysc/usr/lpp/dfs/global/bin/IBM/IOEZH002

The hard link, which is the commonly used name:

/sysc/usr/lpp/dfs/global/bin/zfsadm

The symbolic link which creates a command in the search path:

/sysc/bin/zfsadm -> ../usr/lpp/dfs/global/bin/zfsadm

which resolves to:

/sysc/usr/lpp/dfs/global/bin/zfsadm

which is the hard link to the SMP/E element. So everything fits together neatly.

To confirm, I looked at the live system (removing the service filesystem prefix /sysc):

$ ls -li /usr/lpp/dfs/global/bin/IBM/IOEZH002
22743 -rwxr-xr-x 2 OMVSKERN OMVSGRP 353 Jun 2 2023 /usr/lpp/dfs/global/bin/IBM/IOEZH002

$ ls -li /usr/lpp/dfs/global/bin/zfsadm
22743 -rwxr-xr-x 2 OMVSKERN OMVSGRP 353 Jun 2 2023 /usr/lpp/dfs/global/bin/zfsadm

$ ls -l /bin/zfsadm
lrwxrwxrwx 1 OMVSKERN OMVSGRP 32 Jun 12 2023 /bin/zfsadm -> ../usr/lpp/dfs/global/bin/zfsadm

ls -li shows the inode, and we can see that /usr/lpp/dfs/global/bin/IBM/IOEZH002 and /usr/lpp/dfs/global/bin/zfsadm are entries pointing to the same entry in the filesystem (inode 22743).

/bin/zfsadm is a symbolic link to /usr/lpp/dfs/global/bin/zfsadm.

See the samples or try it yourself​

If you want to see other sample SMP/E queries, they are on Github at: https://github.com/BlackHillSoftware/gimapi-samples/tree/main

More information about the Java GIMAPI interface is available from: https://www.blackhillsoftware.com/easygimapi/

I was recently asked whether Java performance on a PC was adequate for processing SMF data. More generally, people have been sceptical about Java performance. So, I thought why not borrow the idea from the Java 1 Billion Row Challenge, and write a Java program to see how fast you can produce a report on 1 billion CICS transactions using the EasySMF Java API?

The test is to read CICS monitoring facility SMF data and produce a report showing:

• transaction counts,
• maximum, minimum and average elapsed times
• average and total USRCPUT

grouped by APPLID and transaction ID, for 1 billion CICS transactions.

• The CICS SMF records are compressed using CICS compression.
• The SMF records are not sorted or pre-processed for the report.

Watch the video, or read more detail below…

Input Data​

The first problem was I didn’t have enough sample data for 1 billion transactions, so I wrote a small program to build a file containing 1 billion transactions. It reads an input file and writes the data to a new file repeatedly, until the new file contains records for the required number of transactions.

That means the input contains repeated data, but that shouldn’t influence the run time as long as the initial file is reasonably large.

The resulting file with 1 billion transactions was 586 GB.

The program to generate the data is available here: GenerateInput.java

PC Hardware​

The PC used to run the test was a Windows PC, with an i5-14400 processor and 1TB Crucial P310 M.2 SSD. This is a reasonable midrange PC, nothing extreme or expensive.

The Program​

Entries in the original 1 billion row challenge used some pretty advanced techniques to get the winning time down to around 1.5 seconds.

However, with this program I’m trying to keep it simple and use standard techniques that should be familiar to any Java programmer, and at least readable to people familiar with similarly structured languages like C/C++.

The program uses a class to accumulate data about each applid/transaction combination. The entries are stored in a HashMap with a Java record containing the applid and transaction name as the key. This provides very fast access as the data is processed.

The CICS records require a dictionary to access locate the fields in the SMF record. A file containing the CICS dictionary SMF records is read before the main data file.

View the program source on Github (only 134 lines):

Cics1B.java

Results​

The program produced the report from 1 billion CICS transactions in 20 minutes 17 seconds. Some statistics from the run:

• Read SMF data: 492 MB/s
• Decompress the compressed portion of the record: 420 MB/s
• Decompressed data size: 3.1 TB
• Process decompressed data: 2.6 GB/s

Most of the CPU time was spent doing the CICS SMF data decompression.

How to make it faster?​

On a PC, the simple answer is multithreading. This PC has 10 cores (6 performance cores, 4 efficiency cores) and can run 16 threads simultaneously. The CICS data decompression in particular will benefit from multiple threads.

The program can be converted to use Java Streams, specifying parallel() to divide the work across multiple threads.

Note that this might not be a good idea on z/OS. Java by default will attempt to create enough threads to use all the processors on the system – which is probably not desirable on the mainframe. On the other hand, it could work OK in a discretionary service class and will probably be limited by I/O speed anyway so YMMV.

Here’s the parallel version of the program on Github:

Cics1BParallel.java

Java Streams provides methods to do the accumulation independently and merge the results from each thread at the end. That avoids the problem of synchronizing updates between threads.

However, for this test, I elected to minimize the changes from the original version and just moved the processing to a forEach block. That means that any updates to shared data in the forEach block need to be synchronized between threads. Change required were:

• Transaction counts were changed to use the LongAdder class.
• The HashMap was changed to a ConcurrentHashMap
• The add method in the TransactionData class was made synchronized.

In theory, accumulating independently and merging at the end should be faster because it avoids having to synchronize threads. However, most of the time here is spent decompressing data so any gain is probably insignificant.

Results​

The parallel run finished in 7 minutes 18 seconds. Statistics:

• Read SMF data: 1.3 GB/s
• Decompress the compressed portion of the record: 1.1 GB/s
• Process decompressed data: 7.2 GB/s

Reporting on 1 billion transactions in 7 minutes 18 seconds seems pretty good. That’s over 2 million transactions per second. The times are pretty linear with transaction count, so e.g. if you have only 100 million transactions it should run in less than a minute.

Even faster?​

According to the Windows performance monitor, the parallel run didn’t max out the disk or the CPU. In theory the SSD is capable of more than 1.3 GB/s so there might be more performance to be found.

At some point the I/O is being processed sequentially, so I suspect that is the bottleneck. Perhaps it would be faster if there were multiple files e.g. if the SMF data was divided by CICS region or system and I/O could be processed to multiple files in parallel.

A subject for future investigation, perhaps.

Try it on your data​

Contact us at info@blackhillsoftware.com for a free trial. Let us know what reports you would like and we can add them to the samples.

Notes​

• This runs fast enough that I/O is potentially the bottleneck. On z/OS, I have found that zEDC compression for SMF data can significantly improve I/O speed. Using zEDC and CICS SMF record compression might be better than zEDC alone. See some tests here.
• This is 100% Java so it is zIIP eligible on z/OS. This is a particular advantage if the general purpose CPs run at less than full speed.
• For CICS records, most of the time is spent decompressing the records. It is reasonably easy to modify the program to produce multiple different reports from the same pass of the data, which means that additional reports cost almost zero CPU or elapsed time.
• EasySMF can read data using the z/OSMF Dataset and Files REST API. You can run a Java program on the PC and access the mainframe SMF data using a https connection.
• AI is pretty good at writing Java, and Java type safety in the EasySMF API helps keep it on track with the SMF data so it’s very feasible to use AI to help write SMF reports.

Should you use zEDC to compress SMF data? Should you use both zEDC and CICS software compression?

There was a recent discussion on the IBM-MAIN email list about exploiting zEDC. One of the topics was SMF data, including whether to turn off the software compression for CICS SMF records.

I ran some experiments to compare the processing of SMF records with and without the various compression options. The tests were run using a dataset containing 6.6 GB of CICS SMF data compressed with CICS software compression. For the tests without software compression, the data was uncompressed and copied to another dataset compressed with zEDC. The uncompressed data size was about 39 GB.

Disclaimer: As always, your mileage may vary. This isn’t intended as a benchmark – it’s just one set of tests run against one dataset. This is a very lightly loaded system running under VM. It might bear no resemblance to a busy production system. CPU times etc. might not be accurately reported.

Data size​

The data is very compressible. The software compressed data compresses further using zEDC and the final size is smaller than compression by zEDC alone.

Reading data​

To test the overhead of zEDC when reading SMF data, I used the IFASMFDP program with the OUTDD pointing to DUMMY. This reads the data and produces a report on the numbers of each SMF record type, showing that individual records were read.

This program doesn’t do anything with the CICS data so there is no software decompression. It’s just a indication of the cost of reading SMF records with and without zEDC compression.

I was surprised how slow it was to read the records without zEDC. Is this a result of e.g. running under VM, or is it typical? zEDC was very fast, relative to the uncompressed data – better than 10x speed-up.

Copying Data​

I ran the IFASMFDP jobs but this time with OUTDD specifying a real dataset. The difference between the read jobs and the copy jobs should give an indication of the cost of writing the data using zEDC.

Again zEDC was much faster but unsurprisingly there was a CPU cost (less than 0.2s/GB).

SMF Reporting using Java​

I ran the CICS Transaction Summary report from the EasySMF Java samples:

https://github.com/BlackHillSoftware/easysmf-samples/blob/main/sample-reports/src/main/java/com/smfreports/cics/CicsTransactionSummary.java

This report produces a basic summary of CICS transactions grouped by APPLID and transaction name.

Software decompression is performed in Java by EasySMF as required. The program is processing 39GB of data after decompression.

This gives some idea of the overhead of compression on reporting, however this is very dependent on the reporting software you use and your mileage will definitely vary.

One interesting thing here is that zEDC with Java seems to allow more work to move from the CP to the zIIP. The time on CP reduces from 1.49s to 1.25s processing the 6.6GB input. The CP time processing the 39GB compressed with zEDC is only 2 seconds. Reading the same data with IFASMFDP took 5.6 seconds of CPU time. Presumably under Java the rest is included in the zIIP time.

CICS software compression increased elapsed and zIIP times by about 10-15% compared to zEDC compression only, but reduced CP time.

Conclusions​

Based on these tests, zEDC works very well with SMF data.

• Elapsed times were greatly reduced in all cases
• The CPU time required was fractions of a second per GB
• CICS software compression appears to be still worthwhile. Reducing the amount of data being processed by zEDC significantly reduces the elapsed and CPU times for SMF copy operations e.g. SMF dump, weekly/monthly processing.

Don’t forget to measure and verify the results on your own systems!

In this post, I’ll show how you can send SMS text messages from z/OS for failed jobs using Twilio and the z/OS SMF Real Time Interface.

Twilio provides a Java API to use their service, and that can be combined with the EasySMF Real Time Interface to send SMS messages based on real time SMF data.

Sending text messages from z/OS​

Twilio is a paid service for production messaging, but you can sign up for a free trial account and send a limited number of messages to a verified phone number.

Twilio provides quickstart documentation here:

https://www.twilio.com/docs/messaging/quickstart/java

You can ignore the stuff about the CLI, all we are going to do is send an outbound message which requires a Twilio account, the Java program and dependencies i.e. the fat jar.

This is the Twilio quickstart Java program with a couple of minor tweaks:

import com.twilio.Twilio;
import com.twilio.rest.api.v2010.account.Message;
import com.twilio.type.PhoneNumber;

public class TwilioTest {
    // Find your Account SID and Auth Token at twilio.com/console
    // and set the environment variables. See http://twil.io/secure
    public static final String ACCOUNT_SID = System.getenv("TWILIO_ACCOUNT_SID");
    public static final String AUTH_TOKEN = System.getenv("TWILIO_AUTH_TOKEN");

    public static void main(String[] args) {
        Twilio.init(ACCOUNT_SID, AUTH_TOKEN);
        Message message = Message.creator(
                new com.twilio.type.PhoneNumber("+14159352345"), // to
                new com.twilio.type.PhoneNumber("+14158141829"), // from
                args[0])
            .create();

        System.out.println(message.getSid());
    }
}

Upload the Twilio fat jar twilio-x.x-jar-with-dependencies.jar to z/OS. It is available from e.g. https://repo1.maven.org/maven2/com/twilio/sdk/twilio/10.1.3/

Java 11 will run single file Java programs without a separate compilation step, so we can just run the program under BPXBATCH:

//ANDREWRG JOB CLASS=A,
//             MSGCLASS=H,
//             NOTIFY=&SYSUID
//*
//BPXBATCH EXEC PGM=BPXBATCH,REGION=512M
//STDPARM  DD *
sh /usr/lpp/java/J11.0_64/bin/java
 /home/andrewr/java/src/TwilioTest.java
 "Hello from z/OS"
//STDENV   DD *
CLASSPATH=/home/andrewr/java/lib/twilio-10.1.3-jar-with-dependencies.jar
TWILIO_ACCOUNT_SID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_AUTH_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
//SYSOUT   DD SYSOUT=*
//STDOUT   DD SYSOUT=*
//STDERR   DD SYSOUT=*

Twilio can also send the messages to WhatsApp using the same functionality.

Sending notifications for failed jobs​

Prerequisite: z/OS SMF Real Time Interface

The z/OS SMF Real Time Interface must be active. This requires:

• SMF running in logstream mode
• Define an in-memory resource to receive the required records (type 30 in this case)
• Setup RACF definitions to allow the user running Java program to access the SMF in memory resource

The program​

The complete source code for this program can be found here:

https://github.com/BlackHillSoftware/easysmf-samples/tree/main/easysmf-rti/rti-notifications

The following is an overview of the major sections of the program.

Reading the in-memory resource​

The first thing to do is set up the connection to the in-memory resource.

public class RtiNotifications
{
    public static void main(String[] args) throws IOException
    {
        try (SmfConnection connection = 
                 SmfConnection.forResourceName("IFASMF.MYRECS")
                     .onMissedData(RtiNotifications::handleMissedData)
                     .disconnectOnStop()
                     .connect();

             // Set up SmfrecordReader to read type 30 subtypes 4 and 5    
             SmfRecordReader reader = 
                 SmfRecordReader.fromByteArrays(connection)
                     .include(30,4)
                     .include(30,5))
        {
            // process data here
        }
    }

    private static void handleMissedData(MissedDataEvent e)
    {
        System.out.println("Missed Data!");
        e.throwException(false);    
    }
}

This code:

• creates a real time connection to resource name IFASMF.MYRECS
• sets up an action to be taken if data is written faster than the program can read it and the real time buffer wraps (write a message and suppress the exception)
• sets up a command handler to disconnect from the resource when the STOP command is received
• creates a SmfRecordReader to read SMF 30 subtypes 4 and 5 (step and job end) records

The connection and reader will be closed automatically when the program exits the try block.

The connection can be simulated in a development environment by setting environment variables:

SIMULATE_RTI=Y

and

IFASMF_MYRECS=/your/smf/filename

The SmfConnection will read SMF data from the file indicated by the environment variable matching the resource name.

Processing the SMF 30 Records​

We want to send the SMS message when the job ends, which is indicated by a subtype 5 record. However, if steps are skipped due to e.g. COND processing or an ABEND, the information in the subtype 5 record might be from a step that didn’t run. These steps have the “flushed” indicator set in the Completion Section.

We need to keep the step end information for steps that did run and check the last executed step when we see the job end record. The Java HashMap provides a convenient way to store information for failed steps.

We can create a class to use as a HashMap key to identify specific jobs. The class has fields for the identifying information (system, jobname, job number, read date and time) which are populated in the constructor. Eclipse can generate the hashCode and equals methods which are required for a HashMap key.

Map<JobKey, Smf30Record> failedSteps = new HashMap<>();

...

private static class JobKey
{
    String system;
    String jobname;
    String jobnumber;
    long readtime;
    int readdate;

    JobKey(Smf30Record r30)
    {
        system = r30.system();
        jobname = r30.identificationSection().smf30jbn();
        jobnumber = r30.identificationSection().smf30jnm();
        readtime = r30.identificationSection().smf30rstRawValue();
        readdate = r30.identificationSection().smf30rsdRawValue();
    }

    // hashCode and equals generated using Eclipse
    @Override
    public int hashCode() {
        return Objects.hash(jobname, jobnumber, readdate, readtime, system);
    }

    @Override
    public boolean equals(Object obj) {
        if (this == obj)
            return true;
        if (obj == null)
            return false;
        if (getClass() != obj.getClass())
            return false;
        JobKey other = (JobKey) obj;
        return Objects.equals(jobname, other.jobname) 
                && Objects.equals(jobnumber, other.jobnumber)
                && readdate == other.readdate 
                && readtime == other.readtime 
                && Objects.equals(system, other.system);
    }
}

SMF 30 Subtype 4: Step End​

When processing step end records, if the step failed we save the record for later. If it ran successfully, we remove any previous failed step.

if (failed(r30)) 
{  
    // this replaces existing entry if present 
    failedSteps.put(new JobKey(r30), r30);
}
// else it didn't fail, check it wasn't flushed
else if (!r30.completionSection().smf30flh()) 
{
    // If the step wasn't flushed, remove any earlier failed step
    failedSteps.remove(new JobKey(r30));
}

failed is a method which checks whether the step failed:

private static boolean failed(Smf30Record r30) 
{
    return 
            // abended except for S222 (cancelled)
            (r30.completionSection().smf30abd() 
                    && r30.completionSection().smf30scc() != 0x222)
            // or condition code > 8
            || r30.completionSection().smf30scc() > 8
            // or post execution error
            || r30.completionSection().smf30sye();
}

The criteria can be adjusted as required.

SMF 30 Subtype 5: Job End​

When the job ends, we check whether we previously saved a failed step.

JobKey key = new JobKey(r30);                        
if (failedSteps.containsKey(key))
{
    // send notification using information from failed step 
    sendNotification(failedSteps.get(key));                            
    failedSteps.remove(key); // finished with this
}
// or if the type 5 record indicates failure
else if(failed(r30))
{
    // send notification using information from job
    sendNotification(r30);                            
}

Sending the SMS message​

The job information is extracted from the SMF record, the Twilio information is supplied in environment variables, and sending the SMS message uses the same process as the Twilio quickstart.

private static final String ACCOUNT_SID = System.getenv("TWILIO_ACCOUNT_SID");
private static final String AUTH_TOKEN = System.getenv("TWILIO_AUTH_TOKEN");
private static final String TO_PHONE = System.getenv("TO_PHONE");
private static final String FROM_PHONE = System.getenv("FROM_PHONE");

private static void sendNotification(Smf30Record r30) 
{
    String messagetext = 
            String.format("%s Job failed: %s %s Step: %d %s Program: %s CC: %s",
                    r30.smfDateTime().toString(),
                    r30.identificationSection().smf30jbn(), // job name
                    r30.identificationSection().smf30jnm(), // job number
                    r30.identificationSection().smf30stn(), // step number
                    r30.identificationSection().smf30stm(), // step name
                    r30.identificationSection().smf30pgm(), // program
                    r30.completionSection().completionDescription());
    
    // Send a SMS notification through Twilio
    Twilio.init(ACCOUNT_SID, AUTH_TOKEN);
    Message message = Message.creator(
            new com.twilio.type.PhoneNumber(TO_PHONE),
            new com.twilio.type.PhoneNumber(FROM_PHONE),
            messagetext)
            .create();
    System.out.println("Message Sent: " + message.getSid());
}

What Next?​

Browse and build the EasySMF:RTI sample code:

https://github.com/BlackHillSoftware/easysmf-samples/tree/main/easysmf-rti/rti-notifications

The Github sample has some additional functionality not covered here. You can limit the notifications to jobs running in specific job classes. You can use the same functionality to include jobs by job name or create custom failure criteria for specific jobs.

For simplicity, the sample doesn’t include error handling. If sending the message fails, the program will disconnect from the SMF in memory resource and end with an exception. A production version should probably catch certain errors and retry.

EasySMF:JE version 2.2.1 added support for DCOLLECT records.

DCOLLECT reports using Java are fast and powerful. Programs can run on z/OS on zIIP processors, or DCOLLECT data can be downloaded to another platform for reporting.

Sample reports are provided for several different scenarios:

• Space by SMS storage group and volume
• Space by high level qualifier, on primary volumes and HSM ML1 and ML2
• Space by time since last reference (7 days, 1 month, 6 months, 1 year, 5 years)
• Aged datasets – datasets on primary volumes with a last reference data older than a cutoff e.g. 5 years
• Datasets by migration date (7 days, 1 month, 6 months, 1 year, 5 years)
• Frequently migrated datasets – datasets that are being migrated and recalled frequently
• zEDC compression statistics by high level qualifier

Samples are also provided to compare 2 different DCOLLECT runs and find what changed:

• Change in space by storage group
• Change in space by the top 50 high level qualifiers
• Change in space for selected dataset names

Convert DCOLLECT Records to JSON format​

DCOLLECT records can also be converted to JSON format, for use with Splunk, Excel or other JSON reporting tools.

See the Samples​

Sample code is available on Github:

https://github.com/BlackHillSoftware/easysmf-samples/tree/main/dcollect

30 Day Trial​

Get a 30 Day Trial, or request further information:

Finding work running under UID 0 and understanding what it is is important for security on z/OS.

Some UID 0 processes run for hours and are easy to find with tools like SDSF. Other work might only run for a fraction of a second, and is virtually impossible to catch “in the act”.

SMF data can help. SMF type 30 records have Unix Process sections for z/OS unix work. The Unix Process section includes the UID so you can use it to find UID 0 work.

Here is an example of the EasySMF Unix Work report:

EasySMF Unix Work Report, filtered to show UID 0.

The UID field allows you to filter the report to include only work with UID 0.

The report builds a tree using the parent process information so you can see the relationship between different tasks. Information from SMF about the running program is included, which can help to understand what the work is doing.

The Elapsed column shows how long these tasks were running. Most of these tasks were part of system startup and many ran for less than half a second, so it would be very difficult to catch them in real time.

You can download a 30 day trial to see what the data from your system looks like here:

EasySMF 30 Day Trial

Black Hill Software does not use or distribute Apache Log4j in any of our products.

EasySMF:JE does use SLF4J which can be configured by the customer to use Log4j, if the customer provides the Log4j components. Even in this case EasySMF:JE does not log any information from untrusted sources so we do not believe it is vulnerable to this exploit.

However, if customers have configured logging to use Apache Log4j they should upgrade Log4j to a fixed version.

Which language is faster on z/OS, Java or C++? People will tell you C++ is fast and Java is slow, but does that stand up to a drag race?

Dave Plummer is a retired operating systems engineer from Microsoft. He has created an interesting series of videos “drag racing” different languages and different hardware with a small program searching for prime numbers. The initial video raced C++, Python, and C#. Then he raced an Apple M1 vs an AMD ThreadRipper 3970X vs a Raspberry Pi.

I thought it would be interesting to run the drag race on z/OS, putting C++ up against Java. z/OS people like to tell you that Java is slow – but is that really true?

The program uses the sieve of Eratosthenes to search for prime numbers. The program works through odd numbers starting at 3 and marks each multiple as “not prime”. Then it moves to the next number that has not already been marked as a multiple of another number and repeats the process. At the end, numbers that have not been marked are prime.

This is repeated for numbers up to 1,000,000 as many times as possible in 5 seconds, and the number of passes is the result.

The “drag race” description acknowledges that this isn’t a comprehensive benchmark, just a test of speed at a particular task like drag racing a car.

Setup​

The C++ and Java programs had been developed and refined on other platforms. The Java code ran without modification, but the C++ code required a few changes:

• I couldn’t find <chrono> on z/OS so I used gettimeofday for the timing
• Some changes to initialization etc. were required due to unsupported syntax

The C++ code was compiled from the unix command line:

xlc -o PrimeCPP31 -O3 -Wl,xplink -Wc,xplink,-qlanglvl=extended0x PrimeCPP.cpp

I configured the zIIP offline for the tests so that the C++ and Java code were running on the same processor.

All source code is available here, if you want to try it out on your own system:

https://github.com/andrew890/Primes-zOS

Note: z/OS CPU speeds vary widely based on the capacity purchased. The z15 LSPR ratios list z15 systems with single CPU MSU ratings from 12 MSU to 253 MSU – a 20x difference! The numbers here should be a reasonable comparison between the languages tested, but be careful comparing them with a different system.

Round 1​

Source code:

• C++ : https://github.com/andrew890/Primes-zOS/blob/main/PrimeCPP/solution_1/PrimeCPP.cpp
• Java : https://github.com/andrew890/Primes-zOS/blob/main/PrimeJava/solution_1/PrimeSieveJava.java

Results (higher number is better):

I was surprised – I expected Java to do well, but I didn’t expect C++ to do so badly.

There wasn’t anything I could see in the C++ code to make it slower than the Java code. However, marking and checking numbers is the majority of the work, and this processing is hidden inside a vector<bool> in the C++ code. Using vector<bool> was apparently a big gain on other platforms, but maybe not on z/OS?

I changed the C++ code to use bits in an unsigned char array, explicitly testing and setting bits. This was the method Dave used in his initial code. The Java code used a boolean array. To give the closest possible comparison between C++ and Java I also changed the Java code to use a byte array with the same bit testing/setting.

Round 2​

Source Code:

• C++ : https://github.com/andrew890/Primes-zOS/blob/main/PrimeCPP/solution_2/PrimeCPP.cpp
• Java : https://github.com/andrew890/Primes-zOS/blob/main/PrimeJava/solution_2/PrimeSieveJava.java

Results (higher number is better):

This was a much better result for C++. It looks like the vector<bool> implementation on z/OS is not as good as other platforms. However in Java the original solution was much better. The improved C++ version didn’t significantly beat the original Java solution.

On other platforms C++ was faster than Java by 40-70%. The versions using the byte array showed a similar margin. I don’t doubt that you could write a C++ version to beat the fastest Java version on z/OS, but I don’t think it would be easy.

Bonus: COBOL​

Someone contributed a COBOL version. I tried that out of interest, compiled with OPT(2):

Source Code:

• https://github.com/andrew890/Primes-zOS/blob/main/PrimeCOBOL/solution_2/PRIMES

Result:

Better than the worst C++, but not as good as Java. To be fair, this program is a long way from the type of work COBOL was designed for. I don’t know COBOL well enough to judge if it could be improved.

Scaling it up​

The other interesting test is to scale up from 1,000,000 to larger numbers. I repeated the tests using the different solutions for primes up to 10,000,000, 100,000,000 and 1,000,000,000.

The most interesting result here is the Java boolean[] version. This version is as fast as the fastest C++ version for 1,000,000, but the speed declines much faster as the maximum increases. I guess Java is doing some optimizations that don’t work as well for 1 billion element arrays!

The trend was strong enough that it seemed interesting to try a smaller number as well, so I added a 100,000 run. Very interesting – for 100,000, the Java version using the boolean array was more than 20% faster than C++!

Java Overhead​

Java has some overhead starting the Java Virtual Machine. This can be seen in the SMF data.

The SMF data shows the C++ programs had about 4.95 seconds CPU time and 5.02 seconds elapsed time for the 5 second duration measured by the program.

The Java programs had about 5.24 seconds CPU time and 6.16 seconds elapsed. This presumably reflects the overhead of starting the JVM. There was only one CPU online, so any runtime overhead after the program records the start time will be reflected in the score. Java GC etc. threads could not run in parallel on another CPU and accumulate CPU time without slowing the main program. This startup overhead should be less significant for longer running programs.

Conclusion​

Java on z/OS is not slow. It can match C++ for speed, to the point where the selection of algorithms and data structures is more important than the language itself. Java deserves to be considered a high performance language on z/OS, as much as C++ or COBOL. There is one caveat: there is significant overhead starting the JVM, so it might not be a good choice for small programs that run very frequently.

Java’s reputation for being slow probably comes from the ease of combining existing components into very large applications, where the programmer may not even be aware of the size of what they have built.

Many z/OS systems have general purpose CPs running less than full speed to reduce software bills. If you have zIIPs running full speed, Java might actually be the fastest language on your system by a fair margin, with the bonus that the Java work probably doesn’t contribute to software costs.

Dave’s Videos​

Here are direct links to the first 2 of Dave Plummer’s Software Drag Racing videos:

EasySMF can now load SMF data using the z/OSMF Dataset and File REST API.

The z/OSMF REST API uses HTTPS instead of FTP so it is a good option for sites that don’t want to use FTP. HTTPS works better with firewalls because it uses a single port instead of separate control and data connections used by FTP.

z/OSMF will compress the data for transfer. If the connection is bandwidth-limited, that can make loading SMF data up to 10 times faster.

Limitations​

z/OSMF cancels the REST API task if it hasn’t completed after approximately 15 minutes. This limits the amount of data that can be transferred. However, with 10 times faster data transfer, that could be the equivalent of over 2 hours transfer time using FTP.

Hopefully IBM will relax this limitation in the future.

• Are all my z/OS TCP/IP connections encrypted?
• How do I know what level of TLS is being used?
• Which TCP/IP clients or servers are using insecure ciphers?

zERT – the z/OS Encryption Readiness Technology is designed to answer these questions.

zERT is a function of TCP/IP on z/OS. It collects information about cryptographic security attributes of TCP/IP connections and writes it to SMF. IBM provides some free zERT reports in z/OSMF, but the data needs to be loaded into DB2 before you can view the reports.

EasySMF allows you to view zERT SMF reports without DB2.

zERT can produce 2 types of records – Connection Detail and Aggregation. Like z/OSMF, EasySMF reports on zERT Aggregation records: SMF type 119 subtype 12.

zERT Aggregation records contain similar information to the zERT Connection Detail records, but information for multiple connections with the same security characteristics are combined. This reduces the number of records generated.

The aggregation records still break the information down to the IP address and port level, but they combine information from multiple connections with the same security settings from the same client.

Finding the Important Information​

Even using aggregation records, zERT reports have a lot of information. Records are produced for each client connecting to TCP/IP. Most of these records are not interesting. The entries you probably want to see are connections with specific security attributes, e.g. insecure ciphers or old TLS versions.

EasySMF makes it easy to find the important entries. EasySMF groups connections by security attributes and server port.

Here we can see there are multiple clients connecting to FTP and z/OSMF using TLS V1.0.

zERT Grouping in EasySMF

We can filter the report to show only the TLS 1.0 entries, and expand the groups to show the individual client addresses. To save the report data, you can export it to Excel or in CSV format.

Filtering and expanding groups to view individual clients

zERT is a very useful facility to help you secure your z/OS system. Download a 30 day trial of EasySMF and see how EasySMF can help you interpret your zERT data.