Cullen Programming

VM Programmable Operator Automation Management Facilities


Users Guide


Pre/Co-requisite Software

IBM's zVM Programmable Operator Facility (PROP)

IBM's WAKEUP Facility
IBM's VMARC module




How to Enable the full base of OPERLOG and PROP Facilities

Implement the VM Programmable Operator (PROP) as per IBM documentation.

Next configure your PROP Routing Table and have it reside on OPERATOR's 291-disk.
If you need examples see the SAMPLE.RTABLE, files supplied on the OPERATOR 291-disk.

Logon to your assigned Logical Operator LOGLOPER. Note that if you use IBM's PERFKIT, then have this machine act as the Logical Operator and disregard the use of the PROPROLL MODULE. The PROPROLL MODULE is for non-PERFKIT users that need a rolling operator console and a facility to forward unfiltered messages. The PROPROLL MODULE is a licensed feature that is available with the professional version.
Start up the Logical Operator.

Leave the Logical Operator LOGLOPER Logical Operator terminal for now.

Log on to the OPERATOR virtual machine from another terminal and ensure that PROP is running in DISCONNECTED MODE.

If needed, see the OPERATOR PROFEXEC sample provided with the VMTOOLS 2192 disk components.

IPL CMS. This will start the VM Programmable Operator and then disconnect from its physical console.

Logon back to the Logical Operator LOGLOPER virtual machine.

Set the CPAUTOCP system global variable to ENABLED so that next IPL is automation ready.

Please note for implementations prior to zVM6.4 use the execs SETGLV and QUERYGLV to change and query the global variables.
For zVM6.4 and later we exploit the newer system variables. Use the execs SETCPGLV and GETCPGLV to manipulate system variables.

OP SETGLV CPAUTOCP ENABLED
Filtered Messages will now be routed by the Routing Table rules to this console.

All message traffic, (filtered or otherwise), will be viewable from any CMS machine via the OPERLOG facility.




How to Disable the OPERLOG and PROP Facilities



Please note for implementations prior to zVM6.4 use the execs SETGLV and QUERYGLV to change and query the global variables.
For zVM6.4 and later we exploit the newer system variables. Use the execs SETCPGLV and SETCPGLV to manipulate system variables.
Logon to the Logical Operator LOGLOPER

Set the CPAUTOCP system global variable to DISABLED so that automation is suppressed at the next IPL.

OP SETGLV CPAUTOCP DISABLED
Shutdown the Logical Operator at its command prompt.

Log on to the OPERATOR virtual machine. DO NOT HIT ENTER.

Type STOP. This command will terminate the VM Programmable Operator.

The OPERATOR console will now be running connected with CMS standard message handling.

The Logical Operator LOGLOPER will no longer receive messages since none will be forwarded by the OPERATOR.

The console is now operating in the original VM mode.




PROPROLL Display

PROPROLL is a facility that will capture and display in fullscreen mode all messages that pass filtering from the OPERATOR. That is, OPERATOR will transmit all messages that have not been filtered, or acted upon, to the LOGICAL OPERATOR where PROPROLL is running.

PROPROLL was designed to execute from a terminal display located in the Operations Area. The OPERATOR virtual machine must be running the VM PROP module in the disconnected state.

PROPROLL will accept both CP and CMS commands to be directed to the Operator. These commands will be transmitted to OPERATOR for execution on behalf of the Logical Operator. The response being logged and returned to the Logical Operator.

Prepend all commands destined for execution on the OPERATOR with OP. The OP Exec will insert the "CP SMSG OPERATOR CMD" before each command to save you typing effort.

All commands to be executed in the Logical Operator virtual machine itself, must start with CP.



PROPROLL Simple Navigation

ROLLEXIT will terminate the program.
ORIGIN will toggle the message origin on and off.
PF3 will cause the termination sequence which requires confirmation via PF2.
PF6 will retrieve the last entered command.
PF7 will scroll backward one screen of messages.
PF8 will scroll forward one screen of messages.
PF12 will return to normal scroll mode.




OPERLOG Display

The OPERLOG display is used to monitor the VM OPERATOR message traffic. It does this by simply accessing and reformatting the VMPROP logging performed on the OPERATOR machine by PROP.

Since OPERLOG was written mostly in REXX using XEDIT, then it will run anywhere that CMS runs, on remote or local displays.

STARTUP
Initiate the facility by typing OPERLOG.
To suppress the logo type OPERLOG (NOLOGO

You will see the CP logo followed by the initial screen.
When in the INITIAL display you will see (I) at the upper right-hand corner.



OPERATING MODES
The OPERLOG display operates in two modes:

When in STATIC display mode (S), Commands prepended with OP will be accepted on the command line and directed to the VM OPERATOR for execution on your behalf. The originating user must be authorized by the PROP Routing Table.
Upon refresh of the display (PF2) the results will be logged and displayed.

CP commands will also be accepted, but keep in mind that these are not directed to OPERATOR.
So be cautioned that you are on your own with these commands since they will execute in your virtual machine.

Some examples:

OP CP QUERY NAMES
OP CP SMSG VMCRON IPL
OP CP XAUTOLOG LINUX7
OP CP SET ZONE EDT

Caution: DO NOT enter commands that will pause, prompt, suspend or bring down the OPERATOR virtual machine.



When the facility is operating in ROLL mode (R) any commands you issue will be ignored. ROLL mode is activated by PF9.

All messages directed to the VM OPERATOR, filtered and otherwise, are captured in the OPERATOR log and therefore are available for display by OPERLOG. Messages since last midnight are available for viewing.

OPERLOG can also retrieve a past OPERATOR console log by specifying the option (-n ,where n is the number of days in the past. The past console logs are archived in the CONSOLES repository at the start of each new day. The logs filenames are date stamped.

OPERLOG ( -2
OPERLOG ( -14

OPERLOG keeps the most recent 2000 lines of message traffic in its buffers. Should you wish to view all the traffic since the last midnight then enter the option (ALL.

OPERLOG ( ALL


SEARCHING
You can find all occurrences of a particular string in the log by entering ONLY some_string on the command line.

ONLY TCPIP
ONLY CULLPROG
ONLY HCPxxxxx
ONLY SUSE29T
ONLY RHEL64

Note that the string arguments are CASE-SENSITIVE.
All messages that contain the search string will be displayed.
To enter another search you will need to use PF2 to refresh the display.



MESSAGE TRAFFIC FILTERING
You can choose to view only message traffic from a particular source or containing a given string. First use the macro ONLY to select some_string from the command line.
Then once you have a first screen that contains only logging lines matching your selection hit the PF9 Key to start automatic refreshment





PROGRAM FUNCTION KEYS
Note that you have PF keys assigned for backward/forward (PF7/PF8) and side-to-side (PF4/PF5) movement.
Also you can use the commands TOP and BOTTOM to reposition the view to the beginning and end of the log buffer.

Toggling the PF9 key will place OPERLOG into and out of ROLL Mode. The screen will automatically refresh itself with the most current messages every 10 seconds.

The PF2 key will refresh the screen with the most current data when in STATIC Mode.

The PF6 key will retrieve the last commands issued.

The PF3 key will engage the termination sequence for the facility.

OPERLOG will roll over automatically at midnight to access the next day's OPERATOR logging.
If the CONSOLES repository facility is installed and the CONSEND exec activated, then the past day log will be archived.






Theory of Operation


This automation facility uses the IBM Programmable Operator Facility as its information foundation. No VM PROP feature is superceded or altered. We simply exploit its capabilities. Below is a description of component areas:




Events and Triggers

Both solicited and unsolicited message events directed to the VM OPERATOR are candidates for an automated response. Also Time-of-Day events signaled by the VMTOD service machine.



Event Capture Grid

The VM PROP ROUTING TABLE is the grid or matrix used to define which message events get captured and acted upon. Time-of-Day events appear as messages directed to the OPERATOR by VMTOD.



Rules

Rules are the fast execution, first-level REXX execs (Action Routines) optionally called by the ROUTING TABLE for a matched event. Events that do not invoke a rule are simply filtered out or suppressed from appearing in the message traffic. Thus, these message events are not forwarded to the Logical Operator, but do appear in OPERATOR's logging.

Please note that if a rule is missing then the VM Programmable Operator will fail to initialize. An error message will call out the rule that is missing.



Rule invoked Action Routine Execs

These are the second-level REXX execs that may be invoked by the Rule Execs. Note that if these execs should wait for something (block), then further message event processing is delayed and some message traffic may be missed. These execs run in the OPERATOR address space. Ensure that they do not block.

Please note that if an exec is missing then you will get an error message on the Operator log when the calling RULE fires.



Execs that Block (Professional Version)

These are the second-level relatively long-running REXX execs called by Rules that may block due to WAITS (such as SLEEP 2 SECS). The blocking execs are dispatched to secondary delegated servers that can tolerate waits. Since the delegated servers run asynchronously to the OPERATOR, the main server is free to handle new incoming message traffic. The delegated servers will run the execs and forward results and messages to the OPERATOR.


Accessor and Mutator Methods for System Global Variables



All VM PROP execs, known as RULES, are activated/deactivated by shared SYSTEM GLOBAL VARIABLES maintained by the OPERATOR virtual machine.

All rules have a corresponding global variable of like name.

The values of all global variables can be accessed by the GETCPGLV Exec (the accessor).

The values of all global variables can be altered by the SETCPGLV Exec (the mutator).

The values used by automation are in the state of: ENABLED, DISABLED and TRACE

If ENABLED the corresponding rule will fire when its triggering message event designated by the Routing Table entry occurs.

If DISABLED the rule will NOT fire when its trigger message event occurs.

If set at TRACE then the exec will run as ENABLED but with tracing of the exec written to the log.

All rules are initially set by the PROP123H EXEC that executes just after IPL time, dispatched by the AUTOLOG1 virtual machine's PROFILE EXEC.

Modify the PROP123H EXEC exec to set the initial ENABLED/DISABLED/TRACE status of each of your rules.

To ENABLE/DISABLE/TRACE a rule immediately request OPERATOR with the SETCPGLV EXEC. You will receive a confirmation message following its execution.

OP SETCPGLV PAGEZERO DISABLED


To QUERY the setting of a rule, request OPERATOR with the GETCPGLV EXEC. You will receive a confirmation message following its execution.

OP GETCPGLV PAGEZERO


The CONSOLE Logging Repository

This repository will contain the archived OPERATOR logs from prior days.

A virtual machine named CONSOLES should execute, on an hourly basis, a REXX program named CONSOLES is employed to read spooled console data sent to its virtual reader by any guest, server or user that so spools its console in such manner as it is directed to be delivered to the repository.

The CONSOLES program will file the data and associate it with a dated timestamp, keeping the most recent N number of days.

Console data for the same day will continue to be appended. The number of days can be modified by changing the constant in the CONSOLES Exec found on the VMTOOLS 2192 (D disk).

The repository can be accessed by the CONSLINK exec.

You must set up VMCRON to cause an hourly XAUTOLOG of the CONSOLES machine.



CONSOLES is used to store general user logs.



Optional SERVCONS is used to store more sensitive service machine logging. You must set up VMCRON to cause an XAUTOLOG of both machines.




The VMCRON Scheduler

The VMCRON facility performs much of the same function at VMUTIL.

Used for the dispatching of processes needed by OPERLOG Automation Management.



This virtual machine application can be controlled by the CP SMSG command.

CP SMSG VMCRON HELP will display each of the control functions.

To perform any control function simply:

CP SMSG VMCRON <function>



Example:

CP SMSG VMCRON help

CP SMSG VMCRON SHUTDOWN






The VMTOD Heartbeat Facility

The VMTOD facility provides a Time-of-Day heartbeat function for PROP.

Used by PROP Rules/Execs to calculate time displacements where needed.



This virtual machine application can be controlled by the CP SMSG command.

CP SMSG VMTOD HELP will display each of the control functions.

To perform any control function simply:

CP SMSG VMTOD <function>



Example:

CP SMSG VMTOD help

CP SMSG VMTOD IPL

CP SMSG VMTOD SHUTDOWN

CP SMSG VMTOD CONSOLE






Implementing Changes to Automation

The systems programmer can effect change to automation management by altering either or both of the PROP Routing Table (see IBM documentation) and the EXEC programming invoked by the PROP mechanism.

First make your changes to the execs on the OPERATOR 291 (O-disk) via a MR link. Recall that OPERATOR maintains only a READ-ONLY link to this disk.

Next make any changes to the execs on the VMTOOLS 192 (D-disk) or the VMTOOLS 2192 (E-disk) or the VMTOOLS 3192 (F-disk) via a MR link. Again, OPERATOR maintains a READ-ONLY link here also.

Then, make any changes to the PROP Routing Table on the OPERATOR 291 (O-disk).

Finally, cause OPERATOR to "recognize" the changes and place them into effect by issuing from the Systems Programmers maintenance virtual machine LOADTBL. This exec resides on the OPERATOR 291 (O-disk).

Note on the OPERATOR log, via OPERLOG or Logical Operator console that the change went into effect.

LOADTBL




Changes to the VMCRON Scheduler will take effect upon the next CMS IPL command sent to VMCRON via the SMSG facility.

OP CP SMSG VMCRON IPL





Operator Sub-Server Machines

The Sub-Server machines are delegated for handling asynchronous units of work that may be termed as blocking. That is these tasks may be delayed due to a WAIT for I/O or an expiration of a TIMER POP or for some message EVENT occurrence before processing on this workunit can continue.

The main server, (OPERATOR running PROP), must always be free to quickly handle any and all message events being presented when they occur.

Any tasks determined by the Systems Programmer that exhibit properties of "blocking" or waiting for some event should be delegated and dispatched to one of the sub-server tasks, where they can be permitted to WAIT for their next signaled event.



OP OPSTATUS will display the current state of each of the Servers.

OP OPSREADY will change the state of each Server to Waiting-for-Work.

OP OPSINIT will logon each Server.

OP OPSHUT will shutdown each Server and log it off.

OP OPSRSTRT will shutdown and restart each Server.


Some of the VM Tools

The VMTOOLS "nologged" virtual machine is used to house any REXX exec programming and data that would be common tools useful to many CMS users and used by installed Servers.

Most if not all programming code provided by Cullen Programming in support of the zVM environment will be stored in this area.

CONSLINK

Purpose: Provide linkage to the CONSOLES Repository.

Operands: userid | all (Default: this userid)



Usage Notes:

Dependencies:
This exec is expecting the READ password on the CONSOLES userid to be ALL. If this is not the case then the exec needs to be modified to supply the password.

Return Codes:



AUTODALY

Purpose: Processes kicked off daily.

Operands: NONE



Usage Notes:

Return Codes:



AUTOHOUR

Purpose: Processes kicked off hourly.

Operands: NONE



Usage Notes:

Return Codes:



GCOPY

Purpose: XEDIT Macro to write to clipboard the lines selected by CC prefix command.

Operands: CC prefix command block

Usage Notes:
Replaces last contents of scratchpad.

Return Codes:



GPASTE

Purpose: XEDIT Macro to copy clipboard contents to area following F prefix command, or preceding P prefix command.

Operands: F or C prefix command

Usage Notes:
Clipboard contents remain undisturbed.

Return Codes:



LCONSEND

Purpose: Retrieve consoles and send to repository of any guest enabled for SIGNALS.

Operands: NONE



Usage Notes:

Dependencies:


Return Codes:



LOGNAUTO

Purpose: Rule that is given control when PROP detects a virtual machine logon event.

Usage Notes:
Commands to configure a virtual machine after its creation by CP has occurred.

Return Codes:



OP

Purpose: Exec to send CP commands to OPERATOR from PROP authorized users.

Operands: CP command

Example: OP CP QUERY NAMES

OP CP VARY OFF 2345

OP CP WARNING VM System coming down at 10:15



Usage Notes: Do not send anything that will cause OPERATOR to TERMINATE or enter a WAIT state.
This exec can be used from the CMS terminal screen or OPERLOG while in STATIC MODE.

Return Codes:



OPERLOG

Purpose: Initiates real-time viewing of the Operator console with scroll and refresh capabilities.

Operands: None | ALL , if for current day. (Default: None)

(-n ,for prior n day

Example: OPERLOG ( -7 will display last week's 7 day old logging.

Usage Notes: Not for submitting commands to the Operator while in ROLL MODE.

Dependencies:
PROP must be up and running.

Return Codes:



PAGEZERO

Purpose: Locks the Prefixed Storage Area of the guest virtual machine address space.

Operands: None

Usage Notes:
This program should be trigged following the IPL of every high performance guest or SVM.

Return Codes:



PROP123H

Purpose: Rule to Initialize Automation State Global Variables.

Usage Notes:
This program should be trigged following the IPL of zVM.
Disallowed to run if after 30 mins since last IPL. Changeable by encoded constant.
Any additional global variables should be listed here to support OPERLOG automation.

States: ENABLED (rule fires), DISABLED (rule dormant),
TRACE (rule fires with tracing),
WEAK (reserved),
WAITING (server is enabled and waiting for work).

Return Codes:



REXXTRAP

Purpose: REXX EXEC Error capture and diagnostic display.

Operands: None

Usage Notes: Reacts to SIGNAL events.
This program is used by every supplied Exec.

Return Codes:



SCANSTRG

Purpose: Menu driven Exec to scan accessible files or filetype on an accessed minidisk for all occurences of a given string of characters.

Operands: None, this is a fullscreen GUI application



Usage Notes: Case sensitivity can be ignored or respected on search.

Optional CHANGE field to change string upon match.

Produces a list of files and lines where the string match occurs.

Return Codes:



STARTNET

Purpose: Exec to commence the orderly startup of the network.

Operands: None

Usage Notes: To be issued from logical operator.
None

Return Codes:

SHUTNET

Purpose: Exec to commence the orderly shutdown of network. PROP and OPERATOR remain active.

Operands: None

Usage Notes: To be issued from logical operator.
User must respond to confirmation.

Return Codes:

NETRECYC

Purpose: Exec to commence the orderly shutdown of network, followed by timed wait, then restart of network.
PROP and OPERATOR remain active.

Operands: None

Usage Notes: To be issued from logical operator.
User must respond to confirmation.

Return Codes:

SHUTVM

Purpose: Exec to commence the orderly shutdown of guest operating systems, the network, VM service machines. PROP and OPERATOR remain active.

Operands: None

Usage Notes: To be issued after PROPROLL is exited
User must respond to confirmation.

Return Codes:


SHUTPROP

Purpose: Exec to commence the orderly shutdown of PROP and the VM Control Program.

Operands: None

Usage Notes: To be issued after TCPIP has posted down from SHUTVM process.
User must respond to confirmation.

Return Codes:



STATEMGR (Professional version)

Purpose: Manage the state of global variables of monitored processes.
Monitors state changes of : DOWN, STARTING, UP, QUIESCING.

Operands: None

Usage Notes: Pulls arguments of the CMS STACK
This program is not called directly by the PROP Routing Table but by rules which are fired by PROP.

Dependencies:


Return Codes:



SHUTSERV (Professional version)

Purpose: Capture event that a server is commencing shutdown processing.
Changes state from UP to QUIESCING.

Operands: data supplied by the PROP event.

Usage Notes:
This program calls STATEMGR for continued processing.

Dependencies:
Correctly selected PROP triggering message event that causes this rule to fire.

Return Codes:



STRTSERV (Professional version)

Purpose: Capture event that a server is being brought online.
Changes state from DOWN to STARTING.

Operands: data supplied by the PROP event.

Usage Notes:
This program calls STATEMGR for continued processing.

Dependencies:
Correctly selected PROP triggering message event that causes this rule to fire.

Return Codes:



UPSERV (Professional version)

Purpose: Capture event that a server is successfully up.
Changes state from STARTING to UP.

Operands: data supplied by the PROP event.

Usage Notes:
This program calls STATEMGR for continued processing.

Dependencies:
Correctly selected PROP triggering message event that causes this rule to fire.

Return Codes:



DOWNSERV (Professional version)

Purpose: Capture event that a server has completed shutdown processing.
Changes state from QUIESCING to DOWN.

Operands: data supplied by the PROP event.

Usage Notes:
This program calls STATEMGR for continued processing.

Dependencies:
Correctly selected PROP triggering message event that causes this rule to fire.

Return Codes:



INITDASD

Purpose: Full screen facility to CPFORMAT and DSF Init of multiple VM fullpack DASD.

Operands:

Usage Notes:

Dependencies:
This program requires that you have established a R/W link to the DASD (or have the volume attached to your userid).

Return Codes:



VMCONIPL

Purpose: Full screen facility to IPL a zVM guest with/without console designation.
PF keys become armed with diagnostic calls for when debugging is necessary.

Operands:

Usage Notes:
Requires user to enter the IPL address and any LOADPARMS.

Dependencies:
Executed from the virtual HMC 3270 console for the Guest machine.

Return Codes:



LINUXIPL

Purpose: Full screen facility to IPL a zLinux guest.
PF keys become armed with diagnostic calls for when debugging is necessary.

Operands:

Usage Notes:
Requires user to enter the IPL address and any LOADPARMS.

Dependencies:
Executed from the virtual HMC 3270 console for the Guest machine.

Return Codes:



MVSIPL

Purpose: Full screen facility to IPL a zOS guest.
PF keys become armed with diagnostic calls for when debugging is necessary.

Operands:

Usage Notes:
Requires user to enter the IPL address and any LOADPARMS.

Dependencies:
Executed from the virtual HMC 3270 console for the Guest machine.

Return Codes:



SPOOLPRG

Purpose: Discard SPOOL files that are older than some number of days.

Operands: None

Usage Notes:
Number of days can be changed by altering constant in the exec.

Dependencies:


Return Codes:



VDEVSCAN

Purpose: Find and List real address of a given machine's virtual device blocks

Operands: userid or virtual machine name

Usage Notes:
Requires virtual machine to be IPL'd

Dependencies:


Return Codes:




[Return to Index]

[Return to Cullen Programming Home Page]


Cullen Programming logo