C03 Andy Nguyen - Kiesslich Consulting

IMS in a Sysplex Environment:
Challenges and Solutions
Andy Nguyen, IMS Tools Developer
The (R)Evolution Continues
© 2014 IBM Corporation
Acknowledgements and Disclaimers
 Availability. References in this presentation to IBM products, programs, or services do not imply that they will be available
in all countries in which IBM operates.
 The workshops, sessions and materials have been prepared by IBM or the session speakers and reflect their own views.
They are provided for informational purposes only, and are neither intended to, nor shall have the effect of being, legal or
other guidance or advice to any participant. While efforts were made to verify the completeness and accuracy of the
information contained in this presentation, it is provided AS-IS without warranty of any kind, express or implied. IBM shall
not be responsible for any damages arising out of the use of, or otherwise related to, this presentation or any other
materials. Nothing contained in this presentation is intended to, nor shall have the effect of, creating any warranties or
representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license
agreement governing the use of IBM software.
 All customer examples described are presented as illustrations of how those customers have used IBM products and the
results they may have achieved. Actual environmental costs and performance characteristics may vary by customer.
Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities
undertaken by you will result in any specific sales, revenue growth or other results.
 © Copyright IBM Corporation 2014. All rights reserved.
– U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule
Contract with IBM Corp.
– IBM, the IBM logo, ibm.com, InfoSphere, IMS, DataPower, DB2, and Optim are trademarks or registered trademarks
of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM
trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these
symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published.
Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM
trademarks is available on the Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml
– .NET is a trademark of Microsoft; SAP is a trademark of SAP.
– Other company, product, or service names may be trademarks or service marks of others.
1
2014
© 2014 IBM Corporation
Agenda
 IMS Sysplex Review
 IMS Sysplex Challenges and Solutions
– Shared Queues
– Data Sharing
– IMSplex Resources
Q&A
2
2014
© 2014 IBM Corporation
IMS Sysplex Review
Source: IMS Redbook (SG24-6908-00)
3
2014
© 2014 IBM Corporation
IMS Sysplex Review - Components
IRLM
Source: IMS Redbook
4
IRLM Lock
Source: IMS Redbook (SG24-6908-00)
2014
© 2014 IBM Corporation
IMSplex Configuration
Source: IMS Redbook (SG24-6908-00)
5
2014
© 2014 IBM Corporation
IMS Sysplex Challenges and Solutions
 Shared Queues
– Transaction affinity
– CF structures and log streams
– Queue monitoring and management
– Full Queue Protection
 Shared Databases
– Long lock detection
– Investigate lock holders and waiters
– IRLM statistics
 Shared Resources
– Sysplex serial programs
6
2014
© 2014 IBM Corporation
IMS Sysplex Challenges and Solutions
 Shared Queues
–Transaction affinity
– CF structures and log streams
– Queue monitoring and management
– Full Queue Protection
 Shared Databases
– Long lock detection
– Investigate lock holders and waiters
– IRLM statistics
 Shared Resources
– Sysplex serial programs
7
2014
© 2014 IBM Corporation
Transaction affinity – The Challenges
 Force local affinity for efficiency
 Assign affinity to specific IMS due to resource availability
 Reduce false-scheduling overhead
 Reduce contention
8
2014
© 2014 IBM Corporation
Transaction affinity - Roll your own solution
 SYSGEN update – Define Trans with SERIAL=YES
 Dependent regions or transactions classing
 /STOP transactions where they should not run
 User exit
– Implement the DFSMSCE0 exit
– Issue UPDATE TRAN NAME(tranname) START(SCHD) or
STOP(SHCED) with the OPTION(AFFIN) command on the local IMS
to get notifications of messages’ arrival (Note: AFFIN status is lost
across cold starts, so this command must be re-issued)
9
2014
© 2014 IBM Corporation
SM Solution - Transaction Affinity Highlights
 Finer control of transaction scheduling
 Non-invasive to existing definition and operation
– No omission of transaction definitions in sysgen
– No stopping of transactions
– No re-classing of dependent regions
– No operational impact for loss of a system
 User defined affinity to route transaction messages
– Any IMS in the shared queues group
– Any subset of IMS systems
– Equal or weighted distribution
10
2014
© 2014 IBM Corporation
Affinity Routing Statistics – System View
11
2014
© 2014 IBM Corporation
Affinity Routing Statistics – Update System
12
2014
© 2014 IBM Corporation
Affinity Routing Statistics – Destination View
13
2014
© 2014 IBM Corporation
Affinity Routing Statistics – Update Destination
14
2014
© 2014 IBM Corporation
Affinity Routing Statistics – Add Destination
15
2014
© 2014 IBM Corporation
IMS Sysplex Challenges and Solutions
 Shared Queues
– Transaction affinity
–CF structures and log streams
– Queue monitoring and management
– Full Queue Protection
 Shared Databases
– Long lock detection
– Investigate lock holders and waiters
– IRLM statistics
 Shared Resources
– Sysplex serial programs
16
2014
© 2014 IBM Corporation
IMS Sysplex Review – IMS CF structures and log streams
Log streams
(CQS, OM,
RS)
Source: IMS Redbook (SG24-6908-00)
17
2014
© 2014 IBM Corporation
IMS CF Structures and log streams – The Challenges
 Growing use of Coupling Facility Structures
– Data Sharing, Shared Message Queues, Resource Manager, CQS log streams
– No single source for list of in use structures and details
 Check log stream attributes
 Monitor offload activities
18
2014
© 2014 IBM Corporation
IMS CF Structures and log streams – Roll your own solution
 Commands
– XCF command
• D XCF,structure
– RM structure
• QUERY STRUCTURE SHOW(ALL)
– CQS command
• /CQQUERY STATISTICS STRUCTURE ALL
• /CQCHKPT SYSTEM or SHAREDQ STRUCTURE structurename
– Check log stream attributes
• DISPLAY LOGGER,L
 Exits
– View Statistics for structure rebuild and checkpoint
• Implement BPE-based CQS user-exit (STATINV= and TYPE = STRSTAT)
– Monitor offload activities
• Write an ENF 48 listener exit
• ENFREQ service
19
2014
© 2014 IBM Corporation
SM Solution - IMS CF Structures and log streams
 Real-time display of structure list
 Statistics, Connections, Coupling Facility information
 Structure rebuild and checkpoint stats
 Log streams attributes and offload monitoring
20
2014
© 2014 IBM Corporation
IMS Coupling Facility Structures
21
2014
© 2014 IBM Corporation
CQS Structures
22
2014
© 2014 IBM Corporation
CQS Structures
23
2014
© 2014 IBM Corporation
CQS Structures
24
2014
© 2014 IBM Corporation
CQS and System Logger
Source: IMS Redbook (SG24-6908-00)
25
2014
© 2014 IBM Corporation
SM Solution - CQS Log Stream info
26
2014
© 2014 IBM Corporation
SM Dashboard
27
2014
© 2014 IBM Corporation
IMS Sysplex Challenges and Solutions
 Shared Queues
– Transaction affinity
– CF structures and log streams
–Queue monitoring and management
– Full Queue Protection
 Shared Databases
– Long lock detection
– Investigate lock holders and waiters
– IRLM statistics
 Shared Resources
– Sysplex serial programs
28
2014
© 2014 IBM Corporation
IMS Shared Queues
29
Source: IMS Redbook (SG24-6908-00)
2014
© 2014 IBM Corporation
Shared Queue Management – The Challenges
 Visualize queue content
 Browse, Delete, and Recover messages
 View CQS processing counters
30
2014
© 2014 IBM Corporation
Shared Queue Management - Roll your own Solution
 Commands:
– Visualize queue content
• /DISPLAY QCNT command
– Browse, Delete and Recover messages from Cold queue
• /DEQ command
• QUEUE LTERM|TRAN NAME(XXX) OPTION(DEQ1 or DEQALL)
 Programmatically:
– CQS processing counters
• Write your own CQS client
31
2014
© 2014 IBM Corporation
SM Solution - Shared Queue Management
 Filter based on Destination name or ID
 High level queue depth information, Drill down to levels of detail
 Browse, Delete and Recover messages from Cold queue
 CQS processing counters
32
2014
© 2014 IBM Corporation
Destination Queue Depth
33
2014
© 2014 IBM Corporation
Destination Queue Depth
34
2014
© 2014 IBM Corporation
Destination Queue Depth
35
2014
© 2014 IBM Corporation
Destination Queue Depth
36
2014
© 2014 IBM Corporation
Destination Queue Depth
37
2014
© 2014 IBM Corporation
IMS Sysplex Challenges and Solutions
 Shared Queues
– Transaction affinity
– CF structures and log streams
– Queue monitoring and management
–Full Queue Protection
 Shared Databases
– Long lock detection
– Investigate lock holders and waiters
– IRLM statistics
 Shared Resources
– Sysplex serial programs
38
2014
© 2014 IBM Corporation
Full Queue Protection Challenges
 Local Queue
– Local buffer overflow can abend or stall IMS control region
 Shared queue structures
– Full structures will stall IMS
39
2014
© 2014 IBM Corporation
Full Queue Protection – Roll your own solution
 Local Queue
– Write your own Queue Space Notification Exit (DFSQSSP0)
 Shared queue structures
– z/OS XCF and CQS commands to monitor structure fullness
– No Exit to prevent structure fullness yet
40
2014
© 2014 IBM Corporation
SM Solution - Full Queue Protection
 Local Queue
– Employ the Queue Space Notification Exit (DFSQSSP0)
– Automatic actions for warning, critical or Severe thresholds
– Identify programs using large number of buffers
 Shared queue structures
– Alert structure utilization exceeding predefined thresholds
– Unable to reject new messages (a possible future enhancement)
41
2014
© 2014 IBM Corporation
IMS Sysplex Challenges and Solutions
 Shared Queues
– Transaction affinity
– CF structures and log streams
– Queue monitoring and management
– Full Queue protection
 Shared Databases
–Long lock detection
– Investigate lock holders and waiters
– IRLM statistics
 Shared Resources
– Sysplex serial programs
42
2014
© 2014 IBM Corporation
Data Sharing Components
43
Source: IMS Redbook (SG24-6908-00)
2014
© 2014 IBM Corporation
Diagnosing Long Locks - The Challenges
 DB Lockouts by applications holding IRLM locks for an inordinate amount of
time
– Could go unrecognized until it becomes critical
– Lack of supported tools to assist in recognition and identification of problem
– Manual intervention required to resolve
44
2014
© 2014 IBM Corporation
Diagnosing Long Locks - Roll your own solution
 Activate IRLM timeout detection. DXR162I messages are issued.
– F irlm,RUNTIMEO
 Automatically kill transactions (status code BD or abend 3310) that
wait for lock after # of seconds
– MODIFY,irlmproc,SET,TIMEOUT=nn where nn=number of seconds
– the LOCKTIME parameter in the DFSVSMxx member
 Obtain Holder/Waiter report
– Keyword MSG2291I= in the DIAGNOSTICS_STATISTICS section of
DFSDFxxx PROCLIB member
– Analyze RMF record 79.15
 Turn on IRLM trace
– TRACE CT,ON,COMP=IRLM
45
2014
© 2014 IBM Corporation
SM Solution - Data Sharing Long Locks
 Automatic real-time recognition when IRLM detects
 Information consolidated, analyzed for top blocker, and presented
 Information recorded in exceptions file and sent to z/OS console
 Messages can be sent to z/OS console using user exit so that
automated operations can resolve
 Problem quickly resolved with automation
46
2014
© 2014 IBM Corporation
Data Sharing Long Lock Exceptions
47
2014
© 2014 IBM Corporation
Data Sharing Long Lock Exceptions
48
2014
© 2014 IBM Corporation
IMS Sysplex Challenges and Solutions
 Shared Queues
– Transaction affinity
– CF structures and log streams
– Queue monitoring and management
– Full Queue protection
 Shared Databases
– Long lock detection
–Investigate lock holders and waiters
– IRLM statistics
 Shared Resources
– Sysplex serial programs
49
2014
© 2014 IBM Corporation
Real-time IRLM / PI Locking Display – The Challenges
 Identify resource contention
– Which IMS, dependent region, application is holding resources
– Untangle queue of holder/waiters
– Determine resources with contention
50
2014
© 2014 IBM Corporation
Real-time IRLM / PI Locking Display – Roll your own solution
 None??
51
2014
© 2014 IBM Corporation
SM Solution - Real-time IRLM / PI Locking Display
 Display of resources with contention
– Select by dependent region and drill down to resource
– Select by resource and drill down to dependent region
52
2014
© 2014 IBM Corporation
Real-time IRLM Locking Display
53
2014
© 2014 IBM Corporation
Real-time IRLM Locking Display
54
2014
© 2014 IBM Corporation
Dependent Region Display
55
2014
© 2014 IBM Corporation
IMS Sysplex Challenges and Solutions
 Shared Queues
– Transaction affinity
– CF structures and log streams
– Queue monitoring and management
– Full Queue protection
 Shared Databases
– Long lock detection
– Investigate lock holders and waiters
–IRLM statistics
 Shared Resources
– Sysplex serial programs
56
2014
© 2014 IBM Corporation
IRLM Statistics – The challenges
 Managing the well being of IRLM(s)
– Deadlocks, false contentions, storage utilization?
– Multiple IRLMs to check
57
2014
© 2014 IBM Corporation
IRLM Statistics – Roll your own solution
 Analyze IMS log records X’45’ sub-record X’21’ and X’22’
58
2014
© 2014 IBM Corporation
SM Solution - Aggregated IRLM Statistics
Information gathered from IRLMs across Sysplex
– Aggregated into single system image
– Drill down for information from individual IRLMs
59
2014
© 2014 IBM Corporation
Aggregated IRLM Statistics
60
2014
© 2014 IBM Corporation
IMS Sysplex Challenges and Solutions
 Shared Queues
– Transaction affinity
– CF structures and log streams
– Queue monitoring and management
– Full Queue protection
 Shared Databases
– Long lock detection
– Investigate lock holders and waiters
– IRLM statistics
 Shared Resources
–Sysplex serial programs
61
2014
© 2014 IBM Corporation
IMSplex Resource Management
62
Source: IMS Redbook (SG24-6908-00)
2014
© 2014 IBM Corporation
IMSplex Resource Management
Source: IMS Redbook (SG24-6908-00)
63
2014
© 2014 IBM Corporation
Sysplex Serialized Program Management – The Challenges
 The IMS who scheduled the serial program abnormally terminated and not
coming back for a while, the scheduled serial program is locked and no
other IMS in the Sysplex can use it.
64
2014
© 2014 IBM Corporation
Sysplex Serialized Program Management – Roll your own
solution
 Write your own RM Client
65
2014
© 2014 IBM Corporation
SM Solution - Sysplex Serialized Program Management
 Delete the locked RM entry to allow the serial program to be
scheduled by other IMS via ISPF
66
2014
© 2014 IBM Corporation
IMS Resource Structure Content
67
2014
© 2014 IBM Corporation
IMS Resource Structure Content
68
2014
© 2014 IBM Corporation
IMS Resource Structure Content
69
2014
© 2014 IBM Corporation
IMS Resource Structure Content
70
2014
© 2014 IBM Corporation
IMS Resource Structure Content
71
2014
© 2014 IBM Corporation
Summary
Challenges
RYO Solutions
Sysplex Manager
IMS Commands/User
Exit/Classing scheme
Seamless, persistent,
dynamic update / ISPF
z/OS and CQS Commands
ISPF
Write ENF exit
ISPF and alert
IMS Commands
ISPF
Activate IMS timeout exit
Real time WTO alerts
Lock Contention report
RMF 79.15
Holder/Waiter real-time via
ISPF
IRLM Statistics report
IMS log records X’45’ subrecord X’21’ and X’22’
ISPF
Write RM client
ISPF
Transaction Affinity
management
CF Structures monitoring
Log streams offload
monitoring
Shared Queues
Management
Long Lock Detection
RM structure management
72
2014
© 2014 IBM Corporation
Q&A
73
2014
© 2014 IBM Corporation
Contact information
 Andy Nguyen
 E-mail address: [email protected]
74
2014
© 2014 IBM Corporation
Thank You!
Your Feedback is Important to Us
75
75
2014
© 2014 IBM Corporation

Download Report