◆ 招 聘 講 演 丁h e G u e s t L e c t u r e ◆ “ Trends and lssues of Qualitative(Evaluative) Research of Nursing in the United States'' Faye G.Abdellah RN,Ed D,Sc D,FAAN /Former Deputy Surgeon General U S Public Health.Service (transcribed by Noreen Murata) Thank you Dr Tajima Also want to thank Professor Miyazaki, v′ that you will have a little better understand― ho has beey very helpful in inviting me to this 」 apanese Society of Nursing Research 」 ust very happy tO be ing about Qualitative Research Also,a little about the historical development of Quality Research, some measures or suggested ap― here and to share with you this aftern00n proaches to measuring Quality,the definition, Because l wanted very much to have a chance and most important, I think the most impOr― to really be with yOu and share more tirne tant message l have for you this afternoon is with you, as probably been mentioned, after the phrase called Outcomes lvleasurement By Outcomes S/1easurement really l mean the meeting a complete copy of my paper with the figures and references will be avail― the effect Of v′ hat v′e do upon our patients or able So just relax and don't take any notes clients M″e use the term patient― client inter― and you'1l have that available as well as op― changeably because some of our clients are in pOrtunity for some questions the home,in the community (And they are not) ・…I M′ ant you to think about our poten― I'm very much interested in this topic of quality care and quality research And this tial group as being not only in the hospital afternoon, I would like to take you on a iou■ but in the home, ambulatory care, in the ney which l believe will move us, move nurs― comrnunity ing And I'In very proud to be a nurse as wё ll As this group is primarily interested in as a research scientist But rm always a nurse Research, there are many steps that need to And l wanted to share with you opportunities be taken l want to share with you 、 rhat kinds that we hav6 befOre us to move into the 21st of data we need in relatiOn to Qualitative century And when l received the invitation to Research l will share 、 /ith you also a new be 、′ ith you l knew no better country where agency that we have in the Public Health Ser― you could ready tO move in that direction vice and sOme of the potential layer and the I M/ill briefly talk about why the interest number of nurses who are working in that in Qualitative Research We very commonly agency I'd like to give you a couple of ex― hear about Quantitative Research but I'm talk―amples Of how Qualitatiave Research can be ing abOut Qualitative Research And l hope applied tO nursing education, to the clinical 日本看護研究学会雑誌 Vol.15 No.3 1992 “ Trends and lssues of Qualitative(Evaluative) Research of Nursing in the United States'' setting, and some of the data needed there to the development of the patient classiflcation And tO just finish up with a suggested moni― system, particularly in relation to the elderly toring and evaluation process (So that'sv/hat So sornetimes,you will find that Quality Of we have before us this afternoon) Care is Out of reach But we can at least diflne Now, the quest for Quality is ongoing v′ hat would be from a professional standpoint, a n d t h e r e i s a c o m m i t m e n t t o Q u a l i t y i n tw oh a―t i s n e e d e d f o r c a r e day's(competitive)envirOnment The patients really see Quality of Care They don't call it “ Quality Of Care'' What they call it would be they're “satisfied with care", “my nurse is concerned about me" They are concerned with what will happen to them so that Quality of Care tO them means satisfaction and getting out of the hospita1 0r getting back to the FIGURE QUALITY OF CARE ELEMENTS O APPROPRIATE THE RIGHT CARE PROVIDED IN THE RIGHT PLACE O EFFICIENT community To us,if we were looking at Quality Of Care as a researcher, we want to look at the outcomes in terms of our plan of care fOr that individual What our goals are and as health professionals we would be more con― cerned with health promotion and disease CARE PROVIDED IN THE RIGHT WAY AT THE RIGHT TIME O EFFECTIVE CARE THAT IS PROVIDED APPROPRI ATELY AND EFFICIENTLY AND ACHIEVES THE RIGHT OUTCOMES prevention The patient may be less so, but our emphasis 、 /ould be much broader in terms of our goal There are really three elements of Quality of Care to simplify it We need to say “ Now the challenge to provide Quality Is the care appropriate and the right care, the of Care¨ ・ it's not iust clinical expertise And right place, the right setting for that care?" it is difficult to measure Quality Of Care We Is it efficient?Care provided in the right way also, in order to convey to health policy and in the right time, and effective makers the importance of Quality Of Care, we There are three key v/ords:Appropriate, need to link cost to this M/hat does it cost? Efficient, and Effective2)The last one, effective, And One of the most difficult challenges l had is another way of saying Outcome lvleasures while working in the Federal Government was Sometimes as researchers we ca11 0utcOme `effect varia― Measures `dependent variables'一― to define Quality of Care in terms of a cOst that cOuld be provided (This is difficult to achieve especially when there is pressure to contr01 00stsl)) bles' And that is where the term comes from `Effective' meaning `the impact of what we do upon nursing practice' As nurses, v/e want the best fOr our patients or clients But very often that's nOt vre v′。uld all agree that patients are entitled to (expect)safe, Good― Quality of Care available So l define Quality of Care as `that And v′ e need to build in safeguards withOut care which is needed' and nO mο re The criti― feeling pressured to fol10M/ this We need to ¨ cal v′ Ord is `need' And after l define rneed'・ be flexible though, to accomlnodate this care it was the effort to define `need' that led` me and there are some limitatiOns Particularly 日本看護研究学会雑誌 Vol.15 No_3 1992 “ Trends and lssues Of Qualitative(Evaluative)Research of Nursing in the United States" if we get carried away with this area of Out― at least one approach So if we have to cut com es, that 、/e might become too rigid So down on services, at least we can emphasize we need to keep in mind to be flexible in what is ilnportant and set some standards providing for health care By flexible, I mean meeting the individual needs of patients in different settings Now recent studies have demonstrated, ・ …SO We need to do that and again in the United States, that about 35% WHY SO MUCH INTEREST IN QUALITA― of all surgical deaths and 50% of pos,operative TIVE(E∨ ALUA丁 !VE)RESEARCH? cOmplicatiOns can be prevented (and 359イ Nov′ v′ hy is there so much interest in Qualitative Research? And if you read the Nurses, I feel, and l say this with great literature, it's not iuSt in the health area. It's confidence, provide an essential resOurce to also in the area of education, engineering, health care to maintain and provide Quality technology, and in this wonderful cOuntry of of Care in a cOst effective v′ ay ln the United e refer tO nursing as `the caring pro― yours You have really pioneered in Quality States, v′ Control in Qualitative Research fession' Nov/some physicians are not happy And every other cOuntry, including the United States, we with us doing that But we feel that nursing learn from you So there are a lot of princi― is `the caring profession' and our American ples there that can be adapted to health care pushed to really be concerned about Quality Care' dollars, 600 or 700 billiOn this year (about 12%Of the GNP)700 billion American dol up from that as researchers there is a v′ HISTORiCAL OVERVIEW OF QUALi丁 abOut 260 1nillion people, 37 million people SURANCE still have no health care nOr access to care h。lo Y AS´ QUAL!TY ASSURANCE THE‐ ORISTS IN THE MEDICAL AND NURSiNG Now, we're very cOncerned about this because F!ELDS: of the 37 milllon, about 13 million of those A brief note about the history of Quality And so there's something wrong of Care Actually there are, over a span of 25 lf we years, two physicians who have provided us spend more money than any other country in with ilnportant definitions, important guide‐ the v′ orld but we still have 37 milllon people without access to health care So there is・ e know that just lines and two nurses … One physician was Dr Lawrence ヽ Veed, in Massachusetts, who developed the very familiar Problem Oriented N/1edical Record (for fel10w physicians and their students). system isn't the ansv″er So we are looking fOr a way to better That was in the 1960's I've also accused him e can get out of this dilemma understand hov′ v′ and still protect that patient And that's why M′ e are moving with more emphasis into Qualitative Research rord And picking nities for nursing research abound in the world And yet of our population of putting more dollars, more yen, into the `Care' being the key 、 science of care being developed with opportu‐ lars, which is mOre than any other country because of this concern v′ Nurses Association, introduced just this last une, a health care plan called a `Core Of 」 of Care because M′ e spend more Arnerican are children apan Nurses Association, comparable to your 」 For us in the United States,we have been with our health care delivery system of all hospital admissions are unnecessary) of stealing the plan from the 21 nursing prob_ lems, which was essentially a Problern Ori― ented Record approach for nursing But we're We feel that that is goOd friends The Problem Oriented Record 日本看護研究学会雑誌 Vol.15 No 3 1992 “ Trends and lssues of Qualitative(Evaluative)Research of Nursing in the United States'' linked medical diagnosis from treatment to the Diagnostic Related Groups and that is Outcomes That v′ ord again `OutcOmes' maybe familiar to many of you The tragedy Later, I had the privilege, Or actually, of that, and l hope you can learn from our it was about the same time in the 1960's, mistakes,there was some 467 diagnoses which developing the 21 nursing problems Another were identified but there was not a nursing physician, Dr component Avedis Donabedian, provided us with some helpful language to give us a related to these Diagnositc Related GrOups(DRGs)And since 1981, model so we could understand what 、 ′ e mean problems have been created because nursing when we talk about Qualiけ Care He developed component has not been defined And the rea― a structure for us with emphasis on process son it was nOt defined ― and the Outcome Dr NOrma Lang, a nurse, come lVIeasures I'rn sure you're familiar v/ith some of her we don't have Out― You see, this is so critical for us and I work, v′ ho pioneered in developing what we again… ・ it's almost too late but it is never too call a common definition, common data to be late in a way・ ¨but 、′ e need to move ahead collected, data―set…・ this nationally recognized on that Qualitative Research is theoretically for the development of practice standards based on (phenOmenology)culture, the sym‐ Dr Lang's work has been reflected in Canada and bolic interaction of our history,of our values5) v′ orldwide 3) I was reminded of this yesterday aFter― Donabedian continued his work and noon when l had the privilege and wonderful pushed On for a better definition of Quality visit to your national history museum and assessment and certainly we, to him, we owe institute at Chiba, which is the whOle history our basic vocabulary in that area of the」 apanese culture Of the skills,the atti― (Dona― bedian has pointed out that advances in the tudes, and achievements over thousands Of early 1930s and in the mid 1950s ln the 1960s, years The basis is already there for Qualita_ Donabedian established the basic vocabulary tive and cOncepts we noM′ represent of the supreme ability in defining use in discussing qua― lity of care4)) Research and certainly archeologists Qualitative Research in that category Now the increased interest in Quality 〕 ヾo、/ what made that national history today stems from this (continued)rapid rise museum an institute at Chiba so exciting to of health care costs ヽVe're “hitting the roof" me v′as to see several rooms devoted to data in Our cOuntry on costs and we don't have collectiOn ― maps, r00ting, development of any national health insurance Ⅵ /e have nation― tools, some medicinal things, and sο forth al health obiectiVes with a national agenda Data, v′e cannot get at Outcomes with data But we don't know how to getfrOm Y to Z collection And when l speak of data collectiOn We have set some goals but we have major we rnust have again two componentt of uniform health prOblems For example, in the devel― definitiOn and agreement upon what data we oped countries we have the highest percentage will collect Those two are very essential of infant mOrtality rate per 100,000 Which is (Data collection techniques in qualitative disgraceful! research includes observing, listening, and We have also looked at different ways interviewing The ski1l of 「 Bracketing」 of cOntaining cost ln 1981 there was an ap― alsO essential which requires the researcher proach of prospective payment intrOduced called to suspend a prior assumption to understand 日本看護研究学会雑誌 Vol.15 No 3 1992 is “ Trends and lssues Of Qualitative(Evaluative)Research of Nursing in the United States" the experience of the participants6)) Quality Assurance is an assessment of tO Quality Care We speak of one which is `Implicit Review' which is iudgemental ― the care the patient receives and usually in_ bringing together experts and utilizing their volves measuring or monitoring areas of care judgement We use this a great deal in what that are exarnined including medical care, we call `Consensus Conferences', where we nursing care, and the envirOnment, we musn't have a controversial issue We bring together forget that, and the efficiency of that care experts in the field and they agree upon this TWO METHODS OF MEASUREMENT TO at the moment is the best approach or treat― MON!丁 OR QUALITY: ment in relation to whatever the condition is The other is `Explicit Review' which is much more rigid in terms of lists of criteria and DEFINITION OF TERMS(7) established in advance of care I'1l speak a little rnore about that in a bit QUALITY CONTROL: So the Quality Assurance program in― MONITORING OF A PROCESS TO ASSURE THAT THE QUALITY OF THE RESULTS clude both methods A/1ost Quality Assurance prOgrams measure three general categories : ARE CONSISTENT structure, prOcess, and outcome QUALITY ASSURANCE: DEFINIT10N OF QUALITY CARE AND HOW QUALITY CONTROL OF THE ENTIRE OPE― RAT10N SO THAT RESULTS ARE RELIABLE THREE CATEGORIES ARE MEASURED : STRUCTURE,PROCESS,AND OUTCOME !T CAN BE MEASURED: Now to define Quality Of Care requires a deflnition of the attributes of care provided as well as the criteria of what cOnstitutes QUALITY ASSESSMENT: PROCESS IN WHICH STANDARDS OF QUAL― ITY ARE, ESTABLISHED BY PROCESS OF good care And activities can be divided into technical care//science as well as prevention and management of care CONSENSUS And it's interesting as the American Nurses Association introduced its new core So here we have suggested again the plan of care there was great emphasis upon little specific emphasis on v′ h at we really managed care Where the nurse would have mean and define in terms of Quality Care the total management of the care of that V′ e speak about Quality Control, moni― patient or family But also emphasize the toring our process to assist the quality of the social― psych01ogical aspects of that care as results The key word, most important word it affected that individua13) Again the three t h e r e , i s ` c o n s i s t e n t ' Q u a l iAサ ssurance,Qual approaches(to assessment of quality are pro― ity COntr01 of the entire operation, looking posed) at again this structure of process and out― come So there you have the key word and that would be`reliable' Quality Assessment, the last one, the key thing there would be process Of `consensus' agreed upon by the majority There are two methods of measurement 日本看護研究学会雑 誌 Vol.15 No 3 1992 “ Trends and lssues of Qualitative(Evaluative)Research of Nursing in the United States" FiGURE FIGURE MEASUREMENT OF QUALITY OF CARE EXAMPLES OF CLINICAL INDICATORS TO MEASURE QUALITY OF CARE O STRUCTURE:ORGANIZAT10N AND FINANC‐ ING OF CARE O PROCESS: NURSING TECHNICAL ASPECTS OF O PHYSICAL ASSESSMENT/NURSING HISTORY CARE INTERPERSONAL INCLUDES ALL BODY SYSTEヽ ASPECTS OF CARE TIFIES HEALTH CARE NEEDS 0 0UTOCOME: IMPACT OF PRACTICE ON lS AND IDEN‐ O VITAL SIGNS WITHIN NORMAL LIMITS PATIENT/CLIENT MONITOR VITAL SIGNS EVERY 15 MINUTES POSTOPERATIVE. DOCUMENTAT10N IS REQUIRED IN ALL QUALITY ASSU膵 O LAB WORK WITHIN NORMAL LIMITS OR REFERRD TO PHYSICIAN ANCE PROGRAMS ACTIVE PARTICIPAT10N OF THE CARE CIVERS IS ALSO REQUIRED HEALTH CARE PROVIDERS O PATIENT MEETS DISCHARGE CRITERIA PER POLICY SELECT ASPECTS OF CARE THAT ARE IMPORTANT AND NEED TO BE MONITORED THESE ASPECTS ARE KNOWN O DOCUMENTAT10N PER PROCEDURE AS ・ NDICATORS OF OUALITY・ O PERIOPERATIVE TEACHING AND DIS― WHICH ARE REVIEWED CHARGE PLANNING IMPLEMENTED TO DETERMINE WHETHER THE STANDARD OF CARE IS BEING MET O SAFE AND ASEPTIC ENVIRONMENT PRO‐ VIDED As you see, these components 、 vill appear over and over again― ―structure, process and O SPONGE NEEDLE,BLADE,INSTRUMENT COUNTS CORRECT outcome9)Again, our key v/ord, and that is O SURGICAL PROCEDURES PERFORMED IN 、″ here 、ハ re are the weakest at the moment ithe ACCORDANCE TO INSTITUT10N CRITERIA Outcome ln a1l of this,just as at the museum I visited yesterday, a1l of the artifacts and all of the history presented there v′ O ALL EQUIPMENT AND SUPPLIES PREPARED TO PREVENT UNNECESSARY DELAYS IN SURGERY ould have little O NURSING CARE PLAN FORMULATED TO meaning 、 ′ ithout documentation So in Order MEET THE CHANGING NEEDS OF THE to really move and get into this are of Out― PATIENT comes, we need to obtain documentation ヽ Ve need one thing also l hope you'1l re― O NO VERBALIZED DISSATISFACT10N FROM PATIENT/FAヽ IILY mernber, that we need to involve the consumers of care in obtaining thls documentation、 Very often we make serious mistakes in Our country in not involving consumers of care, 、 /ho are very knoM′ ledgeable today about their health SO uRCE: FOWLER, ME (1990) TRENDS OUALITY ASSURANCE: DEFINIT10N AND IMPORTANCE PLASTIC SURGICAL NURSING,SUMMER.lo(2)196 needs and who really want tO have a say in their health care Figure 4 gives you a fev′ examples of clinical indicators tO measure Quality Of Care NOM′, I put these up here because they'd be famlliar to you All v′ e've tried to do here is to give some order to them To bring them 日本看護研 究学会雑誌 V01.15 No 3 1992 “ Trends and lssues of Qualitative(Evaluative)Research of Nursing in the United States" into some grouping But many of these are WHEN AND HOW TO CONTACT STAFF familiar to you and already been involved IN CASE OF PROBLEM OR EMERGENCY So there's nothing new So you shouldn't be Ⅳ HEALTH PROMOTION afraid or cOncerned about clinical indicators A PATIENT WILL DEMONSTRATE UNDER― Of Quality of Care See this is essential to get STANDING OF BREAST SELF― to Outcome WIeasures And l won't go through a1l of these EXAMI NATION But you're farniliar with the B PATIENT WILL IDENTIFY WARNING physical assessment, patient needs, and so SIGNS OF CANCER forth These are quite familiar C PATIENT WILL KEEP SCHEDULED FOLLOW― UP APPOINTMENTS FIGURE 5 V PATIENT COUNSELING EXAMPLES OF PATIENT OUTCOME INDICATORS A PATIENT WILL IDENTIFY HOSPITAL IN AMBULATORY CARE SETTINGS AND COMMUNITY RESOURCES I NURSING PROCESS B PATIENT WILL STATE REALISTIC OR A THE PATIENT EXPRESSES SATISFAC― T10N WITH NURSING CARE ACCOMPLISHABLE GOALS C PATIENT WILL COMPLY WITH RECOM_ B THE PATIENT DEMONSTRATES RESO‐ LUTION OF COMPLAINT OR PROBLEM Ⅱ USE OF STANDARD CARE PLANS MENDATIONS AND INSTRUCT10NS Ⅵ PATIENT SAFETY A PATIENTS WILL BE FREE FROM IN」 A THE PATIENT DEMONSTRATES AGREE‐ MENT WITH PLAN OF CARE U― RIES OR FALLS B PATIENTS WILL RECEIVE CORRECT B ALL PREOPERATIVE AMBULATORY SURGERY PATIENTS WILL DEMON― DRUG AND DOSAGE C PATIENTS WILL NOT SHOW SIGNS OF STRATE REDUCED ANXIETY AFTER INFILTRATION OR EXTRAVASAT10N FROM A CHEMOTHERAPY DRUG C SEDATED PATIENTS ヽ ハ ′ ILL BE MONI TORED ACCORDING TO CLINIC STAN― SO uRCE: DARD OPERATING PROCEDURE GATES, R A AND PRZYKUCKL」 I PATIENT EDUCAT10N M (1989) IMPROVING PRACTICE IN AMBULATORY CARE i DEVELOPMENT OF A THE PATIENT OR CARE GIVER VERB― ALIZES UNDERSTANDING OF DIAG― AN AMBULATORY NURSlNG OA PROGRAM JOURNAL OF NURSING OUALITY ASSURANCE′ 3(4) NOSIS AND OPT10NS FOR THERAPY CONSISTENT WITH THEIR INTELLEC― TUAL AND EMOTIONAL STATE B THE PATIENT OR CAREGIVER IDEN― TIFIES SIGNS/SYMPTOMS TO REPORT I hear examples of outcome indicators in ambulatory care settings Again, familiar to you so you should feel very comfortable about understanding this You have the nursing process The pa― TO HEALTH CARE TEAM THE PATIENT OR CAREGIVER DEMON― STRATES UNDERSTANDING OF CLINIC OPERAT10N, FUNCTIONS OF PERSON― NEL, APPOINTMENT SYSTEM, AND tient expresses satisfaction with nursing care, which is very ilnportant As you can see the patient satisfaction is very irnportant Use of standard care plans ― there is a lot of change in relation to standard care plans And you'11 日本看護研究学会雑誌 Vol.15 No 3 1992 “ Trends and lssues of Qualitative(Evaluative)Research of Nursing in the United States" find that there's rnore focus on certain aspects you measure that? of that And the figure is continued up there l won't take tirne to go into all of that l do wantto mentiOn one category here: Health Promotion I never forget ― I just love teaching students and l spent about 5 years at Yale l do feel we don't put University teaching undergraduate students abOut principles of nursing And I'1l never enough emphasis on Health PrOmotion and forget 127 1 was a very young instructor at disease prevention And in any standard of the time and l was very proud to be able to care, any plan that you develop, you need to do this But when l did some extensive graduate include that Mre know frOm some early Surgeon 、/ork in physiology and some of the sciences, General's repOrts that have been involved in I realized everything l taught was wrong And that there are many conditions v′hich we our― l spent the rest of my career really trying to selves have control over That v′ e can do som← undo those and challenge those principles thing about Because v′ e didn't have Outcome Measures For example, alcohol consumption, You see, if v′ e had Outcome トノ Ieasures, smoking, weight gain, blood pressure, you I wOuldn't have fallen into that trap But I know, many conditions that we really can dO 、 vasn't alone something about And Health Promotion, of who also fell into that course, is very key And as l mentioned ear― There were many colleagues Now the ideal measurement of Quality lier, this is sOmething 、ァ hich a standard care is M″ hat we call the patient's quality of life, plan needs to address and the patient may v′ hich might be quite different than our per― not be as excited about getting invOlved in that ception for that patient OUTCOMES 一 MEASUREMENT 一 THE KEY 丁0 0 U A L I T A 丁 │ ∨E R E S E A R C H : lt's irnportant, /here 、 monies for health are very tight, because we need to think about Let me move on quickly because 、 re want the patient and the Quality Of Life in terms t o ¨・ I w a n t n o w t o t a l k a b o u t t h i s w o r d ` O u t ― Of whO will get the most for the dollar The comes Measurement' It's the key to Qualitative Federal Government or the State, the local Research lt's the Key government providing some of that, the pa― ― About 30 years ago, I 、 ras chastised for tient whO gets that care, or we as providers― using the term `nursing diagnosis' Today, health care providers it's quite common in the States v′here vre talk ヽ Vhat assurance we have that those Outcome lvleasures are really about nursing intervention, nursing diagnOsis viewed in a way that we will get, as we say related to that Outcomes Measures refer tO colloquially, the “ biggest bang for the buck" any measurement system used tO uncover Or Data Needed; identify the health outcome or treatment for that patient or client Now data As l mentioned, this 、 vhole problem of Quality is hindered because Of the Currently, for many years with emphasis lack of documentation and data. And I'd like On Quantitative Research,we really didn't move to talk a little about that too far because the Qualitative Research The crucial need is for reliable and use― relied pretty heavily on mortality, morbidity ful data There is heavy reliance on quantita― data, but not really on the tough ansv′ ers tive data and not on the qualitative lt's dif_ Sirnply put, `what is the impact of what we ficult tO measure Qualitative data ln fact, do as nurses upon the patient' And hov′ do v/hen we publish this 3rd edition of our bOok 日本看護研究学会雑誌 Vol.15 No.3 1992 “ Trends and lssues of Qualitative(Evaluative)Research of Nursing in the United States" (and I'm nOt selling this book, this is a cere― I didn't go up and say “ Well, I need so many monial b00k), we've put our emphasis there more hundreds of nurses, so many hundreds On Quantitative― not as much on Qualitative of physicians, dentists lt's the number game So now I'm involved in moving in the other is a trap direction TOwards Qualitative Research And So what l did was to turn that around the most important method in relation to and point Out that the problems in relation Qualitative Research is to obtain that data to fetal alcoholism(this is where the mother through clinical trials who drinks even one coctail can damage her And you say “ iuSt fOr drugs" Oh, clinical trials! That's very serious problern among American lndians Not so Not so The only way we can really document ― I shouldn't say the only way but most effective way to document Outcome ヽ fetus), the prOblems of diabetes, which is a leasures, is through clinical and Native Alaskans, alcoholisnl, dental care So what l did was to move away from numbers and talk about results, outcomes, documented trials where you introduce an intervention, it What would happen if this care was not you measure that, you deflne it, you quantify provided to this population? Diabetes, alco― it holism, drug abuse, a1l of that So that the And now as nurse researchers, it's not totally easy, but we have more access to pati― health policy makers, not interested in num― ents and clients for research which makes this bers, or nursing theory lt doesn't mean that possible so that we, again, need to remember we need that but we need to publish and to involve the consumer communicate that theory in a way that policy Every advisory committee l appoint I always・ ¨we'd make sure l had a consumer represented on that group makers can understand and say “ I knoM′ what you mean¨ Oh yeah, ・ that's important… understand that l understand about diabetes Now another lesson I've learned in my l understand about teeth falling out l under― many years in the bureaucracy of the Federal stand about mothers who drink alcohol and Government is the need to translate research the fetal alcoholism" findings in a way that policy makers can understand what you're saying That is why it's so important those of you nurse researchers who publish, to translate l fight all the tilne with nurse theorists because they say “ ・ I I'm a nurse theorist Don't bother me I'm nOt going tO translate this" what you do in terms of the policy maker can understand what you say W e a l s o ¨・ the Data that we need We need If you don't translate yOur nurse theory or involvement of the family and caregivers your findings into ways in which a policy could go on and on but time is running out maker, the decisiOn maker, can understand and this you'1l find in the paper l list really that ― ― you're wasting your time Because I almOst 2 pages of kind of data that we need today cost drives Quality of Care, rationing to collect The assurance that there's a true of care, 、 /hat care will be provided There's commitment at a1l levels of care Now ever since Florence Nightingale¨・ I no way you can escape it l learned this lesson l used to have to was reading this morning with the iet lag…・ defend the health budget befOre Congress for I had a manuscript to review on Florence the American lndian and the Alaskan Natives Nightingale And even in her time she was And l learned that before going to Congress talking about Quality of Care and would you 日本 看護研究学会雑誌 Vol.15 No 3 1992 “ Trends and lssues of Qualitative(Evaluative)Research of Nursing in the United States" believe it, OutcOme A/1easures? She 、 ′ as one my best ideas for research from the bedside of Our first health statisticians in the world nurse or the nurse in the home So you don't And she again was looking at Outcome S/1eas― haveto have a PhD,well we need all the PhDs, ures but you do need to have an inquiring mind She didn't call it that She called it Standards Of Practice about what is the research question What is Our American Nurses Association issued Vhat is this patient saying to you? ilnportant? ヽ a social policy statement in 1980 with the A/1eans listening to that patient And you can (four)characteristics of nursing(:phenomena, pick up some fantastic research ideas! theory applicatiOn, nursing action, and evalua― I used to teach research methods and tiOn Of effect in relatiOn to phenomena) Six my students were very unhappy because they years later, the ANA Board developed a clas― 、/ould spend about 90% of their time develop― sification Of diagnosislOl ing statistical tables and beautiful charts and NO、′, historically, Outcome X/1easures color and everything And l said l don't want are ernbedded in several classic studies Nttany that ヽVhat l v/ant to knov′ is what is the Of yOu you're familiar with research question V/hat is it・that yOu really at Bellviev′ , studied the effects on premature 、′ ant to? ヽ Vhat are yOu asking? And l said “ I'd rather you spend 909イ On defining the One, HasselmeyeriD, who was a nurse infants of a diaper role for support And we question and 10% on the statistics" But they k n O v ′ t h a t i n f a n t s 、/ h O h a v e s o m e k i n d o f v′ eren't quite convinced of that support can be quite effective think as they groM′ 。lder they'1l realize that Dora Schwartz10 at New York COrnell― research nursing on the social― psychological aspects in relation to ambulatory patients A/1yrtle Aydelotte131 eXamined the patient welfare and outcome of nursing care And this v′as 25 years ago, beginning to emerge Although I that is One approach Again because of time, I'In not going tO go into ali the kinds of data and measure― ments that you need They're in the paper and you'1l find that there l want tO move quickly now to Out― in the literature And then l had my self in― comes lプIeasurements related to evaluation volved in the patient welfare and also the pa― feel nurse researchers have the opportunity to 1つ lD tient satisfactioni。 1'1° generate nev/ scientific knowledge and lmprove The carly beginnings of Outcome N/1easures I the utilization process N o w y o u w i l l f i n d i n t h e l i t e r a t u r e ( n ov、 ) a much more interest in classificatiOn systems in relation to Outcorne Measures So that if you have related Outcome ノ ゝ leasures of care, the next thing you need to do is to build the classificatiOn system so yOu AGENCY FOR HEALTH CARE POLICY AND RESEARCH(AHCPR): I M′ ant tO mention briefly, to me l think is one Of the most exciting things that's happened in our Federal Government A nev′ ―is the agency for agency has been formed ― can group them in a way that are manageable health care pOlicy and research (AHCPR) Irs Qualitative Research Needed; one of the 8 agencies of the United States Qualitative Research We need to measure specific aspects of care and the determinance of satisfactiOn Public Health Service V/e don't have a ヽ linister of Health in the United States We have what v′ e call a Department of l want tO mention here that I've gotten Health and Human Resources The Health part 日本看護研究学会雑誌 Vol.15 No 3 1992 “ Trends and lssues of Qualitative(Evaluative)Research of Nursing in the United States" is the public health service The Other half is fracture, on HIヽ た, on、 lyou name it There are 、′ elfare, social security So this agency is part several of these But we feel that by developing of Public Health Service And one of the things these basic Standards Of practice and again that's very exciting about it that we have a looking at need, that this is a better approach number of nurses on the advisory council than saying, “ we're running Out of money・ Linda Akin is on that advisory council and l understand she'1l be coming to visit let's ration care", which l think is a cope― out in terms of doing that 」apan in September Norma Lang, I mentioned earlier, v′ ho's done a great deal of 、 ′ ork on This agency l mentioned has a number of grants which they give tO Universities and OutcOme lvleasures, and there's several other there are several nursing schools applying for nurses that And l feel that on this horizOn, they SO we've been very excited about this ¨ provide(s)an Opportunity for nursing tO really agency because the agency got involved because do some exciting research in getting Out― the some states like Oregon and California come Measures said “we're going broke… ・ 、 re can't cover health A very practical part of doing this is care…・ 、/e're going to ration health care'' 一 that it v/ill lead to some decision about direct which is very scary because who will fall below reimbursement of care l don't knOw if that's the net and not get care? So this agency is an issue in 」 apan By that, I mean instead of looking at alternative approaches to providing the physician being paid for everything then health care and guess what? Outcome llleasures, the nurse not getting her share, this would Quality of Care And I'rn excited about it be― permit direct reimbursement for that portion cause several of the panels inv。 l ve nurse leader of care provided by the nurses)hospitals, ship And nurses are key They're working, homes, so forth The physicians don't like it they knoM′ M′ hat the standards are, they know but it's very exciting But again, in order to what the data and… ・ but they do need help in get Over that horizon, we must collect the getting that and assessment tools l persOnally have been involved in one of the panels in developing Standards of Care data about Outcome X/1easures and put a cost factor So that researchers need to look at cost in relation t0 0ne area of a longterm care, You say you're not interested in cost Only …v′ e estimated some interested in theory Not so Today, in this incontinence We found・ cost data that the Federal Government, our world, you need to look at(link cost to)that. government, spends 13 billion dollars a year, It's a 、′ ay of communicating the need and federal monies, just on the area of incontinence selling that need So that is an exciting agency And what v/e did¨ ・ are¨・ we have a panel look― And you'1l find more about that in the paper ing at all the literature in relation to inconti―O U A L ! T A T I V E R E S E A R C H A P P L I E D T O nence, all the various techniques that are avail―N ∪ R S I N G E D U C A T ! O N : able And the panel will come up with basic A qualitative research can be applied to Standards Of Care that need to be provided nursing education, to clinical settings And these panels are interdisciplinary ―physi nursing education, there are such things called And cians, pharmacists, nurses, urologist, surgeons, p e r f o r m a n c e e x a m i n a t i o n s b a c c a l a u r e a t e p r o ― all v′ orking together v′ hich is very exciting And there'd be various panels on hip grams, using simulated techniques191 You could use an interactive video201 You could 日本看護 研究学会雑誌 Vol.15 No 3 1992 “ Trends and lssues of Qualitative(Evaluative) Research of Nursing in the United States" SUREMENT use a set―up unit on the floor And again, you would evaluate the whole situation using Out― Now data care needs for quality and corne ゝ/1easures care monitoring and outcome measurements Results Certainly a lot better than some lVIc Cormick20 1ists, and I've included this in hich of the performance documentation which doesn't my paper, an integrative data system v′ mean too much today But this M′ uses a predeterrnined framev7ork Of care, valid ould really be performance assessment right at the patient's and reliable data sets, standardized and uni― side And with the technology available, inter― form data sets And these can be done for the v/hole country You don't have to do it iust active video, various or even simulation tech― niques, makes a lot possible And there are… ・ for One setting You can have various sub― I included in my reference a couple of examples systems of business management and clinical that you might、 management of care and identify the various /ant to look up QUALITATIVE RESEARCH APPL!ED TO elements C L I N I C A L S i t t U A 丁! O N S : FUttURE RESEARCH AND THE CHALLENGE In clinical situations, a v′ ritten nursing OF THE 1990s care plan's tO continue to be useful to teach Now let's look at future research and undergraduate students But are not essential the challenge of the 1990s Certainly the 1990's in guiding professional nursing practice nor will seek increased emphasis on the development assuring quality(patient)Outcomes Of Quality Assurance systems So any research So there is a lnovement in our country that you do you would be one step ahead if to move a、 ′ ay from the traditiOnal nursing yOu focused on the Quality NIeasures, develop― care plan v′ ith much more emphasis on look― ment of outcome measures, to measure the ing at 、 vhat is it that ve're 、 putting in proboress irnpact of our interventions notes I― Io、′can M′e avoid redundancy in recOrd― ing? Hov′ can we look at terms of documen― Now hospitals are iOint accrediting which is a joint accrediting hospital associatiOn, tation, of what's important tο the patient and vhich is the crediting body for hospitals 、 record significant observations So that 、/e're Again, has used the foresight to include nurses 100king at this in a different way rather than on their accrediting body and their emphasis iuSt in Our recording progress notes there, interesting enough, Outcome leasures, lν Ve need to assure quality patient Out― ヽ comes by including the physician and nurse and Quality Care And they have listed some 10 step monitoring and evaluation process20 other health disciplines in this collaborative FIGURE 6 practice We need to make sure that services are delivered on time, that there is a support THE TEN‐ STEP MONITORING AND unit, and explore ne、 v unit services The time is long overdue for cost―effec― EVALUAT10N PROCESS l ASSIGN RESPONSIBILITY FOR MONITORING tiveness and nursing intervention and nursing AND EVALUAT10N ACTIVITIES diagnosis202020 Yes nursing diagnosis, nursing 2 DELINEATE THE SCOPE OF CARE PROVI DED interventions, and related to cost, and linkage BY THE ORGANIZAT10N there DAttA CARE NEEDS FOR QUALITY OF 3 1DENTIFY THE MOST IMPORTANT AS‐ PECTS OF CARE PROVIDED BY ORGANI CARE MONITORING AND OUttCOME MEA― 日本看 護研究学 会雑誌 Vol.15 No 3 1992 “ Trends and lssues of Qualitative(Evaluative)Research of Nursing in the United States" SUMMARY: ZAT10N 4 1DENTIFY INDICATORS (AND APPRO‐ NOw in summary Because we do want PRIATE CLINICAL CRITERIA)FOR MONI‐ to have a little time fOr questions TORING THE IMPORTANT ASPECTS OF The 1990's hopefully will see a much CARE more research of indicators and thresholds for 5 ESTABLISH THRESHOLDS FOR THE INDI_ evaluation lmportant that 、 vith all the com― CATORS TO TRIGGER EVALUAT10N OF puter technology, and l must confess having THE CARE had two secretaries for a number of years 6 MONITOR THE IMPORTANT ASPECTS OF and then when l retired in November '89, I CARE BY COLLECTING AND ORGANIZING DATA FOR EACH INDICATOR didn't have any secretary l spent two days learningヽVORD PERFECT and DOSS So now 7 EVALUATE CARE WHEN THRESHOLDS l have my computer, I'm computer literate ARE REACHED TO IDENTIFY PROBLEMS Not all the way but I'rn getting there OR OPPORTUNITIES FOR IMPROVEMENT and so l have great respect for the technology 8 TAKE ACTION TO CORRECT IDENTIFIED and it's a lot of fun But we need to provide PROBLEMS OR IMPROVE CARE this linkage and v/ith the opportunities through 9 ASSESS THE EFFECTIVENESS OF THE AC― TIONS AND DOCUMENT THE IMPROVE‐ computers and NATA analysis, we can link one data base with the other and this is really MENT IN CARE 10 COMMUNICATE THE RESULTS OF THE an ёxciting period to move in that direction A qualitative research, insuring that the MONITORING AND EVALUAT10N PROCESS TO RELEVANT INDIVIDUALS, DEPART― MENTS,OR SERVICE AND TO THE ORGANI‐ patient outcomes must meet accepted standards will help to assure an improved level of care for patients I'd like to finish 、 vith a quote ZAT10NWIDE QA PROGRAM from who else but F10rence Nightingale: Once said, “ No system can endure that SO uRCE: 」01NT COMMISS10N FOR ACCREDITAT10N OF HEALTHCARE ORGAN ZAT10NS OVERVIEW OF ouALITY ASSURANCE AND MONITORING AND EVALUAT10N FOR OBSTETRICS AND GYNECOLOGY CHICAGO:」 01NT COMMISS10N′ 1988′ does not march" (“ March"meaning mov― ing forward as a step at a time) She asked, “ Are we walking into the future, or are we going to remain in the past?'' We are still at the threshold of nursing P ,2 because v′ e as the caring profession can iden― But again, the importance of this is that tify these Outcome Measures And most impoⅢ this」 oint cOmmission which is listedtant, downit's there as a reference, is the ioint accrediting body for all hospitals in Our country So if what's important to the patient and Quality of Care Let us march together to accomplish this there's acceptance there, you kno、 / that the goal of providing Quality Of Care for everyone emphasis on Outcome lVIeasures will be very Sayonara important 日本看護研究学会雑誌 Vol.15 No.3 1992 “ Trends and lssues of Qualitative(Evaluative)Research of Nursing in the United States" 16 Abdellah,FG,et al(1960)Pα REFERENCES l COyne,W (1990)Nurses Are the Key to Quality Puta, D F (1989) Nurse― んθ s ι Oハ7zrsjれ prοαι g New york:ヽ Iacrnillan CO 17 HOver,」 and Zimmer,M (1978)Nursing Quality Health Care RA″ , February, 69-74 2 Physician Collaboration Assurance:The WiscOnsin system ハ raηcο , 3(2)111‐18 18 Mc Cormick, KA (1988) Urinary lncontinence in 3 Mnls,w (1990)Is Quanty Assurance Dead?■ ■]ヽ1 2g Cιれjcs o/Nο ´ the Elderly (Ed)ini rん θⅣ2rs」 ι ん 46(9): 11-12 スれorjOαPhiladelphia:W B Saunders Co 23(1) 4 Donabedian,A (1985)Twenty Years of Research 19 Lenberg, C B and ヽ litchel, CA (1991) Assessment ι われ on the Quality of Medical Care 1965 1984 Eυ αttα of Outcome:The Design and use of Real and Simu‐ αれdιん。■rο れS, 8 :243-265 αι ι んPro/essjο lation Nursing Performance ExaminatiOns A/“ Mariano, C (1990) Qualitative Research 」 Vしrsれg & f f οαι ι んC α″ 1 1 ( 7 ) : 3 5 4 3 5 9 21 SOvie,MD (1989)clinical Nursing Practices and 7 Fowlem M E (1990)Trends Quality Assurance: Patient Outcome:Evaluation, Evolution, and Revo― DefinitiOn and lmportance Pι αsι jοSμrgjο αιⅣαおι πg, lution Ⅳ″ぉれg Ecοれο“Jcs, 7(2):7985 22 Abdelah, FG (1961) CriteriOn Measures h Nursing Summer,10(2) 8 Donabedian, A (1980)E″ Ⅳ″βj″ 26 ακた, 10: 21‐ g Rοsο οれι 」 οれs れ o ″αι jι ρι y Assassれ`″ιαれd』イοれじ ι ο′ jれ g,VOl l: rん οDe/1″jι jο れ 23 Holzemer,WL (1990)Ⅳ Arbor,Ml:Health Administrative Press ″rsj′ αι ι た Cα″,11 τ &∬ θ 415 (8): 412‐ as ι 0/Q″ α“ι οαcん οjι s 4ss′ssれ0″ιAnn y α2ご スρρ′ 24 Brooten, D,et al(1986)A RandOmizes Clinical 1う」 d, p 13 Trial of Early Hospital DLscharge and Home Fo1low‐ 10 Griffith,NH (1989)Historical Perspectives and Up of Very Low Birth Weight lnfants AIθ ω Eュgわれ ご 」0レ乃唸 jル θ, 315:934_939 αιo/]イ`ごjο ANA Policies ln ANA(Ed):Classification System for Describing Nursing Practlce:ヽ 「 orking Papers 25 Mc COrmick, KA (1991) Future Data Needs for Kansas City,MO:ANA,pp 4_5 1l Hasselmeyer,EG (1961)Bο ′ sj″ g ι ι んCαに,12(2):68‐ 74 &∬ oα 20 」bjご, p 71 6 fbjご , p 357 9 /“ rsjル ι οοた g Oン ι 26 : 242-248 Toward Quality Jο “′ παιo/ハ ″ jι ″′ sJ″ γ A SS2_ g o“ αι 5 ι ルれιCοれι ′ ″ご スp― Quality of Care mOnitoring, DRG Considerations, んαυjο ′Pαι ′ れso/P″ ― ι ′ Reimbursement and outcome Measurement F′ th Service Publication No 840 Government Printing ηαgο: 」ο″れα′o/Ⅳ “κjれ αrsん g Scんοι ゎ,23(1):2932 れαι ″″ [Ssれ ι S Washington, DC" U S Pubhc Heaト 26 LOwer,MS,and Burton,S(1989)Measuring the Office lmpact of Nursing lnterventions on Patlent Outtomes F Nαrstag Q“αι the Challenge of the 1990s 」oこ′ ら αιο ● り 12 Schwartz,D Henley,B Zeitz,L (1964)『 んo Eι 滋′ ι y ` 「 スれb″ι πι:」 2g αれd Psyttο― sο ο ttι ハJθ ο 法 αι ο′ μrs力 γ Pα `Jο αれι O, 4(1):27‐ 34 Jι ノAss“′ New YOrk:Macmillan Publishing Co 13 Aydelotte,M (1962)The Use Of Paient Werare as 14 a CriteriOn ヽleasure ハ″ “´ sjれ g ROSOα rcん 11 : 10‐ 14 Abdellah, F G and Levine, E (1957 ; revised 1964) 0UEST10NS ANSWERS l ln 1981, the I)RG influenced the approach of nursing Here in」 apan, we seem to be Pα`jθ ιだ αた Pubhc Health Service れι s α2ご P′rsοれれθ わο examining the DRG intrOduction Or simi― PublicatiOn No 527 V′ ashington, D C U S Govern_ lar system So 、 ment Printlng Offlce the DRG on the nursing? /hat kind of influence of (Hattori) 15 Abdellah, FG, and Levine, E (1957) Developing a Measure of Patient and Personnel Satisfaction with Good question Nursing Care Ⅳ “rsれg Resοα´ αcん 5(3) l rnention the Diagnositc Related Cate― 日本看護研究学会雑誌 V。1.15 No 3 1992 “ Trends and lssues of Qualitative(Evaluative)Research of Nursing in the United States" But that is an Outcome measure But as gorles The big prOblern there in terms of nurs_ l rnentiOned in my earlier¨ ・ to the patient, the ing is that the nursing components were not patient looks at patient and quality of care identified some、 vhat differently than M′ e do as health professiOnals The patient v′ ants to know, And M″ ith the big push now for getting outcome measures, v′ e hope that each of the wants to be comfortable, to be free of pain, 467 Diagnostc cOnditions can have a nursing to be able to go home, to be able to return to component M/ithOut that defined then what the comm■ unity happens to nursing is that nursing gets put looking at a much broader goal in terms of under housekeeping, under dietary, and other what brOught that patient here, 、 related duties health promotion, disease preventionr environ_ And for years, v/e have been suffering under that umbrella So what the effort is now Ours includes those but we're ′ hat is the mental factors What's the theory and basis and part of is to really define outcorne measures for each that sO that there are several aspects in rela― of these 467 conditions So there is a specific tion tO that But you need to develop a com― nursing component and there is a cost factOr mon definition of patient satisfactiOn from the related to that Yes there have been sorne stu― patientノ /Client point of view, v/hich is pa― dies and a fev/ 1've mentioned in my paper tient and quality of life But we need a lot more to really move ahead lf we had had in 1981 the outcOme mea― sures for Diagnostic Related GrOups, we v/ould be a lot further ahead But, v′ And then from the health care providers point Of viev′ , what are some of the broader parameters that the patient may not neces― e didn't have them And at that point in '81 our em_ sarily be involved Or interested in, prevention, disease prevention, and that kind of thing (I phasis was on Quantitative Research and not don't know whether I've answered that or not Qualitative Research So it's an opportunity M/e can chat a little bit later if not) and you nurse researchers really need to move I misunderstood l thought you were in that・…take advantage of that talking about the patient 2 satisfactiOn and particularly one Outstanding There have been many studies about nurse 1vleasurements of trying to evaluate and assess the nursing the satisfaction on the part of care provider HOM′ do you think any specific 、/ay of trying to accomlnodate One at Beth_Israel Hospital in Boston, which has been published v′ idely And the thing that comes out very much the satisfaction of care provider so that in terms of nurse satisfaction is the access we will be able to improve the qtality Of to decision― making , the access to managed the care 、′ e can provide? Is there any re― care, the access to budget determination, a search going on in the United States? voice in the whole management structure And (Hattori) v′ e speak about empo、 ′ erment and governance These are v″ ords that are kicked around but The patient satisfaction is an Outcome measure Of all IIny publications l guess l get more re― essentially that's what nurses are seeking N/1uch more emphasis occurs on that in quests for the work we did in developing a terms of salary The salary's important, but patient satisfactiOn tool in terms of individual satisfactiOn and pride 日本看護研究学会雑誌 Vol.15 No 3 1992 “ Trends and lssues of Qualitative(Evaluative)Research of Nursing in the United States'' and what they do,the emphasis is on involve― into the comrnunity care field therefore I ment, decision―making, a voice in the whole had to suspend my research career But I collaborative plan of care And yes there are believe nO、 ′the significance of your early several studies that have documented this and research has been taken up as the current as a result there are some encouraging changes issue right now The level of satisfaction But the Beth_Israel one in BOston is a classic that you defined 30 years ago in the 1960's one, which they have been able, a marvelous and also your cOncept might have still model, they've been able to accomplish the useability in spite of the fact that the (I hope that answers number two) medical involvement has changed greatly The point is made that if the nurse is (Dr Kinoshita) satisfied then hOpefully the patient's care v′ ill be improved Obviously, yes, we hOpe so But Point has been made that our early work on the more we understand outcome measures, patient satisfaction the questionnaire, has been the impact of what we do so we don't iuSt translated intO 」apanese and this research follow a procedure manual but we really that the lady spoke about So glad to have a understand the impact of what v′ e do so we really understand the ilnpact of v′ fan in the audience and another lesson there is that 30 years¨・ 30 years You out― live your don't just follow a procedure manual but we hat we do we can measure this and certainly this 、 /ould enemies and yOu learn to never give up, never give up to bring abOut change. So sometimes provide a great mOre satisfactiOn for both the it does take 30 years and now we are talking nurse and the patient So again it's working about outcomes measures and 30 years ago together, that's why l emphasize so much, to we tried to talk about that too include the¨ ・ don't forget to include the patient, Thank you for sharing that with us and the consumer in the process that you arrive l hope that if someone is interested in using of that They have a voice and can add a that questionnaire now that you might make great deal it available to them That's a good question Thank you 4 3 30 years ago in 1960 you made a great Do you still feel that the usefulness of this type of work still is viable today? researach on the level of satisfaction and (Dr Kinoshita) Professor Umaki of the University of Tokyo School of Nursing and ourselves The patient satisfaction is an outcome measure. made the study on that And we have trans― Absolutely yes lated your checklist into 」 get it published! apanese And we investigated whether if the nursing staff increased then whether the satisfac― tion level increased or not That's、 ′ hat v′ e did a long years ago and you mentioned at the historical part You actually pointed out that you made that research and l dO believe v′hat you have done is a great ac― complishment After that research l went 日本 看護研究学会雑誌 VOl.15 No 3 1992 So pull out your research and
© Copyright 2024 ExpyDoc