Overview of Patent Retrieval Task at NTCIR-3 (2001/4 ~ 2002/10) Organizers: Makoto Iwayama (TIT / Hitachi) Atsushi Fujii (Univ. of Tsukuba / JST) Noriko Kando (NII) Akihiko Takano (NII) Collaboration with: JIPA (Japan Intellectual Property Association) Data provider: PATOLIS Co. What is Patent Retrieval Task - Evaluation workshop for Patent Retrieval - • Evaluating/comparing patent retrieval techniques of participating systems • Investigating evaluation methods of patent retrieval techniques • Constructing/providing test collections for patent retrieval • Providing a forum for research groups of patent retrieval How to Construct Test Collection (Basic Idea) assessors (human experts) search target (doc. collection) system1 search topic system2 relevance judgment search results1 search results2 pooled results pooling runs Test Collection evaluation (recall/precision) relevant docs. Why Patent • Patent information processing is a good test bed of researches on information access – Real task, real user, real information need – Variety of tasks (IR, CLIR, Filtering, Summarization, Text Mining) • Unique document collection which is different from other well-studied document collections – Searching patents may be different from searching news article (in TREC, CLEF) or searching technical abstracts (in NTCIR). Characteristics of Patent Documents • Structured documents – claims, purposes, effects, embodiments, etc. • Unusual style for claims (esp. Japanese) – Many sub-topics in single long sentence • Large variation in document length – The longest patent in our collection has 30,000 unique words ! Characteristics of Patent Documents (cont) • Many technical terms • Many new terms (defined by applicants) • General terms in claims ex. floppy disk → external storage • Classifications – IPC(International Patent Classification) – FI(File Index) and FT(File Forming Term) for Japanese patents Characteristics of Patent Retrieval • Various search purposes – High recall is necessary in some situations – Claim interpretation is necessary in some situations • Difference between industries – Chemical formulas/materials are important in chemistry – Images/diagrams are important in machinery – IPC is primal in some industries, but not in some others (ex. business method patents) Main Task (Technical Survey) non-expert (manager) expert (patent searcher) clipping + reading news articles memo for supplementing, focusing, etc. • patent survey before product development • patents as technical documents (cf. as legal documents) • cross-DB retrieval How to Construct Test Collection (Basic Idea) assessors (human experts) search target (doc. collection) system1 search topic system2 relevance judgment search results1 search results2 pooled results pooling runs evaluation (recall/precision) relevant docs. Example of Search Topic (JA) <TITLE> <ARTICLE> <SUPPLEMENT> <DESCRIPTION> <NARRATIVE> <CONCEPT> <PI> <TOPIC> <NUM>0004</NUM> <LANG>JA</LANG> <PURPOSE>技術動向調査</PURPOSE> <TITLE>バーコードなどの符号を比較し優劣を判定する装置</TITLE> <ARTICLE> <A-DOC> <A-DOCNO>JA-981031179</A-DOCNO> <A-LANG>JA</A-LANG> <A-SECTION>社会</A-SECTION> <A-AE>無</A-AE> <A-WORDS>189</A-WORDS> <A-HEADLINE>エポック社の特許侵害訴訟、バンダイが敗訴－－東京地裁</A-HEADLINE> <A-DATE>1998-10-31</A-DATE> <A-TEXT> カードゲームの特許を侵害されたとして、玩具（がんぐ）製造会社のエポック社がバン ダイに２億６４００万円の損害賠償を求めた訴訟で、東京地裁は３０日、約１億１４００万円の支 払いを命じた。 森義之裁判長は、バンダイが１９９２年７月～９３年３月に製造・販売した小型 ゲーム機「スーパーバーコードウォーズ」のキー操作などの機能について「エポック社が持つ特許 の技術的範囲に属する」と指摘した。</A-TEXT> </A-DOC> </ARTICLE> <SUPPLEMENT>バーコードなどを読み込み、これに基づく数値を比較して勝敗を決定していればよい。 </SUPPLEMENT> <DESCRIPTION>バーコードなどの符号を複数読み込ませ、これら符号に対応する数値を比較するこ とにより、これらの優劣／勝敗の判定を行うことで対戦を行う装置にはどのようなものがあるか。 </DESCRIPTION> <NARRATIVE>「スーパーバーコードウォーズ」とは、小型ゲーム機の一種であり、キャラクターな どが描かれたカードに記録されたバーコードを読み込ませ、プレーヤーが攻撃や防御などのキー操 作を行うことで、半リアルタイムに対戦を行うものである。符号の例としては、バーコードや磁気 コードなどがあるが、これらに限定するものではない。</NARRATIVE> <CONCEPT>符号 バーコード コード 優劣 勝敗 比較 判定</CONCEPT> <PI>PATENT-KKH-G-H01-333373</PI> </TOPIC> Search Topics • 31 Japanese topics created by JIPA • English, Korean and Chinese (simplified and traditional) topics translated from Japanese How to Construct Test Collection (Basic Idea) assessors (human experts) search target (doc. collection) system1 search topic system2 relevance judgment search results1 search results2 pooled results pooling runs evaluation (recall/precision) relevant docs. Patent Collections (from PATOLIS Co.) Type Format Language Years Number of documents Bytes kkh jsh paj Full text Free Japanese 98,99 697,262 abstract SGML-like Japanese 95 - 99 1,706,154 abstract SGML-like English 95 - 99 1,701,339 18,139M 1,883M 2,711M kkh: Publication of unexamined patent applications jsh: JAPIO Patent Abstracts paj: Patent Abstracts Japan Relationships between Patent Collections kkh: (98,99) Full texts with author’s abstracts (in Japanese) Modification of the original abstracts (by JAPIO experts) length normalization around 400 words and term normalization jsh: (95-99) Abstracts (in Japanese) Translation (by JAPIO experts) paj: (95-99) Abstracts (in English) How to Construct Test Collection (Basic Idea) assessors (human experts) search target (doc. collection) system1 search topic system2 relevance judgment search results1 search results2 pooled results pooling runs evaluation (recall/precision) relevant docs. Runs • Mandatory runs – Search topic fields: <ARTICLE> + <SUPPLEMENT> only (i.e. cross-DB) – Search target: full texts (98, 99) – manual or automatic retrieval – mono-lingual or cross-lingual retrieval • Optional runs – TREC-styled ad-hoc runs recommended search from <DESCRIPTION> (+ <NARRATIVE>) How to Construct Test Collection (Basic Idea) assessors (human experts) search target (doc. collection) system1 search topic system2 relevance judgment search results1 search results2 pooled results pooling runs evaluation (recall/precision) relevant docs. Manual Search before Pooling JIPA (Japan Intellectual Property Association) by systems and experts relevant docs. assessors (human experts) by experts only by systems only preliminary search evaluation relevant docs. search results1 original pool systems search results2 pooling skimming relevance judgments skimmed pool relevant docs. Evaluation • Recall/Precision graph • Average precision • R-precision Submitted Runs • 36 runs from 8 groups – 6 groups from Japan, 2 groups from overseas – 5 groups from universities, 3 groups from companies – 20 mandatory runs, 16 optional runs – 5 cross-lingual runs (EJ) from 2 groups Recall/Precision -- mandatory, A -A, mandatory 0.8 0.7 0.6 LAPIN4(A) DTEC1(M) precision 0.5 DOVE4(M) brklypat1(A) 0.4 daikyo(M) SRGDU5(A) 0.3 IFLAB6(A) brklypat3(A,E) 0.2 A: auto M: manual 0.1 E: English topics 0 0 0.1 0.2 0.3 0.4 0.5 recall 0.6 0.7 0.8 0.9 1 Recall/Precision -- mandatory, A+B -A+B, mandatory 0.8 0.7 0.6 LAPIN4(A) DOVE4(M) precision 0.5 DTEC1(M) daikyo(M) 0.4 brklypat1(A) SRGDU3(A) 0.3 IFLAB6(A) brklypat3(A,E) 0.2 A: auto M: manual 0.1 E: English topics 0 0 0.1 0.2 0.3 0.4 0.5 recall 0.6 0.7 0.8 0.9 1 Recall/Precision -- optional, A -A, optional 0.8 0.7 LAPIN1(A,TDNC) brklypat2(A,DN) 0.6 DOVE3(A,DN) DOVE2(A,D) precision 0.5 SRGDU6(A,DN) IFLAB2(A,D) IFLAB3(A,DN) 0.4 IFLAB7(A,T) brklypat4(A,DN,E) 0.3 A: auto M: manual 0.2 T: TITLE D: DESCRIPTION N: NARRATIVE C: CONCEPT 0.1 E: English Topics 0 0 0.1 0.2 0.3 0.4 0.5 recall 0.6 0.7 0.8 0.9 1 Recall/Precision -- optional, A+B -A+B, optional 0.8 0.7 LAPIN1(A,TDNC) DOVE3(A,DN) 0.6 brklypat2(A,DN) DOVE2(A,D) precision 0.5 IFLAB2(A,D) SRGDU4(A,DN) IFLAB4(A,DN) 0.4 IFLAB7(A,T) brklypat4(A,DN,E) 0.3 A: auto M: manual 0.2 T: TITLE D: DESCRIPTION N: NARRATIVE C: CONCEPT 0.1 E: English Topics 0 0 0.1 0.2 0.3 0.4 0.5 recall 0.6 0.7 0.8 0.9 1 Comparable Runs from the same group (brkly) precision 0.8 0.7 ad-hoc run from Japanese 0.6 cross-DB run from Japanese 0.5 ad-hoc run from English 0.4 cross-DB run from English 0.3 0.2 0.1 0 0 0.1 0.2 0.3 0.4 0.5 recall 0.6 0.7 0.8 0.9 1 Some Results • Cross-DB retrieval is more difficult than ad-hoc retrieval • Cross-lingual retrieval (EJ) is more difficult than Mono-lingual retrieval (JJ) • Pseudo relevance feedback may be effective Refer to our SIGIR2003 paper (to appear) for more detail of glass box evaluations Median of Mean Average Precisions 0.8 A A+B 0.7 median of average precisions 0.6 0.5 0.4 0.3 0.2 0.1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 topic ID Human Experts vs. Systems - breakdown of relevant documents 450 BS AS BJ AJ BU AU 400 number of documents 350 300 250 200 150 100 50 0 BS AS BJ AJ BU AU 1 2 3 4 1 34 44 2 0 5 27 2 10 7 2 2 4 0 0 0 7 13 4 4 22 13 6 10 5 6 7 8 9 10 11 12 13 15 43 9 29 7 3 6 15 0 0 0 55 0 1 0 1 57 6 2 0 9 42 10 3 0 47 22 0 0 23 10 17 11 1 173 12 6 0 5 33 7 4 3 29 5 15 12 27 23 18 15 4 101 22 14 15 16 17 6 2 2 15 37 0 2 0 2 8 4 9 10 1 1 2 10 3 1 8 17 5 15 5 topic ID 18 19 20 21 22 5 4 0 1 0 6 9 1 0 1 36 0 5 0 0 11 0 0 2 4 18 2 1 1 0 17 36 4 8 4 23 24 1 7 1 10 2 33 2 108 4 16 4 72 25 15 49 3 12 6 49 26 17 11 16 6 16 12 27 28 29 30 31 26 0 0 1 16 5 35 0 5 15 18 3 0 2 19 7 7 0 1 16 38 7 5 7 4 19 40 6 3 16 Human Experts vs. Systems - recall of the “A” relevant documents experts found Recall of AJ+AU 1 0.9 0.8 0.7 recall 0.6 all mandatory mandatory (auto) 0.5 0.4 0.3 0.2 0.1 0 0 100 200 300 400 500 ranking pooling (rank = 30) 600 700 800 900 1000 Results 2 • Medians of Mean Average Precisions vary topic by topic. • Numbers of relevant documents systems/experts found vary topic by topic. Proposal-based task was also supported • CRL – Using the Diff Command in Patent Documents • TIT – Rhetorical Structure Analysis of Japanese Patent Claims using Cue Phrases Summary • The first evaluation workshop for patent retrieval • Three kinds of patent collections • Technical survey task – Cross-DB / Cross-lingual retrieval • Relevance judgment by expert patent searchers (JIPA) Patent Retrieval Task at NTCIR-4 (currently on “Dry Run”) • Joint project with JIPA • 10 years’ Japanese full texts (5 years’ for search target) • 10 years’ English Abstracts • Invalidity search from claims • Feasibility task for automatically creating “Patent Map”
© Copyright 2024 ExpyDoc
