what mechanism does the medline database use to provide consistency in information retrieval?

• Loading metrics

Open Access

Peer-reviewed

Research Article

MEDLINE search retrieval problems: A longitudinal query analysis of five vendor platforms

• C. Sean Burns,
• Tyler Nix,
• Robert K. Shapiro II,
• Jeffrey T. Huber

10

• Published: May 6, 2021
• https://doi.org/x.1371/journal.pone.0234221

Figures

Abstract

This study compared the results of data collected from a longitudinal query analysis of the MEDLINE database hosted on multiple platforms that include PubMed, EBSCOHost, Ovid, ProQuest, and Web of Science. The goal was to identify variations amid the search results on the platforms after decision-making for search query syntax. We devised xx-nine cases of search queries comprised of five semantically equivalent queries per example to search against the v MEDLINE database platforms. We ran our queries monthly for a yr and collected search event count data to observe changes. We found that search results varied considerably depending on MEDLINE platform. Reasons for variations were due to trends in scholarly publication such as publishing private papers online first versus complete bug. Some other reasons were metadata differences in bibliographic records; differences in the levels of specificity of search fields provided by the platforms and large fluctuations in monthly search results based on the aforementioned query. Database integrity and currency bug were observed as each platform updated its MEDLINE information throughout the year. Specific biomedical bibliographic databases are used to inform clinical decision-making, create systematic reviews, and construct noesis bases for clinical decision support systems. They serve as essential data retrieval and discovery tools to help identify and collect research data and are used in a wide range of fields and as the basis of multiple research designs. This study should help clinicians, researchers, librarians, informationists, and others understand how these platforms differ and inform future piece of work in their standardization.

Commendation: Burns CS, Nix T, Shapiro RM II, Huber JT (2021) MEDLINE search retrieval issues: A longitudinal query analysis of 5 vendor platforms. PLoS One sixteen(v): e0234221. https://doi.org/ten.1371/periodical.pone.0234221

Editor: Muhammad A. Z. Mughal, King Fahd University of Petroleum & Minerals, Kingdom of saudi arabia

Received: May 19, 2020; Accepted: March 28, 2021; Published: May 6, 2021

Copyright: © 2021 Burns et al. This is an open access commodity distributed under the terms of the Artistic Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: The data underlying the results presented in the study are available from https://github.com/cseanburns/medline-longitudinal.

Funding: This project was supported by a 2018 Summer Faculty Research Fellowship from the Higher of Advice and Data, University of Kentucky. The authors received support in the form of i twelvemonth of access to MEDLINE on Ovid from Wolters Kluwer (https://www.ovid.com/). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: Wolters Kluwer provided a i-year subscription to OVID MEDLINE. This was used to collect the COVID-xix data but not the longitudinal information. This does not alter our adherence to PLOS I policies on sharing information and materials. Wolters Kluwer or its representatives take not been involved in this study or offered any comments on information technology.

Introduction

Bibliographic databases are used to identify and collect enquiry papers and role as a disquisitional office of scientific investigation [1]. Studies that employ bibliographic databases include research on information literacy, bibliometrics/scientometrics, information seeking, systematic reviews, and meta-analyses [2]. In item, PubMed and MEDLINE are used to inform clinical conclusion-making in the health professions [3] and to construct knowledge bases for clinical decision support systems [4].

Research on search queries that inform bibliographic database development or on how queries influence information retrieval results were once mutual lines of inquiry, [five] but these studies have subsided in contempo decades [6]. Search query inquiry has largely shifted abroad from a Boolean model of information retrieval and has focused on ranked-based keyword systems [vii] or on database coverage [viii–11].

Researchers, librarians, information scientists, and others rely on bibliographic databases to acquit research, instruct future information and other professionals on how to bear literature searches, and assist those with information needs to locate and admission literature [12–15]. Furthermore, these databases accept standard rules to describe enquiry papers, and are structured using controlled vocabularies, in guild to brand searching for information more than precise and comprehensive [sixteen, 17].

Fine control over bibliographic searching and documentation of search strategies, which are reported in systematic reviews, allow for the replication and reproduction of searches. In the broader scientific community, the replication and reproduction of enquiry, or lack thereof, has garnered increased attention recently [18, xix]. Additional scrutiny has been given to the replication of prior studies [20]. This is true for systematic reviews and other research that rely on commendation or bibliographic records, but in this domain, the evaluation of scientific rigor is centered around the reproducibility of search strategies. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Guidelines [21] and the Cochrane Handbook for Systematic Reviews of Interventions provide examples for scholars in their reporting of methods and organizing of reviews [22].

Dissimilar general search engines, bibliographic databases, such equally those bachelor on EBSCOhost, ProQuest, Spider web of Science, Scopus, Ovid platforms, and others are designed to utilize structured bibliographic records instead of full-text sources to create search indexes. These bibliographic records comprise fields for providing discovery and access, such every bit author proper name fields, document championship fields, publication championship fields, and appointment of publication fields [16]. In sure specialized databases, these records are often supported past a set of thesaurus terms, or a controlled vocabulary, such as the Medical Subject Headings (MeSH). Their goal is to depict the subject matter of works or records that are indexed in the bibliographic database to assist in consequent and predictable information retrieval.

Controlled vocabularies and thesauri are meant to provide users with a high level of control over the search and retrieval procedure [23, 24] and may exist bachelor across multiple platforms or interfaces from different vendors [25]. The MeSH thesaurus is freely available on the U.S. National Library of Medicine's (NLM) PubMed website, and is used to search MEDLINE on other platforms such as EBSCOhost, Ovid, ProQuest, and Web of Science. These commercial vendors provide access to bibliographic data from MEDLINE, and the respective MeSH thesaurus, and add features on their respective platforms beyond what NLM has already provided. The added features are based upon the providers' unique user interface, search features, ability to link to library collections via proxies, related additional database content, or searching multiple databases on a specific platform in a single search session.

However, these added features may create some differences in searching and in the search results [25, 26]. For case, MEDLINE can be searched using PubMed, which is defined by the nearly 6000 publication titles it indexes, structure of its bibliographic records, use of MeSH in those records, and its freely available search interface on the web. When a vendor provides access to MEDLINE, they kickoff with the MEDLINE system merely create a customized subscription-based version that includes a different interface, slightly different search fields, search operators, and other features.

The differences amidst platforms have been recognized every bit important in the systematic review literature in the biomedical sciences, and the forthcoming PRISMA-S Search Reporting Extension recommends that systematic reviewers study which platform, interface, or vendor is used for each database searched [27]. Still, the implications in how bibliographic records are queried on different platforms are not well understood [28, 29]. For example, fifty-fifty though PubMed/MEDLINE, ProQuest/MEDLINE, EBSCOhost/MEDLINE, Ovid/MEDLINE, and Web of Science/MEDLINE are congenital on the same MEDLINE database, it is not fully known how the features that are added by these vendors impact search and retrieval on their corresponding platforms. Fifty-fifty when there is some transparency, such equally with PubMed [thirty], these systems are complicated and differences with other systems are non well understood.

Although the choice of platforms impacts potential source coverage, it is not known how searching a single database like MEDLINE only on unlike platforms might impact source coverage. That is, searchers employ different content-based databases to conduct literature searches [31–33] to avoid missing relevant studies [9, 34], but there is no research to show whether it is necessary to search multiple MEDLINE-based databases to prevent missing relevant studies. If it is important to access multiple MEDLINE-based databases to aggrandize source coverage, and so this is important in cases where data from past inquiry is collected, analyzed, and synthesized based on published and/or greyness literature, such as in systematic reviews or meta-analyses [x, 35].

In improver to source coverage, it is important to provide a detailed description of the search strategies that are used to search databases so that others may investigate the quality of a search strategy or replicate it in the future [36]. However, research that supports this only refers to MEDLINE as a source and non to MEDLINE from whatsoever specific vendor. Not distinguishing which MEDLINE platform is searched assumes consistency between MEDLINE platforms: for case, that using MEDLINE on PubMed is equivalent to using MEDLINE on Ovid, EBSCOhost, ProQuest, or Web of Science. This may take ramifications for those researchers leading clinical trials or conducting bench-side research, and who have to rely on published literature and acquit intensive literature searches when systematic reviews on their topic are not bachelor.

Fifty-fifty if search sessions are methodical and well documented, vendor systems ofttimes operate as black boxes (i.east., the technology is non well documented) and it becomes merely possible to infer how different systems operate past comparing multiple implementations [37]. Lilliputian is known almost what bodily principles are applied past platform vendors in indexing bibliographic records or what specific types of algorithms are used to rank results when sorted by system-defined relevance. This is a ordinarily known problem among search engines, but information technology is problematic in bibliographic resource purchased by libraries [38, 39].

Interface, indexing, and retrieval differences likewise impact reproducibility and replication, which are important aspects of the scientific process, prove-based medicine, and the creation of systematic reviews [32, forty–43]. Although the NLM maintains the MEDLINE records and provides free (federally subsidized) access to them through the PubMed website, they also license these records to vendors to host on their ain platforms. Furthermore, although these systems operate from the same MEDLINE data file, database vendors utilise their own indexing technologies and their own search interfaces, and it is possible that these alterations influence different search behaviors and retrieval results sets [44, 45]. This may be problematic if platform differences are non ordinarily understood, communicated in vendor reports, or amongst research team members using them, and if the separate platforms are unable to replicate results based on the same information files that are used across them.

There is little scientific testify that shows whatever discrepancies between MEDLINE platforms. Some studies have tested reproducibility across systems but not across MEDLINE-based systems [34]. Instead, many studies compare some aspect of the database drove or the call back and precision on the retrieval results sets in these systems [46–48]. All the same, the focus is not often on the query syntax used even though this has been highlighted as an important problem. I study investigated variations among different interfaces to the Cumulative Alphabetize for Nursing and Allied Health Literature (CINAHL) database, and reported reproducible search strategies except for queries that contained subject-keyword terms [28]. In our prior newspaper [49] nosotros found that queries searched in MEDLINE across different platforms resulted in search result discrepancies after decision-making for the search query. This paper builds on that work and examines differences over fourth dimension by evaluating longitudinal information, which is a critical factor in replicating bibliographic database search results. Here we ask the following research questions:

1. How practice search results amidst MEDLINE-based bibliographic database platforms vary over time afterwards controlling for search query syntax?
2. What explains the variance amid search results amongst MEDLINE-based bibliographic database platforms later controlling for search query syntax?

To answer these questions, our belittling framework is based on the concepts of methods and results reproducibility [l]. Methods reproducibility is "the ability to implement, as exactly equally possible, the experimental and computational procedures, with the same data and tools, to obtain the aforementioned results" and results reproducibility is "the product of corroborating results in a new written report, having followed the same experimental methods (A New Dictionary for Inquiry Reproducibility section, para. 2). We do non apply the concept of inferential reproducibility in this paper since this pertains to the conclusions that a study makes based on the reproduced methods, and this would largely exist applicative if we investigated the relevance of the results based on an information need rather than, equally we do, focus solely on the reproducible results of search queries and the records produced past executing those queries.

Materials and methods

We designed 29 cases of search strategies. Each case comprised five (5) semantically like or equivalent queries to perform searches in five MEDLINE-based platforms for a full of 145 searches. We nerveless longitudinal information (October 2018—September 2019) for each case later on ii airplane pilot runs in August and September 2018. The five platforms included what is now legacy PubMed/MEDLINE (PMML), which has undergone an interface update that applies new search algorithms [51], ProQuest/MEDLINE (PQML), EBSCOhost/MEDLINE (EHML), Spider web of Scientific discipline/MEDLINE (WSML), and Ovid/MEDLINE (OML). The data is based on search issue counts for each search strategy in the 29 cases and were collected at the mid-point of each month. Fig 1 illustrates the general process we used to collect data over the twelve-month period, and Fig 2 highlights the search parameters and values used in the searches.

Fig ane. Data collection method.

Diagram outlining the enquiry process used to collect information. Each calendar month we ran 145 searches (29 cases) on the 5 MEDLINE platforms using the specified search parameters.

https://doi.org/x.1371/journal.pone.0234221.g001

The search queries in each instance, tested in the pilot studies, were designed to be semantically and logically equivalent to each other per case. Throughout the paper we refer to the semantically equivalent queries as cases. Differences between queries within cases were made only to attach to the query language and syntax required past each MEDLINE platform. Tabular array ane provides an example search for case #x for Oct 2018. Each of the queries in this instance were designed to search each of the v MEDLINE platforms for the MeSH term dementia, chosen, like the other terms, for simplicity and for its representation in MEDLINE, which has 2 MeSH branches (topical and analytical structure of the MeSH tree [52]), to explode the term, and to limit results by publication appointment from 1950 to 2015. The last column reports the number of records that were retrieved for each of the platforms for the month and year that data were nerveless.

We designed our 29 cases using basic search fields to examine search retrieval counts on each of the five MEDLINE platforms; therefore, our queries were designed to exist brusque and logically clear to enhance the comparability of the results. For instance, Table 1 describes a simple MeSH field search, express by publication date range. Despite this simplified search, our prior findings indicated that a broad range of records were retrieved by the unlike platforms, fifty-fifty when MeSH terms were added to the strategies/queries. On this footing, nosotros created 29 cases and so to exam a wide range of basic search strategies. We stress that the queries were not designed to test user relevance or user needs, which may range from simple searches to complex, multi-part queries designed for systematic review inquiry. Rather, our queries were designed to contain some office or combination of the post-obit search fields:

• Keywords
• Specific fields
• MeSH terms with i (branch) tree number
• MeSH terms with more than one (co-operative) tree number
• MeSH terms that were exploded

The values that we chose for our search strategies were based on the following criteria: search terms must be well represented in MEDLINE in guild to retrieve enough records to examine how results might vary across MEDLINE platforms; the specific MeSH terms must include at to the lowest degree ane term with only one co-operative in the MeSH thesaurus and one MeSH term with more one co-operative in guild to examine differences when the terms are exploded. We limited MeSH terms to four that included 2 primary terms, neoplasms and dementia, that were combined with secondary terms in some cases. The secondary MeSH terms included humans and hazard and were searched with the primary MeSH terms with Boolean operators (AND, NOT) in order to compare search count results to queries constructed specifically for each platform that included Boolean processing. The journal titles we used in our queries were called to ensure that results would include records from the titles based on the topics we searched, and therefore nosotros chose well established titles in the biomedical sciences that fit our query topics and these include Cancer Enquiry, JAMA, and Lancet. For those cases that include publication dates, we chose a wide publication date range to retrieve a substantial number of results, and we ended the publication date range to records with publication dates up to and including 2015 so that nosotros could limit the result of records that have been newly added to the platforms. Specifically, nine of our cases included publication date ranges, as demonstrated in Table 1. We added date ranges to control differences amid the platforms. Most queries likewise included at to the lowest degree one Boolean operator. Figs i and 2 show that for each month during data collection, we searched the v MEDLINE platforms using a range of search parameters. For a more than detailed exam, all search queries and search effect counts are included in S1 Information.

To reply our kickoff research question, our analysis is based on a comparing of search result counts in each case and on modified z-scores (yard _i ) based on the range of counts for each platform within each case for the time period. The modified z-score is a version of the standard z-score, just is more robust against outliers [53]. We used the modified z-score to locate deviations effectually results from the PubMed MEDLINE platform instead of the distribution middle (hateful or median) in order to highlight deviations from PubMed. We apply PubMed equally the reference platform since this platform and the MeSH vocabulary are provided by the National Library of Medicine. Otherwise, this is an arbitrary reference point and we could accept chosen any of the other platforms as reference points since they all presumably are based on the aforementioned underlying MEDLINE data. We also define our search result count outliers equally any modified z-score that deviates more than ±iii.v from PubMed, as recommended by Iglewicz and Hoaglin [53]. Nonetheless, statistical outliers and practical or clinical outliers are different issues. The modified z-score volition just highlight the former, just any difference in counts will assist respond how MEDLINE platforms deviate from each other. To answer our 2nd research question, nosotros inspected several records from our cases in gild make initial inferences about reproducibility bug that might be a result of how the databases index bibliographic records or respond to the queries that we designed.

Results and discussion

The data gathered in the bulk of our searches (20 cases of five searches each for a full of 100 queries) did not include limits on publication dates. All the same, we did include date delimited searches for ix cases of v searches each for a total of 45 queries. We used publication engagement limits from 1950–2015 (encounter Table 1 for an instance case) for these search queries. 30-nine of the publication date restricted queries returned dissimilar search results from Oct 2018 to September 2019, indicating potential changes either in the bibliographic records nigh five years afterward the terminal publication or potential changes with the search mechanisms used among the MEDLINE platforms. This discrepancy yielded insights for the differences nosotros found beyond the retrieved records. For queries that included a limit on the publication date, our findings break the notion that the bibliographic universe (the fix of all pertinent published works) is stable years after works have been published. Offset, based on a broad overview (Fig 3), results show that differences amongst MEDLINE platforms mainly prove differences in search issue counts merely no or little differences in trends between the platforms for publications from 2015 or earlier based on queries to the systems in 2018 and later. Second, and in contrast, based on a detailed examination of the aforementioned cases (Figs 4–6), results bear witness noticeable differences in trends between the platforms; that the bibliographic universe expands and contracts by dropping or adding records in 2018 and afterwards for queries with publication dates ending in 2015 or earlier.

Fig iii. Macro view of cases #03–#11.

A macro perspective of search result counts for cases #03-#11 that were restricted by publication date. The individual plots are case by instance and each plot compares the count of retrieved items for the five MEDLINE platforms across the twelve months of data drove. The macro perspective highlights differences among the ranges of records retrieved amidst the platforms in each example but it does not capture the detailed variation or trends that occurs when comparison retrieved results among the platforms.

https://doi.org/ten.1371/periodical.pone.0234221.g003

Fig 4. Micro view of cases #03–#11.

A micro perspective of search result counts for cases #03-#11 that were restricted by publication appointment. Each row displays all five platforms individually within a example to aid adjacent comparison of results. The micro perspective is able to evidence how the platforms vary in detail and trends; most importantly, they highlight different trajectories and differences in how platforms driblet retrieved records even for publications with publication dates ending years before data drove.

https://doi.org/x.1371/journal.pone.0234221.g004

Fig 5. Micro view of cases #03–#11.

A micro perspective of search effect counts for cases #03-#11 that were restricted past publication engagement. Each row displays all five platforms individually inside a case to assistance side by side comparison of results. The micro perspective is able to show how the platforms vary in detail and trends; almost chiefly, they highlight dissimilar trajectories and differences in how platforms drib retrieved records even for publications with publication dates ending years earlier data drove.

https://doi.org/10.1371/journal.pone.0234221.g005

Fig half-dozen. Micro view of cases #03–#11.

A micro perspective of search result counts for cases #03-#xi that were restricted by publication engagement. Each row displays all v platforms individually within a case to assist adjacent comparing of results. The micro perspective is able to show how the platforms vary in detail and trends; nigh importantly, they highlight different trajectories and differences in how platforms driblet retrieved records even for publications with publication dates ending years earlier data collection.

https://doi.org/10.1371/journal.pone.0234221.g006

We prove other database-based issues that hamper the reproduction of search strategies across MEDLINE platforms. These include the lack of comparable search fields in the studied MEDLINE platforms, an inconsistent pattern of results from some MEDLINE platforms that bespeak a break in functionality, and a difference in lag fourth dimension between updates from PubMed/MEDLINE and other platforms in updating the database. External to these systems, we show how digital publishing complicates the bibliographic record, which was primarily designed in a print era. In the post-obit sections, nosotros draw the major themes of these differences among these platforms.

Macro and micro views reveal different trends

We examined the information of the cases and present variations we identified based on 2 categories which we report in this section: an overview (macro) of the data (Fig 4) and a more than granular (or, micro) view of the variations arising in the information (Figs iv–six).

The macro view of the cases restricted past publication appointment (cases #03–#11) mainly signal that at that place are but differences in total search result counts over time amid platforms inside each prepare, for example, betwixt WSML and PMML (Fig three, top left plot) and that the trends are otherwise reliably comparable beyond time on a per-query, per-platform basis (Fig iii). For example, in case #05, which includes a query with a single MeSH term and an all-text keyword, all five platforms returned a range of 377 records in October 2018 and this increased by one to a range of 378 records in September 2019. However, the granular, micro view indicates variation inside the platforms themselves (Figs four–6) and that platforms are not internally reliable across time on a per-query footing—that they differ from each other at dissimilar dates and among each other at whatsoever given engagement. For instance, case #05 (Fig 4) shows unlike search result patterns for each platform over the course of data collection. Even though this query is restricted by publication date, search results for each platform increment and decrease at different points in the year.

Figs 7 and 8 present the modified z-scores and highlight the deviations for all searches compared to PubMed, equally the reference database, per example. Fig 7 includes all cases that were within 3.v± deviations from the range of search counts from PubMed. Fig 8 includes deviations outside that range and that are outliers. The figures highlight deviations not around the statistical centre of the distribution of ranges but around the annual range of search counts from PubMed.

Fig 7 does not include statistical outliers but shows inconsistencies betwixt PMML and the four platforms over the course of the year of information collection. PQML generally returned fewer records than PMML but returned more records for iv cases, including two with publication dates in the queries. However, for WSML, cases with queries with publication dates returned fewer boilerplate records than PMML, and this is the contrary for OML. Fig 8 shows that all four non-PubMed platforms returned outliers, merely out of the sixteen cases nowadays, only three appear more than once (Cases #03, #05, and #18) among all four platforms. Furthermore, several cases returned vastly dissimilar counts of results, compared to PMML. In Case #21, PQML consistently retrieved thousands more records than PMML for a query containing an all-fields keyword search and journal titles.

Online-first versus upshot-based publications impact bibliographic control

To help explain the changes in search results for queries that were restricted by publication dates, we compared query cases, #02 and #04, which were both designed to search for a single MeSH term ("neoplasms"), not exploded, but differed in that example #04 is publication engagement restricted. Hypothetically, searches for MeSH terms should non be impacted by changes in full-text search algorithms since the search process for a controlled term is based on whether a tape contains the term or not in a well-divers search field. The grand median change over the year in search result counts for query #02 among all five platforms was 16,551 records (max: 17102; min: 15933), indicating the hypothetical annual growth in literature fastened to this term since this query was not restricted by publication date. The grand median change in search results for query #04 among all five platforms was 17 records (max: 70; min: 8). Since this query fix was restricted past publication date, this suggests hypothetical changes that are not related to literature growth just to changes in either the database search indexes or the bibliographic records, four years after publication.

Furthermore, since all platforms reported different search result numbers in this query set up, this indicates that the 5 platforms index different versions of the same MEDLINE file, or that the platforms index the same MEDLINE file differently based not on the MeSH term merely on the publication date field since the growth cannot be explained by differences in how the platforms index MeSH fields. To test this, we traced a record from the query results for case #04 and investigated a top record that was part of the retrieval set for a query that was limited by the publication appointment to 2015 but which the record indicated it was published in 2019.

According to PubMed documentation, the default publication date field [DP] or [PDAT] includes the date of publication for either the electronic or impress version [54]. An investigation of the chosen record from the search results for #04 in the PMML case [55] shows a bibliographic record with a multi-function publication history. The tape indicates information technology was added to PubMed in 2015 merely not formally entered into MEDLINE until 2019 (See: https://www.ncbi.nlm.nih.gov/pubmed/26700484?written report=medline&format=text). On the journal (BMJ) article's web page, there are two versions of the commodity—an "Online First" version for the article that was issued in 2015 (See: https://spcare.bmj.com/content/early on/2015/12/23/bmjspcare-2014-000835.info?versioned=true) and an "online issue" version of the article that was issued in 2019 (Meet: https://spcare.bmj.com/content/9/1/67.info). The periodical article's publication history on its spider web page states that the 2015 version of the commodity is the "Online Start" version, and the 2019 "online upshot" is the version with a publication date for when the article was assigned to a volume and effect, and added to MEDLINE. On the periodical's site, there are thus 2 versions of this article. Withal, on PubMed, there is 1 tape for this commodity with 2 publication dates because of the versioning.

This record for the BMJ article higher up indicates bug with bibliographic control and dependency on journals to maintain bibliographic records that are complicated by two sequences of publications: online publications that precede publications fastened to volume and issue numbers. The latter are complicated past the versioning of articles based on publication history and that include versions prior to their official publication dates when they are assigned to volume and issue numbers and then added to MEDLINE.

This problem with bibliographic control impacts search results across the platforms. The BMJ article described above does not appear in the search results amongst the other 4 MEDLINE platforms for case #04, and this confirms that the PubMed platform captures the electronic publication date by default even though the record was not entered into MEDLINE proper until the outcome publication date. In this case, this was four years after the online publication engagement fifty-fifty though the query was restricted to MEDLINE. The other platforms practice not comport in this way. The ProQuest and Web of Science MEDLINE-based platforms only offering the ability to delimit search past a single publication appointment type, which seems to be defined past the e-publication date. These platforms do not offer the ability to search by other engagement fields. The EBSCOhost and Ovid platforms offer more control over a variety of engagement fields but patently the default publication date field is not inclusive of issue publication dates between these two platforms, similar it is on PubMed.

Reproducibility improved with field searching

Nosotros found that queries in cases that returned well-nigh equivalent search result counts were queries that included additional field specificity, regardless if the queries were restricted by publication date. Case #13 included two MeSH terms, the start one not exploded and the second one exploded, that were connected past one Boolean Not, plus i certificate title term. All five platforms returned results that were inside a range of three records, taking into account the range of results per platform and and then among platforms over the annual period of data collection. This relative consistency beyond platforms was found in other search cases that included additional, specific field searches. For example, in case #eighteen we performed a single author search for an author who was called because they had published in journals indexed by MEDLINE during the middle office of the 20th century. The range of records that were returned varied over the months and numbered within a range of 15 records among the others. However, when a MeSH term was added to the author name search (example #17), chosen because the writer had published papers that had been indexed with the specific MeSH term ("neoplasms"), all five platforms returned the same count of records for all twelve months of data collection.

Database integrity

Aside from issues with bibliographic command due to online versioning, and with differences in indexing, several of the platforms returned results that appeared as outliers, or yielded vastly different search counts, compared to the other queries within their respective cases. The query in case #08 included i MeSH term, on a unmarried branch, exploded, with a publication appointment restriction. PMML returned a range of 219 boosted records across the months. PQML returned a range of 211 records, and OML returned a range of 438 records. Even so, EHML returned a range of 2108 records, and although WSML returned a range of only 11 records for the time menstruum, it also returned an average of 2491138 fewer records than PMML. We could detect no discernible reason for this discrepancy. Cases #10 and #23 both included a single MeSH term, two branches, exploded, and although search result counts were unlike amongst these platforms within these cases, the differences were non as extreme, perhaps so indicating a problem with how WSML explodes unmarried branch MeSH terms.

There were 2 cases where one platform failed to return any results. In cases #24 and #28, WSML returned 0 results across all twelve months, even though the syntax of the queries was correct and one of the queries had returned results in a pilot test but then dropped them in subsequent tests. Additionally, in cases #25, #27, and #29, the WSML search consequence counts were initially within a reasonable range of the other platforms, but and then diverged substantially (Fig 8). For example, in instance #25, WSML returned a maximum of 13021 records and a minimum of 12652 records from October to April. However, the aforementioned query returned a maximum of 629 records and a minimum of 619 records from May to September, indicating a drop of over 12000 records for the aforementioned search query. In case #27, WSML returned search counts that were unlike but comparable to the other 4 databases, simply so in May again, the counts increased by nearly 40000 records and remained within that range until the terminate of information collection. For case #29, the search result counts were again within range of the other 4 databases through April, but then in May and until the end of data collection, the query no longer retrieved any records. The only blueprint amongst these three queries was that the sudden changes in search result counts occurred in May.

Differences in database currency

The time information technology takes to import the overall PubMed file amidst platforms as well impacts retrieval. On Jan 24, 2020, the U.s.a. National Library of Medicine (NLM) recommended an interim search strategy for searching all of PubMed for literature related to the COVID-19 virus [56]. Their initial recommended search query searched all fields for the term 2019-nCoV or for the terms wuhan and coronavirus in the title and abstract fields (1) (ane)

We modified the search strategy to utilise it amongst all platforms and queried PubMed, inclusive of MEDLINE past default, and the other platforms at irregular intervals. Results showed that for the PubMed data file, generally, all of the platforms returned different results for the same query for new literature. Results too included three versions of NLM'southward interface to PubMed. Two versions were for legacy PubMed but show event counts for when records were sorted by Best Match or Most Recent, since legacy PubMed practical ii different sorting algorithms in this version of PubMed [30]. We besides show search results for the new version of PubMed, which does non utilise dissimilar sorting algorithms for Best Match or Near Recent, just which did report different search counts than both legacy PubMed results. As Fig 9 illustrates, these different platforms for PubMed retrieved newly added records at different rates, likely because they received and incorporated the PubMed information file at unlike times (Fig ix).

Fig ix. Search result count differences for COVID-19 related searches beyond PubMed based platforms.

PubMed Legacy Most Recent Sort (PMLMR), PubMed Legacy Best Lucifer Sort (PMLBM), PubMed New (PMN), ProQuest PubMed (PrQPM), EBSCOhost PubMed (EPM), Web of Scientific discipline PubMed (WSPM), Ovid PubMed (OPM).

https://doi.org/10.1371/journal.pone.0234221.g009

Methods and results reproducibility

Overall, we institute several issues that impact the unevenness of search results beyond these platforms and therefore with their use as reproducible instruments in scientific investigation. Due to differences in search fields across MEDLINE platforms, such as with the publication date and the MeSH fields, in developments in publishing, such as online first versions of articles versus volume and issue number versions, in the ability of databases to behave consistently over time, and to differences in updates of the source file across platforms, information technology is difficult to construct queries that perform consistently alike across platforms and to get consistent results beyond them.

Specifically, nosotros institute that queries restricted by publication dates continue to return new records years past the limit on the publication date range. Information from this report provide some explanation for this variance. Kickoff, the growth of "online first" publications has complicated the traditional bibliographic record for periodical articles which relies on the relationship between an private article and its volume and issue. Additionally, although all platforms provide a search field or a manner to limit search results by publication engagement, not all exercise so at the aforementioned level of detail. While elementary publication engagement searching may have been sufficient in a impress just age when there was only ane publication date, information technology is not sufficient when manufactures are published multiple times, via electronic publication dates and via dates with volumes and issues. The implication is that records risk being dropped or added to time-restricted searches.

We establish that queries were more likely to return comparable search result counts when they included multiple and specific field terms, such as queries combining keywords appearing in a journal title or an commodity title, with an author proper name, and a relevant MeSH term (eastward.g., cases #13 and #17). Practically speaking, this indicates that search results may be more compatible across platforms when searching for a known set of articles using a highly specific, multi-faceted search query. Conversely, simple queries using just ane or 2 keywords or MeSH terms appear more than susceptible to pregnant variations across platforms, underscoring the importance of advanced preparation in literature database searching and consultation with trained information professionals. However, future enquiry studies could help identify how more than complex queries might piece of work more reliably across platforms.

Some platforms are not able to handle exploding the MeSH hierarchy similarly (e.g., EBSCOhost and Spider web of Science outliers in case #08), or they drop records from ane calendar month to the adjacent even though the query has non altered. The lack of discernible causes in search result counts over time makes it impossible to adapt for such variance and undermines the trust in using bibliographic databases to inform data-driven decision making.

Our longitudinal study suggested that some differences might be attributed to delays in transferring the MEDLINE file to the vendors, since PubMed updates MEDLINE daily but the other vendors update at unlike times. To test this, we ran a COVID-xix search based on a query provided by the NLM in Jan 2020 and found that there were uneven search result hits for new literature on the COVID-19 pandemic beyond platforms. Although some of the differences in search effect counts might be explained by the previous problems, the main caption here is likely due to delays in receiving and incorporating the PubMed updates to the vendors. This suggests that if researchers need urgent admission to the timely publications, they should exist concerned virtually which version of PubMed they employ to collect information.

Limitations

One limitation of this study is that its comparison of search result counts does not provide whatsoever insight into the consistency of content retrieved across platforms. Without comparing specific papers' inclusion or omission per query, or how duplication affects retrieval, it is hard to illustrate how retrieval inconsistencies might affect clinical care. Therefore, future enquiry should examine the contents of retrieved works to better understand this dynamic.

Some other limitation is that this inquiry focuses on analyzing differences amidst permutations of relatively elementary strategies. Time to come research should compare published systematic review or other advanced search queries to better empathize the range of differentiation betwixt simple and complex searches.

Farther, the strategies adult for this written report were purposive, but not comprehensive. With these search strategies, we intended to reflect many of the basic edifice blocks of searches that users may utilize when conducting research in literature databases. Even so, the number of permutations a search strategy tin accept prevents a comprehensive approach. For example, each term, field, limiter, and combination offers a new chain of outcomes to examine. Consequently, information technology would be benign for future research to include in-depth examinations of each variable across platforms.

Futurity research

In previous work, and in the current study, we highlighted the importance of documenting variances in systematic searches across platforms based on the same data [49]. While systematic reviews often rely on complex combinations of fields and operators that are difficult, if not impossible to map accurately from platform to platform (accept for example, the lack of proximity operators in NLM's PubMed interface), their use in health intendance decision making underscores the importance of examining variances across platforms based on the aforementioned information. If there are variances, why, to what extent, and farther, what impact could those variances have on determination making? For instance, nosotros believe it is of import to know whether justification is needed in choosing 1 MEDLINE-based platform over another when conducting searches and not assume that different MEDLINE-based platforms operate or return the aforementioned content based on semantically equivalent search strategies.

Though this study highlights significant retrieval differences across platforms, the remaining unknowns of how queries influence retrieval mean that offering concrete recommendations for terminate users is even so difficult. All the same, queries using a combination of several metadata fields returned almost equivalent search consequence counts, as compared to queries incorporating fewer metadata fields. Therefore, creating searches that comprise several metadata fields when possible may atomic number 82 to more consequent results than simpler searches. Withal, we must stress that more research is needed in social club to empathise how these databases function or malfunction. We think that the uncertainties that we raise well-nigh these systems, discovered past comparison them to each other, warrant some attending at standardizing the platforms, and that this is important given the critical role that literature retrieved from MEDLINE plays in the health and bioscience fields.

Conclusions

MEDLINE is the National Library of Medicine'south premier bibliographic database that contains more than 26 meg references to journal articles in life sciences with a concentration on biomedicine. The subject scope of MEDLINE is biomedicine and health, broadly defined to encompass those areas of the life sciences, behavioral sciences, chemical sciences, and bioengineering needed by health professionals and others engaged in basic research and clinical care, public health, health policy development, or related educational activities. MEDLINE also covers life sciences vital to biomedical practitioners, researchers, and educators, including aspects of biology, environmental science, marine biology, plant and animal science also as biophysics and chemistry [58].

Multiple vendors utilize the MEDLINE data to offering versions on their own platforms. In this study, we have institute that problems that brand reproducing searches across these platforms difficult, indicating that there is no ane MEDLINE. Given the complexity in how diverse platforms index data fields (for example, automated term mapping, proximity operators, engagement fields, and MeSH explosion), database providers should improve MEDLINE-specific documentation to aid instructors, information professionals, and perhaps to a bottom extent, novice end users, in amend understanding database function and implications of query structure. Since we evidence here that no MEDLINE-based platform is the same as any of the other platforms, specific platforms should be cited in whatever future research that uses one of these platforms for information collection. Alternatively, multiple MEDLINE-based platforms should be used when conducting searches, and all should be cited if multiple ones are used.

Remarkably, these results advise we may be able to level i of the early on critiques of Google Scholar, which was its disability to replicate results from the same search over periods of time [57], on MEDLINE. What followed with research on Google Scholar were several studies recommending confronting using Google Scholar as the sole database for systematic reviews [58, 59]. If this criticism is valid for the MEDLINE platforms, our results may strengthen the recommendation by Cochrane [22] that no single MEDLINE platform should exist considered a sole source. Nosotros add that any platform that is used to gather bibliographic information be specifically cited in futurity research and that researchers enlist the assistance of library and information professionals when a comprehensive literature search is required. We likewise recommend that when multiple platforms are used, that researchers verify results across these platforms, merely further enquiry is needed to better understand generalizable patterns of differences across them, which admittedly may not be possible given constant changes in database technology.

The MEDLINE data is licensed to multiple vendors of information services who provide access to the database on their platforms, such equally EBSCOhost, ProQuest, Ovid, and Web of Scientific discipline. Any of these platforms are used past information specialists, wellness science librarians, medical researchers, and others to carry research, such equally systematic reviews, in the biomedical sciences. Our research examines results based on 29 cases that included 145 advisedly constructed search queries, plus queries related to the COVID-xix pandemic, across these platforms and indicates that these platforms provide uneven access to the literature, and thus depending on the platform used, the validity of research based on the information gathered from them may be affected. Additional research is required to understand other search-related differences amidst these platforms, including differences amid the records that are retrieved, and how they specifically impact research designs like systematic reviews and other biomedical enquiry, and scientific conclusions based on these studies.

Supporting information

References

1. 1. Palmer CL, Cragin MH. Scholarship and disciplinary practices. Annu Rev Inf Sci Technol. 2008;42: 163–212.
   • View Article
   • Google Scholar
2. 2. Sampson M, Barrowman NJ, Moher D, Klassen TP, Pham B, Platt R, et al. Should meta-analysts search Embase in addition to Medline? J Clin Epidemiol. 2003;56: 943–955. pmid:14568625
   • View Article
   • PubMed/NCBI
   • Google Scholar
3. iii. Dunn 1000, Marshall JG, Wells AL, Backus JE. Examining the part of MEDLINE every bit a patient care information resource: an assay of data from the Value of Libraries study. J Med Libr Assoc JMLA. 2017;105: 336. pmid:28983197
   • View Commodity
   • PubMed/NCBI
   • Google Scholar
4. 4. Bezemer T, de Groot MCH, Blasse E, 10 Berg MJ, Kappen TH, Bredenoord AL, et al. A Human being(due east) Gene in Clinical Decision Support Systems. J Med Net Res. 2019;21: e11732. pmid:30888324
   • View Article
   • PubMed/NCBI
   • Google Scholar
5. 5. McCain KW. Sources of polysemy in indexing do: The example of games, experimental in MeSH. Proc Am Soc Inf Sci Technol. 2014;51: 1–four.
   • View Article
   • Google Scholar
6. half dozen. Hou J, Yang X, Chen C. Emerging trends and new developments in information science: a document co-citation analysis (2009–2016). Scientometrics. 2018;115: 869–892.
   • View Article
   • Google Scholar
7. 7. Hjørland B. Classical databases and cognition organization: A case for boolean retrieval and man determination-making during searches. J Assoc Inf Sci Technol. 2015;66: 1559–1575.
   • View Article
   • Google Scholar
8. 8. Beyer FR, Wright K. Tin can we prioritise which databases to search? A case study using a systematic review of frozen shoulder management. Health Inf Libr J. 2013;30: 49–58. pmid:23413793
   • View Article
   • PubMed/NCBI
   • Google Scholar
9. ix. Bramer WM, Giustini D, Kramer BM. Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study. Syst Rev. 2016;5: 39. pmid:26932789
   • View Article
   • PubMed/NCBI
   • Google Scholar
10. x. Rathbone J, Carter M, Hoffmann T, Glasziou P. A comparison of the functioning of vii cardinal bibliographic databases in identifying all relevant systematic reviews of interventions for hypertension. Syst Rev. 2016;5: 27. pmid:26862061
    • View Article
    • PubMed/NCBI
    • Google Scholar
11. 11. Shariff SZ, Sontrop JM, Iansavichus AV, Haynes RB, Weir MA, Gandhi S, et al. Availability of renal literature in six bibliographic databases. Nephrol Dial Transplant Plus. 2012;v: 610–617. pmid:23185693
    • View Article
    • PubMed/NCBI
    • Google Scholar
12. 12. Boruff JT, Thomas A. Integrating testify-based practice and information literacy skills in teaching concrete and occupational therapy students. Health Inf Libr J. 2011;28: 264–272. pmid:22051125
    • View Commodity
    • PubMed/NCBI
    • Google Scholar
13. thirteen. Damarell RA, Tieman JJ. Searching PubMed for a wide subject expanse: how constructive are palliative intendance clinicians in finding the prove in their field? Health Inf Libr J. 2016;33: 49–sixty. pmid:26293160
    • View Article
    • PubMed/NCBI
    • Google Scholar
14. 14. Ondrusek AL, Ren X, Yang C. A Content Assay of Strategies and Tactics Observed Among MLIS Students in an Online Searching Course. J Educ Libr Inf Sci. 2017 [cited 5 Dec 2019].
    • View Article
    • Google Scholar
15. fifteen. Young S, Duffull S. A learning-based approach for performing an in-depth literature search using MEDLINE. J Clin Pharm Ther. 2011;36: 504–512. pmid:21729115
    • View Article
    • PubMed/NCBI
    • Google Scholar
16. 16. Clarke RI. Breaking records: The history of bibliographic records and their influence in conceptualizing bibliographic information. Cat Classif Q. 2015;53: 286–302.
    • View Article
    • Google Scholar
17. 17. Darmoni SJ, Soualmia LF, Letord C, Jaulent M-C, Griffon Northward, Thirion B, et al. Improving data retrieval using Medical Subject field Headings Concepts: a exam case on rare and chronic diseases. J Med Libr Assoc JMLA. 2012;100: 176. pmid:22879806
    • View Commodity
    • PubMed/NCBI
    • Google Scholar
18. xviii. Amrhein V, Korner-Nievergelt F, Roth T. The earth is apartment (p> 0.05): significance thresholds and the crisis of unreplicable research. PeerJ. 2017;five: e3544. pmid:28698825
    • View Article
    • PubMed/NCBI
    • Google Scholar
19. 19. Baker M. 1,500 scientists elevator the hat on reproducibility. Nat News. 2016;533: 452. pmid:27225100
    • View Article
    • PubMed/NCBI
    • Google Scholar
20. 20. Collaboration OS. Estimating the reproducibility of psychological science. Science. 2015;349: aac4716. pmid:26315443
    • View Article
    • PubMed/NCBI
    • Google Scholar
21. 21. Moher D, Liberati A, Tetzlaff J, Altman DG, Group TP. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLOS Med. 2009;6: e1000097. pmid:19621072
    • View Article
    • PubMed/NCBI
    • Google Scholar
22. 22. Cochrane Handbook for Systematic Reviews of Interventions. [cited ii Apr 2020]. /handbook
23. 23. Savoy J. Bibliographic database admission using free-text and controlled vocabulary: an evaluation. Inf Process Manag. 2005;41: 873–890.
    • View Article
    • Google Scholar
24. 24. Tenopir C. Full text database retrieval performance. Online Rev. 1985;9: 149–164.
    • View Article
    • Google Scholar
25. 25. Dunikowski LG. EMBASE and MEDLINE searches. Tin can Fam Medico. 2005;51: 1191. Available: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1479462/ pmid:16190167
    • View Article
    • PubMed/NCBI
    • Google Scholar
26. 26. Bethel A, Rogers Yard. A checklist to assess database-hosting platforms for designing and running searches for systematic reviews. Health Inf Libr J. 2014;31: 43–53. pmid:24751228
    • View Article
    • PubMed/NCBI
    • Google Scholar
27. 27. Rethlefsen M, Ayala AP, Kirtley S, Koffel J, Waffenschmidt South. PRISMA-Due south: PRISMA Search Reporting Extension. 2019 [cited ii April 2020].
    • View Commodity
    • Google Scholar
28. 28. Allison MM. Comparison of cinahl^® via EBSCOhost^®, OVID^®, and ProQuest^®. J Electron Resour Med Libr. 2006;3: 31–fifty.
    • View Article
    • Google Scholar
29. 29. Craven J, Jefferies J, Kendrick J, Nicholls D, Boynton J, Frankish R. A comparison of searching the Cochrane library databases via CRD, Ovid and Wiley: implications for systematic searching and data services. Health Inf Libr J. 2014;31: 54–63. pmid:24751229
    • View Article
    • PubMed/NCBI
    • Google Scholar
30. 30. Updated Algorithm for the PubMed Best Friction match Sort Lodge. https://www.nlm.nih.gov/pubs/techbull/jf17/jf17_pm_best_match_sort.html
31. 31. Lam MT, McDiarmid Yard. Increasing number of databases searched in systematic reviews and meta-analyses between 1994 and 2014. J Med Libr Assoc. 2016;104: 284–289. pmid:27822149
    • View Article
    • PubMed/NCBI
    • Google Scholar
32. 32. Toews LC. Compliance of systematic reviews in veterinary journals with Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) literature search reporting guidelines. J Med Libr Assoc. 2017;105: 233–239. pmid:28670210
    • View Commodity
    • PubMed/NCBI
    • Google Scholar
33. 33. Vassar Thousand, Yerokhin Five, Sinnett PM, Weiher M, Muckelrath H, Carr B, et al. Database selection in systematic reviews: an insight through clinical neurology. Health Inf Libr J. 2017;34: 156–164. pmid:28383159
    • View Article
    • PubMed/NCBI
    • Google Scholar
34. 34. Bramer WM, Giustini D, Kramer BM, Anderson P. The comparative recall of Google Scholar versus PubMed in identical searches for biomedical systematic reviews: a review of searches used in systematic reviews. Syst Rev. 2013;2: 115. pmid:24360284
    • View Commodity
    • PubMed/NCBI
    • Google Scholar
35. 35. Lemeshow AR, Blum RE, Berlin JA, Stoto MA, Colditz GA. Searching ane or two databases was insufficient for meta-analysis of observational studies. J Clin Epidemiol. 2005;58: 867–873. pmid:16085190
    • View Article
    • PubMed/NCBI
    • Google Scholar
36. 36. Yoshii A, Plaut DA, McGraw KA, Anderson MJ, Wellik KE. Assay of the reporting of search strategies in Cochrane systematic reviews. J Med Libr Assoc. 2009;97: 21–29. pmid:19158999
    • View Article
    • PubMed/NCBI
    • Google Scholar
37. 37. Ahmadi Thousand, Sarabi RE, Orak RJ, Bahaadinbeigy M. Information Retrieval in Telemedicine: a Comparative Study on Bibliographic Databases. Acta Inform Medica. 2015;23: 172–176. pmid:26236086
    • View Article
    • PubMed/NCBI
    • Google Scholar
38. 38. Mowshowitz A, Kawaguchi A. Bias on the web. Commun ACM. 2002 [cited 2 Apr 2020]. https://dl.acm.org/doi/abs/x.1145/567498.567527
39. 39. Vaughan Fifty, Thelwall Grand. Search engine coverage bias: evidence and possible causes. Inf Process Manag. 2004;40: 693–707.
    • View Commodity
    • Google Scholar
40. twoscore. Buchanan S, Salako A. Evaluating the usability and usefulness of a digital library. Libr Rev. 2009;58: 638–651.
    • View Article
    • Google Scholar
41. 41. Edwards A, Kelly D, Azzopardi L. The Impact of Query Interface Design on Stress, Workload and Functioning. In: Hanbury A, Kazai Thou, Rauber A, Fuhr N, editors. Advances in Information Retrieval. Springer International Publishing; 2015. pp. 691–702.
42. 42. Ho GJ, Liew SM, Ng CJ, Shunmugam RH, Glasziou P. Development of a Search Strategy for an Testify Based Retrieval Service. PLOS ONE. 2016;11: e0167170. pmid:27935993
    • View Commodity
    • PubMed/NCBI
    • Google Scholar
43. 43. Peng RD. Reproducible research and Biostatistics. Biostatistics. 2009;10: 405–408. pmid:19535325
    • View Article
    • PubMed/NCBI
    • Google Scholar
44. 44. Boddy Thou, Younger P. What a difference an interface makes: merely how reliable are your search results? Focus Altern Complement Ther. 2009;fourteen: five–7.
    • View Commodity
    • Google Scholar
45. 45. Younger P, Boddy K. When is a search non a search? A comparison of searching the AMED complementary health database via EBSCOhost, OVID and DIALOG. Health Inf Libr J. 2009;26: 126–135. pmid:19490151
    • View Commodity
    • PubMed/NCBI
    • Google Scholar
46. 46. Haase A, Follmann 1000, Skipka G, Kirchner H. Developing search strategies for clinical practice guidelines in SUMSearch and Google Scholar and assessing their retrieval performance. BMC Med Res Methodol. 2007;7: 28. pmid:17603909
    • View Article
    • PubMed/NCBI
    • Google Scholar
47. 47. Colina B. Comparison of periodical title coverage between CINAHL and Scopus. J Med Libr Assoc JMLA. 2009;97: 313. pmid:19851497
    • View Commodity
    • PubMed/NCBI
    • Google Scholar
48. 48. Nourbakhsh Due east, Nugent R, Wang H, Cevik C, Nugent Thousand. Medical literature searches: a comparison of PubMed and Google Scholar. Health Inf Libr J. 2012;29: 214–222. pmid:22925384
    • View Article
    • PubMed/NCBI
    • Google Scholar
49. 49. Burns CS, Shapiro R Ii, Naught T, Huber JT. Search results outliers among MEDLINE platforms. J Med Libr Assoc. 2019;107: 364–373. pmid:31258442
    • View Article
    • PubMed/NCBI
    • Google Scholar
50. 50. Goodman SN, Fanelli D, Ioannidis JPA. What does research reproducibility hateful? Sci Transl Med. 2016;8: 341ps12–341ps12. pmid:27252173
    • View Article
    • PubMed/NCBI
    • Google Scholar
51. 51. U.Southward. National Library of Medicine. The New PubMed is Here. U.S. National Library of Medicine; 2020 [cited xv Apr 2020]. https://world wide web.nlm.nih.gov/pubs/techbull/nd19/nd19_pubmed_new.html
52. 52. MeSH Tree Structures. U.S. National Library of Medicine; [cited 13 Aug 2020]. https://web.annal.org/web/20200416182701/https://world wide web.nlm.nih.gov/bsd/disted/meshtutorial/meshtreestructures/alphabetize.html
53. 53. Iglewicz B, Hoaglin DC. How to detect and handle outliers. Milwaukee, Wisc.: ASQC Quality Printing; 1993.
54. 54. Advanced PubMed Searching Resource Packet. U.S. National Library of Medicine; 2017. https://nnlm.gov/sites/default/files/shared/files/trifolds/pubmedpacket.pdf
55. 55. Thomas S, Walsh D, Aktas A. Systematic bias in cancer patient-reported outcomes: symptom 'orphans' and 'champions'. BMJ Support Palliat Care. 2019;9: 67–74. pmid:26700484
    • View Article
    • PubMed/NCBI
    • Google Scholar
56. 56. New MeSH Supplementary Concept Record for the 2019 Novel Coronavirus, Wuhan, China. NLM Technical Bulletin. 2020 Jan–Feb. U.Due south. National Library of Medicine; 2020. https://world wide web.nlm.nih.gov/pubs/techbull/jf20/brief/jf20_mesh_novel_coronavirus.html?utm_medium=email&utm_source=govdelivery
57. 57. Jacsó P. Google Scholar: the pros and the cons. Online Inf Rev. 2005;29: 208–214.
    • View Article
    • Google Scholar
58. 58. Gehanno J-F, Rollin L, Darmoni S. Is the coverage of google scholar enough to be used alone for systematic reviews. BMC Med Inform Decis Mak. 2013;13: 7. pmid:23302542
    • View Article
    • PubMed/NCBI
    • Google Scholar
59. 59. Giustini D, Boulos MNK. Google Scholar is not enough to be used lone for systematic reviews. Online J Public Wellness Inform. 2013;5: 214. pmid:23923099
    • View Article
    • PubMed/NCBI
    • Google Scholar

rebelloased1985.blogspot.com

Source: https://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0234221

Share this post