Text and Data Mining

Wiley encourages innovative use of the content we publish, and supports customers who wish to perform text and data mining (TDM) on Wiley content. We are committed to developing tools and services that will enable subscribers to carry out TDM in the most efficient and effective manner, as well as to providing straightforward access to content for TDM purposes.

Academic subscribers can perform TDM under license (or in accordance with statutory rights under applicable legislation) on subscribed content for non-commercial purposes at no extra cost.

Corporate subscribers should contact their account manager to discuss options available for content access and delivery. Learn more about our options for corporate customers.

In order to maximize platform stability and security for all users, access to content for TDM purposes takes place through our API service, rather than through crawling or scraping Wiley Online Library. Academic subscribers can access subscribed content once they have accepted the Wiley click-through TDM license and received an API token.

Please check with your library or licensing institution to see whether there is a separate Text and Data Mining agreement in place with Wiley before accepting the click-through license. Any such agreement may incorporate terms that vary from our standard click-through license and might supersede the terms of the click-through license. Please note that you will always need to accept the click-through license in order to obtain an API token.

UK users will also need to accept the click-through license in order to obtain an API token, but the terms and conditions of the click-through agreement referring specifically to the activity of Text and Data Mining are superseded by the terms of the Copyright and Rights in Performances (Research, Education, Libraries and Archives) Regulations 2014 covering computational analysis for non-commercial purposes.

Except as expressly permitted by Wiley or by statutory rights under applicable legislation, Wiley reserves all rights to make reproductions and extractions for TDM.

If for any reason access to content through the this service does not meet the requirements of your specific TDM project (for example, you wish to analyse material to which your institution does not subscribe, or you would like to discuss access to other content formats), please email TDM@wiley.com and we will be happy to discuss your requirements with you.

If you have any questions about Wiley’s TDM policy or how the policy works in practice, please email TDM@wiley.com.

If you are having technical problems using the API to access content on Wiley Online Library, please see our technical help documentation below or contact Wiley Customer Services.

  • This overview shows how to access content via the Wiley TDM service using curl as an example. It does not aim to cover the wide variety of software applications available, but the same principles will apply in the majority of cases.

    Wiley requires agreement to the terms and conditions in order to provide access to content for TDM. Once you have accepted the click-through license, then a TDM access token is provided that will enable you to download articles using the service.

    For any inquiries regarding Wiley TDM, please contact: TDM@wiley.com.

    In brief, the process is as follows:

    1. Accept the Wiley terms and conditions and obtain an access token
    2. Identify relevant articles
    3. Download articles.


    1. Accept Wiley Terms and Conditions

    Firstly go to the ‘Get a Text and Data Mining Token’ section on this page to view and accept the terms and conditions. You will need to login as a Wiley user. If you have not already registered with Wiley Online Library, you can go to the registration page to create an account. On acceptance of the license, you will be directed to a page with your access token.

    2. Identify articles you want to analyse

    You will need to identify the DOIs of relevant articles in order to access them using the Crossref Metadata API or other similar services. This process is not covered in this document.

    3. Download articles

    Downloading articles for analysis breaks down into two steps:
    1. Finding the download URL

      You can find the download URLs for a given DOI by checking with Crossref. There is a useful method described in the Crossref TDM documentation (http://tdmsupport.Crossref.org/researchers/, 1 - Fetch the Metadata)
    2. Downloading an article

      Obviously, there are lots of options available for downloading, ranging from freely-available tools to custom-written downloaders. We will illustrate using curl, a commonly-used tool. The basic procedure is to make a request for the download URL of the article, passing your Clickthrough Client token as the value of the Wiley-TDM-Client-Token header. As an example, we'll download the following article:

      Halikiopoulou, D. and Vlandas, T. (2015), The Rise of the Far Right in Debtor and Creditor European Countries: The Case of European Parliament Elections. The Political Quarterly, 86: 279-288. doi:10.1111/1467-923X.12168 Query Crossref metadata for the download link shows that the download URL for this article is: https://api.wiley.com/onlinelibrary/tdm/v1/articles/10.1111%2F1467-923X.12168.

      The command line to use for curl is:

      curl -L -H "Wiley-TDM-Client-Token: xxxxxxxx-xxxxxxxx-xxxxxxxx-xxxxxxxx" \-D 12168-headers.txt\ https://api.wiley.com/onlinelibrary/tdm/v1/articles/10.1111/1467-923X.12168\ -o 12168.pdf

      The options used are:
      -L Follow redirects (this is important because our TDM application will redirect you to a successful request)
      -H Add the Wiley-TDM-Client-Token header to the request.
      -D Write the response headers to a file (useful for debugging purposes)
      -o Write the PDF to a file.

      Assuming you have a license to this article, the PDF should be downloaded and stored in the output file:
      Example article
    Points to note:
    1. Our TDM application always responds initially with a redirect to a server dedicated to the serving of binary resources
    2. We implement rate-limiting at no more than 3 requests per second
    3. Unsuccessful requests should receive one of the errors listed here.

    Error Status Message Remedial Action
    400 No TDM Client Token was found in the request The request which received this error did not contain a Client Token header
    403 TDM Client Token that you have provided is invalid and not registered The Wiley TDM Client API token that was provided in the header is invalid and cannot be validated
    404 An error has occurred. It appears that you or your institution/organization does not have access to the content that you have requested (e.g. through an existing subscription). Please check that you are requesting content for which you have full-text access. If your institution/organization subscribes to the content, you must make the request from their network.  You must have access to the content you wish to download, whether through an institutional license or because the content is open access
    429 Too many requests Our TDM application implements rate-limiting. Please reduce the rate you are calling this API

     

  • Your token enables Wiley to verify that you have agreed to our Terms and Conditions. If you believe your token has been compromised, you should revoke (refresh) it and you will be assigned a new token.
  • NOTE: Please check with your library or licensing institution to see whether there is a separate Text and Data Mining agreement in place with Wiley before accepting this click-through license. Any such agreement may incorporate terms that vary from or supersede our standard click-through license. Please note that you will still need to accept this click-through license in order to obtain an API token.

    UK users will also need to accept the click-through license in order to obtain an API token, but the terms and conditions of this agreement referring specifically to the activity of Text and Data Mining are superseded by the terms of the Copyright and Rights in Performances (Research, Education, Libraries and Archives) Regulations 2014 covering computational analysis for non-commercial purposes.

    John Wiley & Sons, Inc. and its affiliates (‘Wiley’) grants subscribers and other lawful users (‘Authorized Users’) the right to text and data mine subscribed online content for non-commercial purposes. In accepting this Text and Data Mining Agreement (the “Agreement”), Authorized Users consent to the terms and conditions set out below.

    1. Definitions

    ‘Text and Data Mining’ (‘TDM’) as used in this Agreement means any automated computational technique for accessing, extracting, copying, or analytical processing of content subscribed to by Authorized Users or otherwise made available to Authorized Users on Wiley Online Library.  and TDM shall include but not be limited to the identification of entities, structures and relationships within the Wiley Content.

    ‘Text and Data Mining Output’ (‘TDM Output’) means the result of any TDM activity carried out by Authorized Users.

    ‘Wiley Content’ – the electronic content made available (under agreement or otherwise) by Wiley to Authorized Users on Wiley Online Library (or other platforms owned or operated by Wiley).

    2. Text and Data Mining

    During the term of this Agreement, Wiley grants Authorized Users the non-exclusive, non-transferrable right to text and data mine Wiley Content for the purposes of non-commercial, scholarly research related to specific projects. TDM and TDM Output will not be used for direct or indirect commercial purposes without prior consent in writing from Wiley. Except as permitted by this Agreement or by statutory rights under applicable legislation, Wiley reserves all rights to make reproductions and extractions for TDM. By entering this Agreement, Authorized User acknowledges and agrees to Wiley’s reservation of TDM rights.

    Authorized User acknowledges that Wiley Content is protected by copyright and that all right, title and interest in and to Wiley Content remains with Wiley and its licensors and that unauthorized use or redistribution of Wiley Content or the TDM Output would materially harm Wiley and its licensors.

    In the event that certain Wiley Content is made available under a more permissive article-level license, such as a Creative Commons CC BY license, then the terms of the article-level license will apply, and users may use that content in accordance with the article-level permissions. In addition, this Agreement is not intended to restrict any statutory TDM rights held by Authorized Users under applicable legislation.

    3. Security, Grant of Access Rights, Formats and Delivery Mechanisms

    Authorized Users must access Wiley Content for TDM using a Wiley-approved API service and must abide by any rate-limiting conveyed in machine readable form from time to time, and may not bypass the API or disrupt the working of Wiley Online Library. Except in accordance with Wiley’s API requirements (such as load rates and download rates conveyed in machine-readable form from time to time) and the terms and conditions determined by any approved API service, Authorized Users may not use any robots, spiders, crawlers or other automated downloading programs, algorithms or devices to continuously and automatically search, scrape, extract, deep link, or index Wiley Content.

    Authorized Users shall implement and maintain adequate and effective data security systems and measures to safeguard Wiley Content downloaded for TDM, in line with international industry standards and best practice.

    In the event that TDM results in the processing of personal data, Authorized Users shall be responsible for complying with relevant data protection and privacy laws.

    4. TDM Output

    Authorized Users may load and technically format Wiley Content on Authorized Users’ servers in order to enable access to and use of such Wiley Content for TDM purposes by Authorized Users for specific research projects by using automated programs or devices to continuously and automatically:

    • extract and index information such as semantic entities from Wiley Content;
    • mount, load and integrate the results for access and use by User for secure TDM;
    • communicate TDM Output to third parties as part of original non-commercial research carried out by Authorized Users, including in articles that describe, analyse and interpret research. Publications or analyses resulting from TDM of Wiley Content may include brief quotations from the original text as permitted under Section 107 or 108 of the 1976 United States Copyright Act in the United States, or as permitted by other applicable national copyright laws internationally. Any such extracts, as well as bibliographic metadata, must cite the original Wiley Content in the form of a DOI link. Permission to reproduce images shall be required in accordance with clause 5.

    5. Restrictions

    Except as expressly stated in this Agreement or otherwise permitted in writing by Wiley, Authorized Users may not:

    • perform systematic or substantive extracting for the purposes of creating a product or service for use by third parties, or that has the potential to substitute and/or replicate any other existing Wiley product, service and/or solution;
    • create any form of central repository containing Wiley Content except as described above for the purpose of specific TDM projects;
    • make the results of any TDM Output available on an externally facing server or website, except as permitted by clause 4;
    • allow a third party to harvest any Wiley Content or TDM Output to an internal server; extract, develop or use Wiley Content in any direct or indirect commercial activity; abridge, modify, translate or create any derivative work based on Wiley Content, except to the extent necessary to make it perceptible on a computer screen to the User for research purposes; remove, obscure or modify in any way any copyright notices, other notices or disclaimers as they appear in Wiley Content;
    • use any robots, spiders, crawlers or other automated downloading programs, algorithms or devices to continuously and automatically search, scrape, extract, deep link, index or disrupt the working of Wiley Online Library, except as permitted by clause 3; or
    • reproduce any illustrations, including photographs, figures and line drawings, in the TDM Output without the consent of the rights holder (unless permitted under the article level license).

    6. Fee

    Authorized Users shall not be charged an additional fee for TDM, provided the scope remains purely non-commercial. Authorized Users requiring access to non-subscribed Wiley Content should contact their Wiley Account Manager for further information.

    7. Term and Termination

    The term of this Agreement shall commence on the date on which Authorized User accepts the Agreement and shall continue until terminated by either party. The Agreement may be terminated by either party for breach or for insolvency. Authorized Users will be able to access subscription products for TDM if Authorized Users their institution maintains a subscription to such products. Upon completion of any specific TDM project or upon termination of this Agreement for any reason, Authorized User will delete all Wiley Content downloaded for purposes of TDM.

    8. General

    Neither party’s delay or failure to perform any provision of this Agreement as a result of circumstances beyond its control shall be deemed a breach of this Agreement.

    The relationship between the parties does not constitute a partnership, joint venture, or agency relationship, and neither shall have any authority to bind the other in any way.

    Authorized User shall not transfer, license or assign any of Authorized User’s rights or obligations under this Agreement without the prior written consent of Wiley.

    Wiley reserves the right to make changes to the Agreement and its TDM service from time to time and such changes will be clearly posted by Wiley and deemed accepted by Authorized Users.

    Choice of law and venue for this Agreement will be the same as that set forth in the agreement under which Wiley Content is subscribed to or made available to Authorized Users