# Introduction to pynytimes

The New York Times is one of the most trusted news source around the world. All their article metadata is easily available using their API, which is publicly available to everyone (though only for non-commercial use). All this data can be queried using a REST API, however setting it up can be quite time-consuming. This library solves that problem, now you can easily and quickly query the API without having to worry about the specific implementation.

To get started just go to the instructions on the next page.

## Alternative guides

While this is the most torough installation/usage guide, other helpful guides, with their own specific instructions are also available. These might be helpful if a specific part of the documentation is unclear, or you need specific instructions. These are some of the following alternative guides:

* [Berkeley D-Lab: Getting Started with the NYT API](https://dlab.berkeley.edu/news/getting-started-nyt-api) (March 1, 2022; George McIntire)
* [Towards Data Science: New York Times Sentiment Analysis with Tensorflow](https://towardsdatascience.com/nyt-sentiment-analysis-with-tensorflow-7156d77e385e) (January 9, 2022; Anne Bode)
* [pynytimes: (Old, more extensive) README instructions](https://github.com/michadenheijer/pynytimes/tree/0.8.0) (January 4, 2022; Micha den Heijer) 


# Installation

## Default

There are multiple options to install and upgrade pynytimes, but the easiest is by just installing it using `pip` (or `pip3`). Make sure that you are installing pynytimes to the correct Python version.

### Linux and Mac

```bash
pip install --upgrade pynytimes
```

### Windows

```shell
python -m pip install --upgrade pynytimes
```

## Development

You can also install `pynytimes` manually from GitHub itself. This can be done by cloning this repository first, and then installing it using Python. 

*Note: This might install an unreleased version, installation using this method is only advised if you want to modify the code or help maintain this library.*

```bash
git clone https://github.com/michadenheijer/pynytimes.git
cd pynytimes
python setup.py install
```

## Older Python versions

The current version of `pynytimes` only supports the most recent Python versions (3.9, 3.10, and 3.11) however you still might be running older versions of Python. Luckily most of `pynytimes` features are still available. In the table below you can see which version of `pynytimes` still supports your Python version.

| Python version | pynytimes version | Missing features                         |
| -------------- | ----------------- | ---------------------------------------- |
| 3.8            | `0.8.0`           | Some type hings, small bugfixes          |
| 3.7            | `0.7.0`           | Some type hints, small bugfixes          |
| 3.6            | `0.6.1`           | Type hints, small bugfixes, `with` usage |
| 3.5            | `0.4.2`           | Times Tags, no date parsing              |

You can install an older version by `pip install --upgrade pynytimes==0.7.0`.


# Usage

You can easily import this library using:

```python
from pynytimes import NYTAPI
```

Then you can simply add your API key (get your API key from [The New York Times Dev Portal](https://developer.nytimes.com/)):

```python
nyt = NYTAPI("Your API key", parse_dates=True)
```

**Make sure that if you commit your code to GitHub you** [**don't accidentially commit your API key**](https://towardsdatascience.com/how-to-hide-your-api-keys-in-python-fb2e1a61b0a0)**.**

The following parameters can be set, only the `key` parameter is required.

## Parameters

| Variables     | Description                                                                     | Data type                   | Required | Default               |
| ------------- | ------------------------------------------------------------------------------- | --------------------------- | -------- | --------------------- |
| `key`         | The API key from [The New York Times](https://developer.nytimes.com/)           | `str`                       | True     | `None`                |
| `https`       | Use [HTTPS](https://en.wikipedia.org/wiki/HTTPS)                                | `bool`                      | False    | `True`                |
| `session`     | Optionally set your own `request.session`                                       | `requests.sessions.Session` | False    | `requests.Session()`  |
| `backoff`     | Enable [exponential backoff](https://en.wikipedia.org/wiki/Exponential_backoff) | `bool`                      | False    | `True`                |
| `user_agent`  | Set the [User Agent](https://en.wikipedia.org/wiki/User_agent)                  | `str`                       | False    | `pynytimes/[version]` |
| `parse_dates` | Enable the parsing of dates into `datetime.datetime` or `datetime.date` objects | `bool`                      | False    | `False`               |


# Citation

If you use `pynytimes`, citing this repository would be much appriciated. Using BibTeX you can use this citation file:

```latex
@software{Den_Heijer_pynytimes_2023,
    author = {Den Heijer, Micha},
    license = {MIT},
    title = {{pynytimes}},
    url = {https://github.com/michadenheijer/pynytimes},
    version = {[CITATION VERSION]},
    year = {[CITATION YEAR]},
    doi = {10.5281/zenodo.7821090}
}
```

Alternatively you can retrieve [a citation in your preferred format from Zenodo](https://doi.org/10.5281/zenodo.7821090).


# Article Search

Find articles that match search criteria.

## Usage

```python
NYTAPI.article_search(query=None, dates=None, options=None, results=10)
```

### Parameters

| Variables             | Description                                                               | Data type | Required |
| --------------------- | ------------------------------------------------------------------------- | --------- | -------- |
| `query`               | What you want to search for                                               | `str`     | False    |
| `results`             | The amount of results that you want to receive (returns a multiple of 10) | `int`     | False    |
| [`dates`](#dates)     | A dictionary of the dates you'd like the results to be between            | `dict`    | False    |
| [`options`](#options) | A dictionary of additional options                                        | `dict`    | False    |

#### **`dates`**

| Variables | Description                                        | Data type                              | Required |
| --------- | -------------------------------------------------- | -------------------------------------- | -------- |
| `begin`   | Results should be published at or after this date  | `datetime.datetime` or `datetime.date` | False    |
| `end`     | Results should be published at or before this date | `datetime.datetime` or `datetime.date` | False    |

#### **`options`**

| Variables          | Description                                                                                                                                                           | Data type | Required |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | -------- |
| `sort`             | How you want the results to be sorted (`oldest`, `newest` or `relevance`)                                                                                             | `str`     | False    |
| `sources`          | Results should be from one of these sources                                                                                                                           | `list`    | False    |
| `news_desk`        | Results should be from one of these news desks ([valid options](https://github.com/michadenheijer/pynytimes/blob/main/VALID_SEARCH_OPTIONS.md#news-desk-values))      | `list`    | False    |
| `type_of_material` | Results should be from this type of material ([valid options](https://github.com/michadenheijer/pynytimes/blob/main/VALID_SEARCH_OPTIONS.md#type-of-material-values)) | `list`    | False    |
| `section_name`     | Results should be from this section ([valid options](https://github.com/michadenheijer/pynytimes/blob/main/VALID_SEARCH_OPTIONS.md#section-name-values))              | `list`    | False    |
| `subject`          | Results should contain at least one of these keywords                                                                                                                 | `list`    | False    |
| `body`             | Results should contain at least one of the strings in the list in the article text                                                                                    | `list`    | False    |
| `headline`         | Results should contain at least one of the strings in the headline                                                                                                    | `list`    | False    |
| `fq`               | [**Advanced usage: implement your own filter query**](#advanced-usage-filter-queries)                                                                                 | `str`     | False    |

## Example

```python
import datetime

articles = nyt.article_search(
    query = "Obama", # Search for articles about Obama
    results = 30, # Return 30 articles
    # Search for articles in January and February 2019
    dates = {
        "begin": datetime.datetime(2019, 1, 31),
        "end": datetime.datetime(2019, 2, 28)
    },
    options = {
        "sort": "oldest", # Sort by oldest options
        # Return articles from the following four sources
        "sources": [
            "New York Times",
            "AP",
            "Reuters",
            "International Herald Tribune"
        ],
        # Only get information from the Politics desk
        "news_desk": [
            "Politics"
        ],
        # Only return News Analyses
        "type_of_material": [
            "News Analysis"
        ],
        # The article text should contain either the word
        # "Obamacare" or "healthcare"
        "body": [
            "Obamacare",
            "healthcare"
        ],
        # Headline should contain "bill", "costs over", or "victory"
        "headline": [
            "bill",
            "costs over",
            "victory"
        ]
    }
)
```

## Advanced Usage: Filter Queries

The New York Times article search API allows for many more filters, which can be combined much more specifically. You can specify a filter query be specifying the `fq` key of the `options` parameter. This can be done as follows:

```python
# Get all sports articles from New York City
articles = nyt.article_search(options={
    "fq": "news_desk:("Sports") AND glocations:(\"NEW YORK CITY\")"
})

# Get all articles about Obama, but exclude all
# articles from New York City
articles = nyt.article_search(options={
    "fq": "Obama AND -glocations:(\"NEW YORK CITY\")"
})
```

To learn more about all the possibilities of the advanced filter queries, you can visit the [New York Times Developer API reference](https://developer.nytimes.com/docs/articlesearch-product/1/overview).


# Book Reviews

Find New York Times book reviews, you can either search using the author, isbn, or title. 

## Usage

```python
NYTAPI.book_reviews(author=None, isbn=None, title=None)
```

### Parameters

| Variables | Description                       | Data type | Required           |
| --------- | --------------------------------- | --------- | ------------------ |
| `author`  | Reviews of books from this author | `str`     | One of these three |
| `isbn`    | Reviews of books with this ISBN   | `str`     | One of these three |
| `title`   | Reviews of books with this title  | `str`     | One of these three |

## Example

```python
# Get reviews by author (first and last name)
reviews = nyt.book_reviews(author = "George Orwell")

# Get reviews by ISBN
reviews = nyt.book_reviews(isbn = 9780062963673)

# Get book reviews by title
reviews = nyt.book_reviews(title = "Becoming")
```


# Movie Reviews

Get New York Times reviews, you can search using keywords and optionally define other search criteria

## Usage

```python
NYTAPI.movie_reviews(keyword=None, options=None, dates=None)
```

### Parameters

| Variables             | Description                          | Data type | Required |
| --------------------- | ------------------------------------ | --------- | -------- |
| `keyword`             | Reviews of movies with this keyword  | `str`     | False    |
| [`options`](#options) | Dictionary of search options         | `dict`    | False    |
| [`dates`](#dates)     | Dictionary of dates about the review | `dict`    | False    |

#### **`options`**

| Variables      | Description                                                                     | Data type | Required |
| -------------- | ------------------------------------------------------------------------------- | --------- | -------- |
| `order`        | How to sort the results (`by-title`, `by-publication-date`or `by-opening-date`) | `str`     | False    |
| `reviewer`     | Name of the reviewer                                                            | `str`     | False    |
| `critics_pick` | Only return critics' pick if `True`                                             | `bool`    | False    |

#### **`dates`**

| Variables                | Description                                          | Data type           | Required |
| ------------------------ | ---------------------------------------------------- | ------------------- | -------- |
| `opening_date_start`     | Reviews about movies released at or after this date  | `datetime.datetime` | False    |
| `opening_date_end`       | Reviews about movies released at or before this date | `datetime.datetime` | False    |
| `publication_date_start` | Reviews released at or after this date               | `datetime.datetime` | False    |
| `publication_date_end`   | Reviews released at or before this date              | `datetime.datetime` | False    |

## Example

```python
import datetime

reviews = nyt.movie_reviews(
    keyword = "Green Book",
    options = {
        "order": "by-opening-date",
        "reviewer": "A.O. Scott",
        "critics_pick": False
    },
    dates = {
        "opening_date_start": datetime.datetime(2017, 1, 1),
        "opening_date_end": datetime.datetime(2019, 1, 1),
        "publication_date_start": datetime.datetime(2017, 1, 1),
        "publication_date_end": datetime.datetime(2019, 1, 1)
})
```


# Top Stories

Load all the top stories.

## Usage

```python
NYTAPI.top_stories(section=None)
```

### Parameters

| Variables | Description                             | Data type | Required |
| --------- | --------------------------------------- | --------- | -------- |
| `section` | Get Top Stories from a specific section | `str`     | False    |

The possible sections are: `arts`, `automobiles`, `books`, `business`, `fashion`, `food`, `health`, `home`, `insider`, `magazine`, `movies`, `national`, `nyregion`, `obituaries`, `opinion`, `politics`, `realestate`, `science`, `sports`, `sundayreview`, `technology`, `theater`, `tmagazine`, `travel`, `upshot`, and `world`.

## Example

```python
top_stories = nyt.top_stories()

# Get all the top stories from a specific category
top_science_stories = nyt.top_stories(section = "science")
```


# Most Viewed

Load the most viewed articles.

## Usage

```python
NYTAPI.most_viewed(days=1)
```

### Parameters

| Variables | Description                                                  | Data type | Required | Default |
| --------- | ------------------------------------------------------------ | --------- | -------- | ------- |
| `days`    | Get most viewed articles over the last `1`, `7` or `30` days | `int`     | False    | `1`     |

## Example

```python
most_viewed = nyt.most_viewed()

# Get most viewed articles of last 7 or 30 days
most_viewed = nyt.most_viewed(days = 7)
most_viewed = nyt.most_viewed(days = 30)
```


# Most Shared

Load most shared articles.

## Usage

```
NYTAPI.most_shared(days=1, method="email")
```

### Parameters

| Variables | Description                                                  | Data type | Required | Default   |
| --------- | ------------------------------------------------------------ | --------- | -------- | --------- |
| `days`    | Get most viewed articles over the last `1`, `7` or `30` days | `int`     | False    | `1`       |
| `method`  | Method of sharing (`email` or `facebook`)                    | `str`     | False    | `"email"` |

## Example

```python
most_shared = nyt.most_shared()

# Get most emailed articles of the last day
most_shared = myt.most_shared(
    days = 1,
    method = "email"
)

# Get most shared articles to Facebook of the last 7 days
most_shared = nyt.most_shared(
    days = 7,
    method = "facebook"    
)

# Get most shared articles to Facebook of the last 30 days
most_shared = nyt.most_shared(
    days = 30,
    method = "facebook"
)
```


# Best Sellers Lists

Load best seller list.

## Usage

```python
NYTAPI.best_sellers_list(name="combined-print-and-e-book-fiction", date=None)
```

### Parameters

| Variables | Description               | Data type           | Required | Default                               |
| --------- | ------------------------- | ------------------- | -------- | ------------------------------------- |
| `name`    | Name of best sellers list | `str`               | False    | `"combined-print-and-e-book-fiction"` |
| `date`    | Date of best sellers list | `datetime.datetime` | False    | Today                                 |

#### `name`

Get all best sellers lists

```
lists = nyt.best_sellers_lists()
```

## Example

```python
# Get fiction best sellers list
books = nyt.best_sellers_list()

# Get non-fiction best sellers list
books = nyt.best_sellers_list(
    name = "combined-print-and-e-book-nonfiction"
)

# Get best sellers lists from other date
import datetime

books = nyt.best_sellers_list(
    name = "combined-print-and-e-book-nonfiction",
    date = datetime.datetime(2019, 1, 1)
)
```


# Article Metadata

Get all metadata from an article.

## Usage

```python
NYTAPI.article_metadata(url)
```

### Parameters

| Variables | Description        | Data type | Required |
| --------- | ------------------ | --------- | -------- |
| `url`     | URL of the article | `str`     | True     |

## Example

```python
metadata = nyt.article_metadata(
    url = "https://www.nytimes.com/2019/10/20/world/middleeast/erdogan-turkey-nuclear-weapons-trump.html"
)
```


# Archive Metadata

Load all metadata from a specific month. *Note: This loads a very large JSON file (\~20 MB), thus likely it takes a while.*

## Usage

```python
NYTAPI.archive_metadata(date)
```

### Parameters

| Variables | Description                       | Data type           | Required |
| --------- | --------------------------------- | ------------------- | -------- |
| `date`    | Date of month of all the metadata | `datetime.datetime` | True     |

## Example

```python
import datetime

data = nyt.archive_metadata(
    date = datetime.datetime(2019, 1, 1)
)
```


# Latest Articles

Load metadata of the latest articles.

## Usage

```python
NYTAPI.latest_articles(source=None, section=None)
```

### Parameters

| Variables             | Description                                 | Data type | Required | Default |
| --------------------- | ------------------------------------------- | --------- | -------- | ------- |
| `source`              | Source of article (`all`, `nyt` and `inyt`) | `str`     | False    | `"all"` |
| [`section`](#section) | Section of articles                         | `str`     | False    |         |

#### `section`

You can find all possible sections using:

```python
sections = nyt.section_list()
```

## Example

```python
latest = nyt.latest_articles(
    source = "nyt",
    section = "books"
)
```


# Tags

Load New York Times tags.

## Usage

```python
NYTAPI.tag_query(query, max_results=20, filter_options=None)
```

### Parameters

| Variables        | Description                | Data type | Required | Default |
| ---------------- | -------------------------- | --------- | -------- | ------- |
| `query`          | Tags you're looking for    | `str`     | True     |         |
| `max_results`    | Maximum results you'd like | `int`     | False    | `20`    |
| `filter_options` | Filter options             | `list`    | False    |         |

## Example

```python
tags = nyt.tag_query(
    "pentagon",
    max_results = 20
)
```


# Close Connection

Optionally you can close the `requests.Session()` connection with the New York Times server. 

```python
nyt.close()
```


# Automatic Connection

If you want to automatically close the connection then usage using the `with` statement is supported.

```python
with NYTAPI("Your API Key", parse_dates=True) as nyt:
    nyt.most_viewed()
```