---
title: "Discovering Tampa Bay data"
description: "Search the checked catalog and live portals, then interpret and filter the results."
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Discovering Tampa Bay data}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>", eval = FALSE)
```

`tampaBayOpenData` searches a bundled catalog of 52 checked layers or the
live ArcGIS portals of six Tampa Bay publishers. Maintainers have validated
the configured endpoints for checked layers. A `discovered` layer comes from
a portal index and has not received those checks.

Live examples below are displayed but not run while the site is built. The
checked-catalog example runs offline.

## Start offline

```{r checked-catalog, eval=TRUE}
library(tampaBayOpenData)

tampaBayOpenData::tbod_list_portals()[, c("id", "publisher", "jurisdiction")]
tampaBayOpenData::tbod_list_datasets(source = "checked")[,
  c("id", "title", "jurisdiction", "validation_status")]
tampaBayOpenData::tbod_search_datasets(
  "park", source = "checked", spatial = TRUE,
  jurisdiction = c("stpete", "clearwater")
)
tampaBayOpenData::tbod_dataset_info("construction-permits")[
  c("title", "scope_note", "verified")]
```

`source = "checked"` reads the registry without HTTP. IDs such as
`construction-permits` and `stpete-parks` are package names, not ArcGIS item
IDs. `tbod_dataset_info(id)` gives the scope, terms, source URL, and endpoint
verification date; that date says nothing about record freshness. The
[coverage article](coverage.html) has counts by publisher, and the [selection
policy](https://github.com/Jaclenga/tampaBayOpenData/blob/main/docs/checked-catalog-policy.md)
explains which layers enter the catalog.

## Discover live portal layers

```{r live-search}
found <- tbod_search_datasets("housing", source = "live", portals = "city")
found[, c("id", "title", "publisher", "validation_status", "modified")]

# A one-row result carries the source needed for inspection or retrieval.
if (nrow(found)) {
  selected <- found[1, ]
  selected[, c("item_id", "service_url", "layer_id", "portal")]
  tbod_dataset_info(selected)
}
```

By default, `tbod_list_datasets()` and `tbod_search_datasets()` use `source = "all"`:
they search all six portals and include the checked catalog.
`source = "live"` returns only layers found through portal indexes, including
checked layers when a live endpoint matches a checked entry. If a source is not
indexed or an index lags, it may not appear in live results; a compatible public
ArcGIS layer can still be retrieved by URL with `tbod_get_arcgis_layer()`.

Use `portals = "city"` for Tampa, `"tbrpc"` for the regional planning council,
or `"stpete"`, `"clearwater"`, `"hillsborough"`, or `"pinellas"` for the
other publishers. A vector selects several. `tbod_list_portals()` shows their
publisher names and source locations offline. `jurisdiction` defaults to
`"all"` for catalog functions and can be a code or a vector of codes. A
jurisdiction or publisher filter narrows the organizations scanned and filters
checked entries. `portals` only selects live organizations; it does not filter
checked entries when `source = "all"`. Retrieval and download functions default
to `jurisdiction = NULL`: a checked ID resolves across the bundled catalog when
it is unique. If two jurisdictions use the same ID, supply `jurisdiction` to
select one.

When a live endpoint matches a checked entry, the result keeps its package
ID, scope, and terms, combines checked and portal tags and categories, and uses
the portal item's modification time. New IDs have the form
`arcgis:<item-id>:<layer-id>` and can be passed to `tbod_get_dataset()` later.
The item or service can still change or disappear. Retrieval checks query
responses and requires `Query` when the layer advertises a capabilities list.

## Search beyond the bundled publishers

`tbod_discover()` accepts a public ArcGIS sharing REST root, FeatureServer or
MapServer service root, or a layer URL. Portal queries search public service
items; service queries filter layer names. Both return rows accepted by
`tbod_get_dataset()`. Portal results have `discovered` status and stable
`arcgis:` IDs; service results have `not_checked` status and use the layer URL
as their ID. Queryable MapServer layers are supported.

```{r custom-discovery}
# Replace this example address with your portal's public sharing REST root.
portal <- "https://example.maps.arcgis.com/sharing/rest"
other <- tbod_discover(portal, query = "parks", max_items = 10)
if (nrow(other)) {
  data <- tbod_get_dataset(other[1, ], limit = 100)
  # A stable ArcGIS ID can also be resolved later through its own portal.
  same_layer <- tbod_get_dataset(other$id[[1]], portal = portal, limit = 100)
}
```

Custom sources default to an `"unspecified"` jurisdiction. Set `publisher`
and `jurisdiction` in `tbod_discover()` when you know them; these labels then
travel with the discovery row and its retrieval provenance. The `portal`
argument on retrieval applies only to a stable `arcgis:<item-id>:<layer-id>`
ID. When retrieving a direct layer URL, `tbod_get_dataset()` does not accept
a `jurisdiction` override.

## Filter results

Both catalog functions accept these optional filters:

| Filter | Match |
| --- | --- |
| `jurisdiction` | One or more codes from `tbod_list_portals()`; default `"all"`. |
| `publisher` | Exact publisher names from `tbod_list_portals()`, ignoring case. |
| `validation_status` | `"checked"` or `"discovered"`. |
| `spatial` | `TRUE` for geometry layers, `FALSE` for tables; unknown geometry is excluded when filtering. |
| `topic` | Literal substring in tags and categories, ignoring case. |
| `category` | Exact category path or final category label, ignoring case. |
| `modified_after`, `modified_before` | Inclusive bounds on the catalog's `modified` timestamp. |

Character filters accept vectors: values within one filter are alternatives;
separate filters and the search text must all match. Date bounds accept a
`Date` or strict `"YYYY-MM-DD"` string for a whole UTC day, or a `POSIXct`
value for an exact instant. Rows with unknown modification times are excluded
only when a date bound is supplied.

```{r county-search}
county_transport <- tbod_search_datasets(
  "road", source = "live", jurisdiction = "pinellas", spatial = TRUE,
  topic = "transportation", modified_after = as.Date("2025-01-01"),
  max_items = 10
)
county_transport[, c("id", "title", "publisher", "modified", "modified_source")]
```

`modified_source = "portal_item"` identifies a live ArcGIS item timestamp;
`"checked_snapshot"` identifies an upstream timestamp saved in the checked
registry. Neither establishes when every record was last updated. The
separate checked `verified` date records an endpoint check, not feature
freshness.

## Understand scan limits and failures

`max_items` caps **portal service items per organization**, not the number of
expanded layers. The default scans at most 25 items per organization. Use
`max_items = Inf` explicitly for a full live listing, with a suitable
`total_timeout`. A capped listing can miss matching items beyond the scan
limit. `jurisdiction` and `publisher` narrow organizations before scanning;
other filters apply after discovery and the checked overlay. Consequently,
an item matching `topic` or `modified_after` may be missed by a capped scan.

```{r scan-metadata}
results <- tbod_list_datasets(source = "live", portals = "stpete", max_items = 10)
attr(results, "discovery_complete")
attr(results, "discovery_truncated_portals")
attr(results, "discovery_scanned_items")
attr(results, "discovery_issues")
attr(results, "discovery_failed_portals")
```

Check `discovery_complete` before treating results as a complete portal
catalog. The other attributes show scan progress, skipped items, and failed
portals. A combined search retains rows from working publishers or from
earlier pages of a failed scan. If all selected portals fail before returning
rows, `source = "live"` errors; `source = "all"` warns and returns checked
rows.

Continue with [retrieval and downloads](retrieval.html) or read about
[coverage and source limits](coverage.html).