Add the ability to retrieve the exhaustive list of filterable attributes matching the patterns

[ManyTheFish]

Problem

The new filterable attribute API now provides a way to specify patterns instead of listing all the attributes.
However, there is no way in Meilisearch to retrieve the exhaustive list of fields matching each pattern, and having an exhaustive list would ease the implementation of user facing UIs on the cloud, like filter dropdowns:

The only way to have an exhaustive list of fields is to recompute it on the front side

Potential Solution

A way solve this issue would be to add a new subfield in the /stats route linking the attributes with their features like:

{
  "databaseSize": 447819776,
  "usedDatabaseSize": 196608,
  "lastUpdate": "2019-11-15T11:15:22.092896Z",
  "indexes": {
    "movies": {
      "numberOfDocuments": 19654,
      "numberOfEmbeddedDocuments": 1,
      "numberOfEmbeddings": 1,
      "rawDocumentDbSize": 5320,
      "avgDocumentSize": 92,
      "isIndexing": false,
      "fieldDistribution": {
        "poster": 19654,
        "overview": 19654,
        "title": 19654,
        "id": 19654,
        "release_date": 19654
      },

      "fieldMetadata": {
        "poster": {
          "searchable": { "searchable": true, "weight": 0 },
          "sortable": { "sortable": false },
          "distinct": { "distinct": false },
          "ascDesc": { "ascDesc": false, "order": null },
          "geo": { "geo": false },
          "localized": { "localized": true, "locales": ["eng", "fra"] },
          "filterable": { "facetSearch": true, "filter" { "equality": true, "comparison": false } }
        },
        "overview": {
          "searchable": { "searchable": true, "weight": 0 },
          "sortable": { "sortable": false },
          "distinct": { "distinct": false },
          "ascDesc": { "ascDesc": false, "order": null },
          "geo": { "geo": false },
          "localized": { "localized": true, "locales": ["eng", "fra"] },
          "filterable": { "facetSearch": false, "filter" { "equality": false, "comparison": false } }
        },
        "title": {
          "searchable": { "searchable": true, "weight": 0 },
          "sortable": { "sortable": false },
          "distinct": { "distinct": false },
          "ascDesc": { "ascDesc": false, "order": null },
          "geo": { "geo": false },
          "localized": { "localized": true, "locales": ["eng", "fra"] },
          "filterable": { "facetSearch": false, "filter" { "equality": false, "comparison": false } }
        },
        "id": {
          "searchable": { "searchable": false, "weight": 0 },
          "sortable": { "sortable": false },
          "distinct": { "distinct": false },
          "ascDesc": { "ascDesc": false, "order": null },
          "geo": { "geo": false },
          "localized": { "localized": false, "locales": null },
          "filterable": { "facetSearch": false, "filter" { "equality": false, "comparison": false } }
        },
        "release_date": {
          "searchable": { "searchable": false, "weight": 0 },
          "sortable": { "sortable": true },
          "distinct": { "distinct": false },
          "ascDesc": { "ascDesc": false, "order": null },
          "geo": { "geo": false },
          "localized": { "localized": false, "locales": null },
          "filterable": { "facetSearch": false, "filter" { "equality": true, "comparison": true } }
        }
      }

    }
  }
}

these seems a bit big but we could reduce the size by only printing the "activated" features as below:

{
  "databaseSize": 447819776,
  "usedDatabaseSize": 196608,
  "lastUpdate": "2019-11-15T11:15:22.092896Z",
  "indexes": {
    "movies": {
      "numberOfDocuments": 19654,
      "numberOfEmbeddedDocuments": 1,
      "numberOfEmbeddings": 1,
      "rawDocumentDbSize": 5320,
      "avgDocumentSize": 92,
      "isIndexing": false,
      "fieldDistribution": {
        "poster": 19654,
        "overview": 19654,
        "title": 19654,
        "id": 19654,
        "release_date": 19654
      },

      "fieldMetadata": {
        "poster": {
          "searchable": { "searchable": true, "weight": 0 },
          "localized": { "localized": true, "locales": ["eng", "fra"] },
          "filterable": { "facetSearch": true, "filter" { "equality": true, "comparison": false } }
        },
        "overview": {
          "searchable": { "searchable": true, "weight": 0 },
          "localized": { "localized": true, "locales": ["eng", "fra"] }
        },
        "title": {
          "searchable": { "searchable": true, "weight": 0 },
          "localized": { "localized": true, "locales": ["eng", "fra"] }
        },
        "id": {},
        "release_date": {
          "sortable": { "sortable": true },
          "filterable": { "facetSearch": false, "filter" { "equality": true, "comparison": true } }
        }
      }

    }
  }
}

[dureuill]

Hey!

Super interesting idea, but I feel this should be a separate route.

Also it would be nice to be able to filter by features (e.g. "filterable.filter.comparable = true"), and to have a paginated list. We do have users with about 65k fields. Maybe even to be able to search them, so that a UI can put the beginning of a field and retrieve a list of auto completions

[dureuill]

hey @ManyTheFish, @macraig, we're spawning a usage page about this, we'd need some API work.

1. What should the output be. Do we really need "sortable": {"sortable": true}}
2. We anticipate that we might need some kind of query object to filter, what shape should it take?

https://www.notion.so/meilisearch/Get-field-metadata-usage-2164b06b651f80cfa255c08bf19c21e4

[macraig]

1. I'm not sure if we need it, same as "distinct": { "distinct": false } and "geo": { "geo": false }. @ManyTheFish does it have to do with how the frontend would handle it?