oam/knowledge base/evidently.md

Evidently

AI observability tool written in Python.

1. TL;DR
2. ML monitoring dashboard
   1. Remote snapshot storage
3. Collector
4. Further readings
   1. Sources

TL;DR

Setup

pip install 'evidently'
pip install 'evidently[llm]' 's3fs' 'tracely'

Usage

evidently ui
evidently ui … --host='0.0.0.0' --port='8080'
FSSPEC_S3_KEY='AKIA2HKHF74L0123ABCD' FSSPEC_S3_SECRET='jlc/m1sO…' evidently ui … --workspace='s3://bucket/prefix'

evidently collector

ML monitoring dashboard

Visualizes ML system performance over time and allows for issues detection.

Available as cloud service or self-hosted.
Evidently Cloud offers extra features (e.g. user authentication and roles, built-in alerting, no-code interface).

Remote snapshot storage

One can store snapshots in a remote data store such as S3 buckets.
The Monitoring UI service will interface with it to read the snapshots' data.

Evidently connects to data stores using fsspec.
It allows accessing data on remote file systems via standard Python interfaces (built-in implementations, other implementations).

pip install 'evidently[llm]' 's3fs'
evidently ui --host='0.0.0.0' --port='8000' --workspace='s3://bucket/prefix'

Collector

evidently collector

Further readings

• Website
• Main repository
• Docker image
• Python
• fsspec
• ML observability course

Sources

• Documentation