Loading Data Continuously Using Snowpipe¶
Snowpipe is Snowflake’s continuous data ingestion service. Snowpipe loads data within minutes after files are added to a stage and submitted for ingestion. The service provides REST endpoints and uses Snowflake-provided compute resources to load data and retrieve load history reports. The service can load data from any internal (i.e. Snowflake) stage or external stage that points to an Amazon S3 bucket. With Snowpipe’s server-less compute model, Snowflake manages load capacity, ensuring optimal compute resources to meet demand. In short, Snowpipe provides a “pipeline” for loading fresh data in micro-batches as soon as it’s available.