Contributors Guide to the Code¶
Module Overview¶
faust.appDefines the Faust application: configuration, sending messages, etc.
faust.cliCommand-line interface.
faust.exceptionsAll custom exceptions are defined in this module.
faust.modelsModels describe how message keys and values are serialized/deserialized.
faust.sensorsSensors record statistics from a running Faust application.
faust.serializersSerialization using JSON, and codecs for encoding.
faust.storesTable storage: in-memory, RocksDB, etc.
faust.streamsStream and table implementation.
faust.topicsCreating topic descriptions, and tools related to topics.
faust.transportMessage transport implementations, e.g. aiokafka.
faust.typesPublic interface for static typing.
faust.utilsUtilities. Note: This package is not allowed to import from the top-level package.
faust.webWeb abstractions and web applications served by the Faust web server.
faust.windowsWindowing strategies.
faust.workerDeployment helper for faust applications: signal handling, graceful shutdown, etc.
Services¶
Everything in Faust that can be started/stopped and restarted, is a
Service.
Services can start other services, but they can also start asyncio.Task
via self.add_future. These dependencies will be started/stopped/restarted with
the service.
Worker¶
The worker can be used to start a Faust application, and performs tasks like setting up logging, installs signal handlers and debugging tools etc.
App¶
The app configures the Faust instance, and is the entry point for just about everything that happens in a Faust instance. Consuming/Producing messages, starting streams and agents, etc.
The app is usually started by Worker, but can also be started alone if
less operating system interaction is wanted, like if you want to embed Faust
in an application that already sets up signal handling and logging.
Monitor¶
The monitor is a feature-complete sensor that collects statistics about the running instance. The monitor data can be exposed by the web server.
Producer¶
The producer is used to publish messages to Kafka topics, and is started whenever necessary. The App will always starts this when a Faust instance is starting, in anticipation of messages to be produced.
Consumer¶
The Consumer is responsible for consuming messages from Kafka topics, to be
delivered to the streams. It does not actually fetch messages (the
Fetcher services does that), but it handles everything to do with
consumption, like managing topic subscriptions etc.
Agent¶
Agents are also services, and any async function decorated using @app.agent
will start with the app.
Conductor¶
The topic conductor manages topic subscriptions, and forward messages from the Kafka consumer to the streams.
app.stream(topic) will iterate over the topic: aiter(topic).
The conductor feeds messages into that iteration, so the stream
receives messages in the topic:
async for event in stream(event async for event in topic)
TableManager¶
Manages tables, including recovery from changelog and caching table contents. The table manager also starts the tables themselves, and acts as a registry of tables in the Faust instance.
Store¶
Every table has a separate store, the store describes how the table is stored in this instance. It could be stored in-memory (default), or as a RocksDB key/value database if the data set is too big to fit in memory.
