.. _changelog-1.0: ============================== Change history for Faust 1.0 ============================== This document contain change notes for bugfix releases in the Faust 1.x series. If you're looking for changes in the latest series, please visit the latest :ref:`changelog`. For even older releases you can visit the :ref:`history` section. .. _version-1.0.30: 1.0.30 ====== :release-date: 2018-08-15 3:17 P.M PDT :release-by: Ask Solem - **Requirements** + Now depends on :ref:`Mode 1.15.1 `. - **Typing**: :attr:`faust.types.Message.timestamp_type` is now the correct :class:`int`, previously it was string by message. - **Models**: Records can now have recursive fields. For example a tree structure model having a field that refers back to itself: .. sourcecode:: python class Node(faust.Record): data: Any children: List['Node'] - **Models**: A field of type ``List[Model]`` no longer raises an exception if the value provided is :const:`None`. - **Models**: Adds support for ``--strict-optional``-style fields. Previously the following would work: .. sourcecode:: python class Order(Record): account: Account = None The account is considered optional from a typing point of view, but only if the :pypi:`mypy` option ``--strict-optional`` is disabled. Now that ``--strict-optional`` is enabled by default in :pypi:`mypy`, this version adds support for fields such as: .. sourcecode:: python class Order(Record): account: Optional[Account] = None history: Optional[List[OrderStatus]] - **Models**: Class options such as ``isodates``/``include_metadata``/etc. are now inherited from parent class. - **Stream**: Fixed :exc:`NameError` when pushing non-Event value into stream. .. _version-1.0.29: 1.0.29 ====== :release-date: 2018-08-10 5:00 P.M PDT :release-by: Vineet Goel - **Requirements** + Now depends on :pypi:`robinhood-aiokafka` 0.4.18 The coordination routine now ensures the program stops when receiving a :exc:`aiokafka.errors.UnknownError` from the Kafka broker. This leaves recovery up to the supervisor. - **Table**: Fixed hanging at startup/rebalance on Python 3.7 (Issue #134). Workaround for :mod:`asyncio` bug seemingly introduced in Python 3.7, that left the worker hanging at startup when attempting to recover a table without any data. - **Monitor**: More efficient updating of highwater metrics (Issue #139). - **Partition Assignor**: The assignor now compresses the metadata being passed around to all application instances for efficiency and to avoid extreme cases where the metadata is too big. .. _version-1.0.28: 1.0.28 ====== :release-date: 2018-08-08 11:25 P.M PDT :release-by: Vineet Goel - **Monitor**: Adds consumer stats such as last read offsets, last committed offsets and log end offsets to the monitor. Also added to the StatsdMonitor. - **aiokafka**: Changes how topics are created to make it more efficient. We now are smarter about finding kafka cluster controller instead of trial and error. - **Documentation**: Fixed links to Slack and other minor fixes. .. _version-1.0.27: 1.0.27 ====== :release-date: 2018-07-30 04:00 P.M PDT :release-by: Ask Solem - No code changes - Fixed links to documentation in README.rst .. _version-1.0.26: 1.0.26 ====== :release-date: 2018-07-30 08:00 A.M PDT :release-by: Ask Solem - Public release. .. _version-1.0.25: 1.0.25 ====== :release-date: 2018-07-27 12:43 P.M PDT :release-by: Ask Solem - :setting:`stream_publish_on_commit` accidentally disabled by default. This made the rate of producing much slower, as the default buffering settings are not optimized. - The ``App.rebalancing`` flag is now reset after the tables have recovered. .. _version-1.0.24: 1.0.24 ====== :release-date: 2018-07-12 6:54 P.M PDT :release-by: Ask Solem - **Requirements** + Now depends on :pypi:`robinhood-aiokafka` 0.4.17 This fixed an issue where the consumer would be left hanging without a connection to Kafka. .. _version-1.0.23: 1.0.23 ====== :release-date: 2018-07-11 5:00 P.M PDT :release-by: Ask Solem - **Requirements** + Now depends on :pypi:`robinhood-aiokafka` 0.4.16 - Now compatible with Python 3.7. - Setting :setting:`stream_wait_empty` is now disabled by default (Issue #117). - Documentation build now compatible with Python 3.7. - Fixed ``ForwardRef has no attribute __origin__`` error. - Fixed ``DeprecatedInSphinx2.0`` warnings. - **Web**: Adds ``app.on_webserver_init(web)`` callback for ability to serve static files using ``web.add_static``. - **Web**: Adds ``web.add_static(prefix, fs_path)`` - **Worker**: New ``App.unassigned`` attribute is now set if the worker does not have any assigned partitions. - **CLI**: Console colors was disabled by default. .. _version-1.0.22: 1.0.22 ====== :release-date: 2018-06-27 5:35 P.M PDT :release-by: Vineet Goel - **aiokafka**: Timeout for topic creation now wraps entire topic creation. Earlier this timeout was for each individual request. - **testing**: Added stress testing suite. .. _version-1.0.21: 1.0.21 ====== :release-date: 2018-06-27 1:43 P.M PDT :release-by: Ask Solem .. warning:: This changes the package name of ``kafka`` to ``rhkafka``. - **Requirements** + Now depends on :pypi:`robinhood-aiokafka` 0.4.14 + Now depends on :ref:`Mode 1.15.0 `. .. _version-1.0.20: 1.0.20 ====== :release-date: 2018-06-26 2:35 P.M PDT :release-by: Vineet Goel - **Monitor**: Added ``Monitor.count`` to add arbitrary metrics to app monitor. - **Statsd Monitor**: Normalize agent metrics by removing memory address to avoid spamming statsd with thousands of unique metrics per agent. .. _version-1.0.19: 1.0.19 ====== :release-date: 2018-06-25 6:40 P.M PDT :release-by: Vineet Goel - **Assignor**: Fixed crash if initial state of assignment is invalid. This was causing the following error: ``ValueError('Actives and Standbys are disjoint',).`` during partition assignment. .. _version-1.0.18: 1.0.18 ====== :release-date: 2018-06-21 3:53 P.M PDT :release-by: Ask Solem - **Worker**: Fixed ``KeyError: TopicPartition(topic='...', partition=x)`` occurring during rebalance. .. _version-1.0.17: 1.0.17 ====== :release-date: 2018-06-21 3:15 P.M PDT :release-by: Ask Solem - **Requirements** + Now depends on :pypi:`robinhood-aiokafka` 0.4.13 - We now raise an error if the official :pypi:`aiokafka` or :pypi:`kafka-python` is installed. Faust depends on a fork of :pypi:`aiokafka` and can not be installed with the official versions of :pypi:`aiokafka` and :pypi:`kafka-python`. If you have those in requirements, please remove them from your ``virtualenv`` and remove them from requirements. - **Worker**: Fixes hanging in wait_empty. This should also make rebalances faster. - **Worker**: Adds timeout on topic creation. .. _version-1.0.16: 1.0.16 ====== :release-date: 2018-06-19 3:46 P.M PDT :release-by: Ask Solem - **Worker**: :pypi:`aiokafka` create topic request default timeout now set to 20 seconds (previously it was accidentally set to 1000 seconds). - **Worker**: Fixes crash from :exc:`AssertionError` where ``table._revivers`` is an empty list. - **Distribution**: Adds :file:`t/misc/scripts/rebalance/killer-always-same-node.sh`. .. _version-1.0.15: 1.0.15 ====== :release-date: 2018-06-14 7:36 P.M PDT :release-by: Ask Solem - **Requirements** + Now depends on :pypi:`robinhood-aiokafka` 0.4.12 - **Worker**: Fixed problem where worker does not recover after MacBook sleeping and waking up. - **Worker**: Fixed crash that could lead to rebalancing loop. - **Worker**: Removed some noisy errors that weren't really errors. .. _version-1.0.14: 1.0.14 ====== :release-date: 2018-06-13 5:58 P.M PDT :release-by: Ask Solem - **Requirements** + Now depends on :pypi:`robinhood-aiokafka` 0.4.11 - **Worker**: :pypi:`aiokafka`'s heartbeat thread would sometimes keep the worker alive even though the worker was trying to shutdown. An error could have happened many hours ago causing the worker to crash and attempt a shutdown, but then the heartbeat thread kept the worker from terminating. Now the rebalance will check if the worker is stopped and then appropriately stop the heartbeat thread. - **Worker**: Fixed error that caused rebalancing to hang: ``"ValueError: Set of coroutines/Futures is empty."``. - **Worker**: Fixed error "Coroutine x tried to break fence owned by y" This was added as an assertion to see if multiple threads would use the variable at the same time. - **Worker**: Removed logged error "not assigned to topics" now that we automatically recover from non-existing topics. - **Tables**: Ignore :exc:`asyncio.CancelledError` while stopping standbys. - **Distribution**: Added scripts to help stress test rebalancing in :file:`t/misc/scripts/rebalance`. .. _version-1.0.13: 1.0.13 ====== :release-date: 2018-06-12 2:10 P.M PDT :release-by: Ask Solem - **Worker**: The Kafka fetcher service was taking too long to shutdown on rebalance. If this takes longer than the session timeout, it triggers another rebalance, and if it happens repeatedly this will cause the cluster to be in a state of constant rebalancing. Now we use future cancellation to stop the service as fast as possible. - **Worker**: Fetcher was accidentally started too early. This didn't lead to any problems that we know of, but made the start a bit slower than it needs to. - **Worker**: Fixed race condition where partitions were paused while fetching from them. - **Worker**: Fixed theoretical race condition hang if web server started and stopped in quick succession. - **Statsd**: The statsd monitor prematurely initialized the event loop on module import. We had a fix for this, but somehow forgot to remove the "hard coded super" that was set to call: ``Service.__init__(self, **kwargs)``. The class is not even a subclass of Service anymore, and we are lucky it manifests merely when doing something drastic, like ``py.test``, recursively importing all modules in a directory. .. _version-1.0.12: 1.0.12 ====== :release-date: 2018-06-06 1:34 P.M PDT :release-by: Ask Solem - **Requirements** + Now depends on :ref:`Mode 1.14.1 `. - **Worker**: Producer crashing no longer causes the consumer to hang at shutdown while trying to publish attached messages. .. _version-1.0.11: 1.0.11 ====== :release-date: 2018-05-31 16:41 P.M PDT :release-by: Ask Solem - **Requirements** + Now depends on :ref:`Mode 1.13.0 `. + Now depends on :pypi:`robinhood-aiokafka` We have forked :pypi:`aiokafka` to fix some issues. - Now handles missing topics automatically, so you don't have to restart the worker the first time when topics are missing. - Mode now registers as a library having static type annotations. This conforms to :pep:`561` -- a new specification that defines how Python libraries register type stubs to make them available for use with static analyzers like :pypi:`mypy` and :pypi:`pyre-check`. - **Typing**: Faust codebase now passes ``--strict-optional``. - **Settings**: Added new settings - :setting:`broker_heartbeat_interval` - :setting:`broker_session_timeout` - **Aiokafka**: Removes need for consumer partitions lock: this fixes rare deadlock. - **Worker**: Worker no longer hangs for few minutes when there is an error. .. _version-1.0.10: 1.0.10 ====== :release-date: 2018-05-15 16:02 P.M PDT :release-by: Vineet Goel - **Worker**: Stop reading changelog when no remaining messages. .. _version-1.0.9: 1.0.9 ===== :release-date: 2018-05-15 15:42 P.M PDT :release-by: Vineet Goel - **Worker**: Do not stop reading standby updates. .. _version-1.0.8: 1.0.8 ===== :release-date: 2018-05-15 11:00 A.M PDT :release-by: Vineet Goel - **Tables** + Fixes bug due to which we were serializing ``None`` values while recording a key delete to the changelog. This was causing the deleted keys to never be deleted from the changelog. + We were earlier not persisting offsets of messages read during changelog reading (or standby recovery). This would cause longer recovery times if recovery was ever interrupted. - **App**: Added flight recorder for consumer group rebalances for debugging. .. _version-1.0.7: 1.0.7 ===== :release-date: 2018-05-14 4:53 P.M PDT :release-by: Ask Solem - **Requirements** + Now depends on :ref:`Mode 1.12.5 `. - **App**: ``key_type`` and ``value_type`` can now be set to: + :class:`int`: key/value is number stored as string + :class:`float`: key/value is floating point number stored as string. + :class:`decimal.Decimal` key/value is decimal stored as string. - **Agent**: Fixed support for ``group_by``/``through`` after change to reuse the same stream after agent crashing. - **Agent**: Fixed ``isolated_partitions=True`` after change in v1.0.3. Initialization of the agent-by-topic index was in :ref:`version-1.0.3` moved to the ``AgentManager.start`` method, but it turns out ``AgentManager`` is a regular class, and not a service. ``AgentManager`` is now a service responsible for starting/stopping the agents required by the app. - **Agent**: Include active partitions in repr when ``isolated_partitions=True``. - **Agent**: Removed extraneous 'agent crashed' exception in logs. - **CLI**: Fixed autodiscovery of commands when using ``faust -A app``. - **Consumer**: Appropriately handle closed fetcher. - New shortcut: :func:`faust.uuid` generates UUID4 ids as string. .. _version-1.0.6: 1.0.6 ===== :release-date: 2018-05-11 11:15 A.M PDT :release-by: Vineet Goel - **Requirements**: + Now depends on Aiokafka 0.4.7. - **Table**: Delete keys when raw value in changelog set to :const:`None` This was resulting in deleted keys still being present with value :const:`None` upon recovery. - **Transports**: Crash app on CommitFailedError thrown by :pypi:`aiokafka`. App would get into a weird state upon a commit failed error thrown by the consumer thread in the :pypi:`aiokafka` driver. .. _version-1.0.5: 1.0.5 ===== :release-date: 2018-05-08 4:09 P.M PDT :release-by: Ask Solem - **Requirements**: + Now depends on :ref:`Mode 1.12.4 `. - **Agents**: Fixed problem with hanging after agent raises exception. If an agent raises an exception we cannot handle it within the stream iteration, so we need to restart the agent. Starting from this change, even though we restart the agent, we reuse the same :class:`faust.Stream` object that the crashed agent was using. This makes recovery more seamless and there are fewer steps involved. - **Transports**: Fixed worker hanging issue introduced in 1.0.4. In version :ref:`version-1.0.4` we introduced a bug in the round-robin scheduling of topic partitions that manifested itself by hanging with 100% CPU usage. After processing all records in all topic partitions, the worker would spin loop. - **API**: Added new base class for windows: :class:`faust.Window` There was the typing interface :class:`faust.types.windows.WindowT`, but now there is also a concrete base class that can be used in for example ``Mock(autospec=Window)``. - **Tests**: Now takes advantage of the new :class:`~mode.utils.mocks.AsyncMock`. .. _version-1.0.4: 1.0.4 ===== :release-date: 2018-05-08 11:45 A.M PDT :release-by: Vineet Goel - **Transports**: In version-1.0.2_ we implemented fair scheduling in :pypi:`aiokafka` transport such that while processing the worker had an equal chance of processing each assigned Topic. Now we also round-robin through topic partitions within topics such that the worker has an equal chance of processing message from each assigned partition within a topic as well. .. _version-1.0.3: 1.0.3 ===== :release-date: 2018-05-07 3:45 P.M PDT :release-by: Ask Solem - **Tests**: + Adds 5650 lines of tests, increasing test coverage to 90%. - **Requirements**: + Now depends on :ref:`Mode 1.12.3 `. - **Development**: + CI now builds coverage. + CI now tests multiple CPython versions: * CPython 3.6.0 * CPython 3.6.1 * CPython 3.6.2 * CPython 3.6.3 * CPython 3.6.4 * CPython 3.6.5 - **Backward incompatible changes**: + Removed ``faust.Set`` unused by any internal applications. - **Fixes**: + ``app.agents`` did not forward app to :class:`~faust.agents.manager.AgentManager`. The agent manager does not use the app, but fixing this in anticipation of people writing custom agent managers. + :class:`~faust.agents.manager.AgentManager`: On partitions revoked the agent manager now makes sure there's only one call to each agents ``agent.on_partitions_revoked`` callback. This is more of a pedantic change, but could have caused problems for advanced topic configurations. .. _version-1.0.2: 1.0.2 ===== :release-date: 2018-05-03 3:32 P.M PDT :release-by: Ask Solem - **Transports**: Implements fair scheduling in :pypi:`aiokafka` transport. We now round-robin through topics when processing fetched records from Kafka. This helps us avoid starvation when some topics have many more records than others, and also takes into account that different topics may have wildly varying partition counts. In this version when a worker is subscribed to partitions:: [ TP(topic='foo', partition=0), TP(topic='foo', partition=1), TP(topic='foo', partition=2), TP(topic='foo', partition=3), TP(topic='bar', partition=0), TP(topic='bar', partition=1), TP(topic='bar', partition=2), TP(topic='bar', partition=3), TP(topic='baz', partition=0) ] .. note:: ``TP`` is short for *topic and partition*. When processing messages in these partitions, the worker will round robin between the topics in such a way that each topic will have an equal chance of being processed. - **Transports**: Fixed crash in :pypi:`aiokafka` transport. The worker would attempt to commit an empty set of partitions, causing an exception to be raised. This has now been fixed. - **Stream**: Removed unused method ``Stream.tee``. This method was an example implementation and not used by any of our internal apps. - **Stream**: Fixed bug when something raises :exc:`StopAsyncIteration` while processing the stream. The Python async iterator protocol mandates that it's illegal to raise :exc:`StopAsyncIteration` in an ``__aiter__`` method. Before this change, code such as this:: async for value in stream: value = anext(other_async_iterator) where ``anext`` raises :exc:`StopAsyncIteration`, Python would have the outer ``__aiter__`` reraise that exception as:: RuntimeError('__aiter__ raised StopAsyncIteration') This no longer happens as we catch the :exc:`StopAsyncIteration` exception early to ensure it does not propagate. .. _version-1.0.1: 1.0.1 ===== :release-date: 2018-05-01 9:52 A.M PDT :release-by: Ask Solem - **Stream**: Fixed issue with using :keyword:`break` when iterating over stream. The last message in a stream would not be acked if the :keyword:`break` keyword was used:: async for value in stream: if value == 3: break - **Stream**: ``.take`` now acks events *after* buffer processed. Previously the events were erroneously acked at the time of entering the buffer. .. note:: To accomplish this we maintain a list of events to ack as soon as the buffer is processed. The operation is ``O(n)`` where ``n`` is the size of the buffer, so please keep buffer sizes small (e.g. 1000). A large buffer will increase the chance of consistency issues where events are processed more than once. - **Stream**: New ``noack`` modifier disables acking of messages in the stream. Use this to disable automatic acknowledgment of events:: async for value in stream.noack(): # manual acknowledgment await stream.ack(stream.current_event) .. admonition:: Manual Acknowledgment The stream is a sequence of events, where each event has a sequence number: the "offset". To mark an event as processed, so that we do not process it again, the Kafka broker will keep track of the last committed offset for any topic. This means "acknowledgment" works quite differently from other message brokers, such as RabbitMQ where you can selectively ack some messages, but not others. If the messages in the topic look like this sequence: .. sourcecode:: text 1 2 3 4 5 6 7 8 You can commit the offset for #5, only after processing all events before it. This means you MUST ack offsets (1, 2, 3, 4) *before* being allowed to commit 5 as the new offset. - **Stream**: Fixed issue with ``.take`` not properly respecting the ``within`` argument. The new implementation of take now starts a background thread to fill the buffer. This avoids having to restart iterating over the stream, which caused issues. .. _version-1.0.0: 1.0.0 ===== :release-date: 2018-04-27 4:13 P.M PDT :release-by: Ask Solem - **Models**: Raise error if ``Record.asdict()`` is overridden. - **Models**: Can now override ``Record._prepare_dict`` to change the payload generated. For example if you want your model to serialize to a dictionary, but not have any fields with :const:`None` values, you can override ``_prepare_dict`` to accomplish this: .. sourcecode:: python class Quote(faust.Record): ask_price: float = None bid_price: float = None def _prepare_dict(self, data): # Remove keys with None values from payload. return {k: v for k, v in data.items() if v is not None} assert Quote(1.0, None).asdict() == {'ask_price': 1.0} - **Stream**: Removed annoying ``Flight Recorder`` logging that was too noisy.