Change history for Faust 1.4

This document contain change notes for bugfix releases in the Faust 1.4 series. If you’re looking for previous releases, please visit the History section.

1.4.6

release-date:2019-01-29 01:52 P.M PDT
release-by:Ask Solem (@ask)
  • App: Better support for custom boot strategies by having the app start without waiting for recovery when no tables started.
  • Docs: Fixed doc build after intersphinx
    URL https://click.palletsprojects.com/en/latest no longer works.

1.4.5

release-date:2019-01-18 02:15 P.M PDT
release-by:Ask Solem (@ask)
  • Fixed typo in 1.4.4 release (on_recovery_set_flags -> on_rebalance_start).

1.4.4

release-date:2019-01-18 01:10 P.M PDT
release-by:Ask Solem (@ask)

  • Requirements

  • App: App now starts even if there are no agents defined.

  • Table: Added new flags to detect if actives/standbys are ready.

    • app.tables.actives_ready

      Set to True when active tables are recovered from and are ready to use.

    • app.tables.standbys_ready

      Set to True when standbys are up to date after recovery.

1.4.3

release-date:2019-01-14 03:01 P.M PDT
release-by:Ask Solem (@ask)

  • Requirements

    • Require series 0.4.x of robinhood-aiokafka.

      • Recently version 0.5.0 was released but this has not been tested in production yet, so we have pinned Faust 1.4.x to aiokafka 0.4.x. For more information see Issue #277.

    • Test requirements now depends on pytest greater than 3.6.

      Contributed by Michael Seifert (@seifertm).

  • Documentation improvements by:

  • CI: Added CPython 3.7.2 and 3.6.8 to Travis CI build matrix.

1.4.2

release-date:2018-12-19 12:49 P.M PDT
release-by:Ask Solem (@ask)

  • Requirements

  • Agent: Allow yield in agents that use Stream.take (Issue #237).

  • App: Fixed error “future for different event loop” when web views

    send messages to Kafka at startup.

  • Table: Table views now return HTTP 503 status code during startup when table routing information not available.

  • App: New App.BootStrategy class now decides what services are started when starting the app.

  • Documentation fixes by:

1.4.1

release-date:2018-12-10 4:49 P.M PDT
release-by:Ask Solem (@ask)
  • Web: Disable aiohttp access logs for performance.

1.4.0

release-date:2018-12-07 4:29 P.M PDT
release-by:Ask Solem (@ask)

  • Requirements

  • Worker: The Kafka consumer is now running in a separate thread.

    The Kafka heartbeat background corutine sends heartbeats every 3.0 seconds, and if those are missed rebalancing occurs.

    This patch moves the aiokafka library inside a separate thread, this way it can send responsive heartbeats and operate even when agents call blocking functions such as time.sleep(60) for every event.

  • Table: Experimental support for tables where values are sets.

    The new app.SetTable constructor creates a table where values are sets. Example uses include keeping track of users at a location: table[location].add(user_id).

    Supports all set operations: add, discard, intersection, union, symmetric_difference, difference, etc.

    Sets are kept in memory for fast operation, and this way we also avoid the overhead of constantly serializing/deserializing the data to RocksDB. Instead we periodically flush changes to RocksDB, and populate the sets from disk at worker startup/table recovery.

  • App: Adds support for crontab tasks.

    You can now define periodic tasks using cron-syntax:

    @app.crontab('*/1 * * * *', on_leader=True)
async def publish_every_minute():
    print('-- We should send a message every minute --')
    print(f'Sending message at: {datetime.now()}')
    msg = Model(random=round(random(), 2))
    await tz_unaware_topic.send(value=msg).

    See Cron Jobs for more information.

    Contributed by Omar Rayward (@omarrayward).

  • App: Providing multiple URLs to the broker setting now works as expected.

    To facilitiate this change app.conf.broker is now List[URL] instead of a single URL.

  • App: New timezone setting.

    This setting is currently used as the default timezone for crontab tasks.

  • App: New broker_request_timeout setting.

    Contributed by Martin Maillard (@martinmaillard).

  • App: New broker_max_poll_records setting.

    Contributed by Alexander Oberegger (@aoberegg).

  • App: New consumer_max_fetch_size setting.

    Contributed by Matthew Stump (@mstump).

  • App: New producer_request_timeout setting.

    Controls when producer batch requests expire, and when we give up sending batches as producer requests fail.

    This setting has been increased to 20 minutes by default.

  • Web: aiohttp driver now uses AppRunner to start the web server.

    Contributed by Mattias Karlsson (@stevespark).

  • Agent: Fixed RPC example (Issue #155).

    Contributed by Mattias Karlsson (@stevespark).

  • Table: Added support for iterating over windowed tables.

    See Iterating over keys/values/items in a windowed table..

    This requires us to keep a second table for the key index, so support for windowed table iteration requires you to set a use_index=True setting for the table:

    windowed_table = app.Table(
    'name',
    default=int,
).hopping(10, 5, expires=timedelta(minutes=10), key_index=True)

    After enabling the key_index=True setting you may iterate over keys/items/values in the table:

    for key in windowed_table.keys():
    print(key)

for key, value in windowed_table.items():
    print(key, value)

for value in windowed_table.values():
    print(key, value)

    The items and values views can also select time-relative iteration:

    for key, value in windowed_table.items().delta(30):
    print(key, value)
for key, value in windowed_table.items().now():
    print(key, value)
for key, value in windowed_table.items().current():
    print(key, value)
  • Table: Now raises error if source topic has mismatching

    number of partitions with changelog topic. (Issue #137).

  • Table: Allow using raw serializer in tables.

    You can now control the serialization format for changelog tables, using the key_serializer and value_serializer keyword arguments to app.Table(...).

    Contributed by Matthias Wutte (@wuttem).

  • Worker: Fixed spinner output at shutdown.

  • Models: isodates option now correctly parses timezones without separator such as -0500.

  • Testing: Calling AgentTestWrapper.put now propagates exceptions raised in the agent.

  • App: Default value for stream_recovery_delay is now 3.0 seconds.

  • CLI: New command “clean_versions” used to delete old version directories (Issue #68).

  • Web: Added view decorators: takes_model and gives_model.