Streamlining Haystack Data Processing with Rust Tools and Libraries

Project details for haystack client, library and utilities for ETL and interacting with haystack servers and haystack native data types and formats.

Important

Note: This project is actively being developed, and users may encounter breaking changes when using it in libraries or automation scripts and upgrading between versions. Because of this, please ensure you thoroughly test your workflows when adopting new versions of this tool until the project stabilizes.

If you have any feedback or suggestions for improvement, feel free to raise an issue on Github .

Overview

As far as I’m aware, utilities for interacting with zinc or SkySpark native data do not exist. This project aims to fill that gap by providing a suite of tools inspired by the GNU/Linux ecosystem. These tools enable remote access, data reads and writes, and efficient data wrangling of supported data formats.

Users familiar with Linux shells will find it easy to connect to Haystack servers with the same convenience as ssh or mosh, and chain multiple CLI calls without needing to open a browser.

Sub crates

haystack-client (CLI)

Command line utility for interacting with a haystack server and supporting SkySpark-specific REST calls. The CLI also includes a repl.

Usage

# Reuse haystack bearer token
( # () Opens a sub-shell
  export HAYSTACK_AUTH_CONFIG=`hs default auth`;
  
  # Destination does not need to be included if the environment variable
  # HAYSTACK_AUTH_CONFIG is set.
  
  # I also set up my .bashrc to use the alias "hs" instead of the full name.
  # Stick with `haystack-client` if you haven't configured your .bashrc this way
  hs read --filter "point" --limit=1

  # Example below of correct character escaping in bash shell when passing
  # filters enclosed by either single or double quotes.
  hs read --filter 'point and unit==\"kWh\" and equipRef->siteRef'
  hs read --filter "point and unit==\\\"kWh\\\" and equipRef->siteRef"
)

# Read–eval–print loop (REPL) You will see the prompt "hs〉"
# To learn about the $DEST argument, see the configuration section below.
haystack-client $DEST repl
hs〉about

# About
haystack-client $DEST about

# Read by filter
haystack-client $DEST read --filter "point and unit==\"kWh\" and equipRef->siteRef"

# Read by ids
haystack-client $DEST read --ids @p:demo:r:2f70054a-87f6d1de @p:demo:r:2f70054a-314342cd

# His Read (single)
haystack-client $DEST hisRead yesterday @p:demo:r:2f70054a-87f6d1de

# His Read (batch, could use the optional --timezone argument)
haystack-client $DEST hisRead yesterday @p:demo:r:2f70054a-87f6d1de @p:demo:r:2f70054a-314342cd

# WatchSub
## Can create watches with or without an id list
haystack-client $DEST watchSub -c "test" @p:demo:r:2f70054a-51d71f8e

## Subscribing points to a watch
haystack-client $DEST watchSub -s "w-2f8e0d48-64f17e75" @p:demo:r:2f70054a-51d71f8e @p:demo:r:2f70054a-69f26216

# Unsubscribing and closing a watch
haystack-client $DEST watchUnsub --close w-2f8e1d8b-8efac249
haystack-client $DEST watchUnsub @p:demo:r:2f70054a-51d71f8e

# Watch Poll (with or without --refresh)
haystack-client $DEST watchPoll w-2f8e2739-3c4b3bde --refresh

Configuration

When the destination is entered by name, the utility searches for the config.yaml under %LOCALAPPDATA% in Windows and ~/.config in Linux. The tool is configured similarly to ssh config enabling haystack-client to choose pre-configured destinations. If destination is not configured the user must enter entire server URL and credentials as command-line parameters.

# identifier used as $DEST in command line
name: default
url: https://localhost/api/demo
username: superUser
password: accountPassword
# Set to true in my case. Should be false in production
accept-invalid-certs: true

Important

Info on accept-invalid-certs here. If used in production, this should either be set to false or excluded from the config as it will accept any certificate for any site it’s connected to.

Supported REST Operations

• About
• Close
• Defs
• Libs
• Ops
• Filetypes
• Read
  • By filter
  • By id
• Nav
• WatchSub
• WatchUnsub
• WatchPoll
• PointWrite
• hisRead
  • Single
  • Batch
• hisWrite (Must be valid zinc str)
  • Single
  • Batch
• Invoke Action

haystack-client (Library)

The underlying API library implementing an asynchronous haystack client. API may change and will be documented further once finalised.

haystack-types

Rust-native implementations of haystack data types as well as zinc parsers for each type enabling interactivity with haystack servers.

Project Status

The project is currently in active development:

• The CLI and library components are functional and undergoing improvements.
• haystack-types is well under development but its API is subject to change.

Conclusion

This project is actively being developed, and support will continue to improve over time. When the repo is made public, you may download or build the source code, via the repository link below.

Github Repository